C#: activation functions and their ranges

Compare sigmoid, tanh, and ReLU on the same inputs.

using System;
using System.Globalization;

class Program {
  static string F(double v) {
    return v.ToString("F3", CultureInfo.InvariantCulture);
  }

  static void Main() {
    Console.WriteLine("x     sigmoid  tanh    relu");
    foreach (double x in new[] { -2.0, 0.0, 2.0 }) {
      double sigmoid = 1.0 / (1.0 + Math.Exp(-x));
      double tanh = Math.Tanh(x);
      double relu = Math.Max(0.0, x);
      Console.WriteLine(F(x).PadLeft(6) + F(sigmoid).PadLeft(9) + F(tanh).PadLeft(8) + F(relu).PadLeft(8));
    }
  }
}

Output

x     sigmoid  tanh    relu
-2.000    0.119  -0.964   0.000
 0.000    0.500   0.000   0.000
 2.000    0.881   0.964   2.000

Explanation

An activation function adds non-linearity after a weighted sum. Without it, a stack of layers is just one linear function. The sigmoid maps to (0, 1) and is used for probabilities. The hyperbolic tangent maps to (−1, 1) and is centred on zero. ReLU, max(0, x), is cheap and does not saturate for positive inputs.

Sigmoid and tanh flatten out for large positive and negative inputs, so their gradients become tiny, which slows learning in deep networks (the vanishing-gradient problem). ReLU has a gradient of exactly 1 for positive inputs, but 0 for negative ones, so a unit can get stuck at zero. Variants such as leaky ReLU keep a small slope for negative values.

Explore the dry run

Guided walkthroughReady

1using System;

2using System.Globalization;

4class Program {

5 static string F(double v) {

6 return v.ToString("F3", CultureInfo.InvariantCulture);

7 }

9 static void Main() {

10 Console.WriteLine("x sigmoid tanh relu");

11 foreach (double x in new[] { -2.0, 0.0, 2.0 }) {

12 double sigmoid = 1.0 / (1.0 + Math.Exp(-x));

13 double tanh = Math.Tanh(x);

14 double relu = Math.Max(0.0, x);

15 Console.WriteLine(F(x).PadLeft(6) + F(sigmoid).PadLeft(9) + F(tanh).PadLeft(8) + F(relu).PadLeft(8));

16 }

17 }

18}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Print the column headings once.	`—`	x sigmoid tanh relu
Sigmoid(−2) ≈ 0.119, sigmoid(0) = 0.5, sigmoid(2) ≈ 0.881.	`—`	x sigmoid tanh relu
tanh is odd: tanh(−2) = −tanh(2) ≈ −0.964.	`—`	x sigmoid tanh relu
Negative inputs become 0; positive ones pass through unchanged.	`—`	x sigmoid tanh relu
A small table: the outputs lie in different ranges.	`—`	x sigmoid tanh relu -2.000 0.119 -0.964 0.000 0.000 0.500 0.000 0.000 2.000 0.881 0.964 2.000

Cost and common mistakes

O(1) per value.

Overflow is rarely a problem for tanh and sigmoid, but Math.Exp(-x) becomes huge for large negative x. Production code uses numerically stable forms.
Weights should be initialised so that the activations do not saturate at the start of training.

Try it yourself

Which of the three is zero for every negative input?

Reveal answer and reasoning

ReLU: max(0, x) returns 0 whenever x is negative.

C#: activation functions and their ranges

Explanation

Follow the example

Cost and common mistakes

Try it yourself

Further reading