C# · Course navigation

C#: activation functions and their ranges

Compare sigmoid, tanh, and ReLU on the same inputs.

On this page

Before you start

C# Math functions.

Compare sigmoid, tanh, and ReLU on the same inputs.
Predict the sample output and explain the state changes.
Complete the practice task and check the boundary cases.

The concept

An activation function adds non-linearity after a weighted sum. Without it, a stack of layers is just one linear function. The sigmoid maps to (0, 1) and is used for probabilities. The hyperbolic tangent maps to (−1, 1) and is centred on zero. ReLU, max(0, x), is cheap and does not saturate for positive inputs.

Sigmoid and tanh flatten out for large positive and negative inputs, so their gradients become tiny, which slows learning in deep networks (the vanishing-gradient problem). ReLU has a gradient of exactly 1 for positive inputs, but 0 for negative ones, so a unit can get stuck at zero. Variants such as leaky ReLU keep a small slope for negative values.

Code example

using System;
using System.Globalization;

class Program {
  static string F(double v) {
    return v.ToString("F3", CultureInfo.InvariantCulture);
  }

  static void Main() {
    Console.WriteLine("x     sigmoid  tanh    relu");
    foreach (double x in new[] { -2.0, 0.0, 2.0 }) {
      double sigmoid = 1.0 / (1.0 + Math.Exp(-x));
      double tanh = Math.Tanh(x);
      double relu = Math.Max(0.0, x);
      Console.WriteLine(F(x).PadLeft(6) + F(sigmoid).PadLeft(9) + F(tanh).PadLeft(8) + F(relu).PadLeft(8));
    }
  }
}

Output

x     sigmoid  tanh    relu
-2.000    0.119  -0.964   0.000
 0.000    0.500   0.000   0.000
 2.000    0.881   0.964   2.000

Example details

Follow the code

Open the walkthrough to inspect the example one step at a time.

Show guided walkthrough

Guided walkthroughReady

1using System;

2using System.Globalization;

4class Program {

5 static string F(double v) {

6 return v.ToString("F3", CultureInfo.InvariantCulture);

7 }

9 static void Main() {

10 Console.WriteLine("x sigmoid tanh relu");

11 foreach (double x in new[] { -2.0, 0.0, 2.0 }) {

12 double sigmoid = 1.0 / (1.0 + Math.Exp(-x));

13 double tanh = Math.Tanh(x);

14 double relu = Math.Max(0.0, x);

15 Console.WriteLine(F(x).PadLeft(6) + F(sigmoid).PadLeft(9) + F(tanh).PadLeft(8) + F(relu).PadLeft(8));

16 }

17 }

18}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Print the column headings once.	`—`	x sigmoid tanh relu
Sigmoid(−2) ≈ 0.119, sigmoid(0) = 0.5, sigmoid(2) ≈ 0.881.	`—`	x sigmoid tanh relu
tanh is odd: tanh(−2) = −tanh(2) ≈ −0.964.	`—`	x sigmoid tanh relu
Negative inputs become 0; positive ones pass through unchanged.	`—`	x sigmoid tanh relu
A small table: the outputs lie in different ranges.	`—`	x sigmoid tanh relu -2.000 0.119 -0.964 0.000 0.000 0.500 0.000 0.000 2.000 0.881 0.964 2.000

Behind the scenes

How C# processes this code

C# compiles to code for the .NET runtime, which invokes the program entry point.

Cost and common mistakes

O(1) per value.

Overflow is rarely a problem for tanh and sigmoid, but Math.Exp(-x) becomes huge for large negative x. Production code uses numerically stable forms.
Weights should be initialised so that the activations do not saturate at the start of training.

Try it yourself

Which of the three is zero for every negative input?

Reveal answer and reasoning

ReLU: max(0, x) returns 0 whenever x is negative.