Leaky ReLU Function Calculator
Compute the Leaky ReLU value f(x) and its derivative from an input x and slope alpha. Returns x when x is non-negative and alpha times x when negative, with a graph.
Input
Enter x and the negative slope alpha to compute the Leaky ReLU value f(x) and its derivative.
The input value passed through the activation function.
The slope used when x is negative. 0.01 is commonly used.
Result
f(-2) =
-0.02
Derivative f prime of x
0.01
Slope alpha
0.01
Active branch
x below 0 (f = alpha x)
Leaky ReLU graph
How it works
- Leaky ReLU is defined as f(x)=x when x is at least 0 and f(x)=alpha times x when x is negative. Alpha is the slope on the negative side (default 0.01).
- The derivative is 1 for x greater than 0 and alpha for x less than 0. At x=0 the function is non-differentiable, so this tool adopts the common convention f prime of 0 equals 1.
- Standard ReLU outputs 0 for all negative inputs and has a zero gradient there, which can cause the dying ReLU problem where neurons stop learning. Leaky ReLU keeps a small slope alpha on the negative side to ease this.
- When alpha is made a learnable parameter, the function is called PReLU.
Reviews
Tell us what you think of this calculator.
Write a review
- Home
Leaky ReLU Function Calculator