Activation Functions

Implement the four activations you will meet everywhere, each elementwise.

relu(x)             = max(0, x)
sigmoid(x)          = 1 / (1 + exp(-x))
tanh(x)             = (exp(x) - exp(-x)) / (exp(x) + exp(-x))
leaky_relu(x, a)    = x if x > 0 else a * x

Write tanh using np.tanh, which is already stable.

sigmoid is the one that needs care. Written directly, exp(-x) overflows for large negative x. Branch on the sign instead:

x >= 0:  1 / (1 + exp(-x))
x <  0:  exp(x) / (1 + exp(x))

Both branches are the same function, but each only ever exponentiates a negative number, so neither can overflow. np.exp warns and returns inf rather than raising, so an unstable version fails quietly on real data.

At exactly x = 0, relu and leaky_relu both return 0.

Function signature

def relu(x):
def sigmoid(x):
def tanh(x):
def leaky_relu(x, alpha=0.01):

Example

Input

relu(np.array([-1.0, 0.0, 2.0]))
leaky_relu(np.array([-1.0, 2.0]), alpha=0.1)

Output

array([0., 0., 2.])
array([-0.1,  2. ])

Reasoning

relu sets the negative entry to 0 and keeps 2. leaky_relu multiplies the negative entry by alpha, so -1 becomes -0.1, and keeps 2 as it is.

Constraints and notes

• All four functions work elementwise on arrays of any shape.
• sigmoid must stay finite for large negative inputs such as -1000.
• At x = 0, relu and leaky_relu return 0.