Learn about popular weight initialization techniques

The key points of weight initialization:

1>Weight should be small. It means that the weight should be not so small and not so big. It should be a minimum small. Large weights cause exploding gradient problems especially while using the sigmoid activation function.
2>Weight should not be the same. If the weights are same then we will get the same output every time and it will prevent our neural network from learning new features.
3>Weight should have good variance. This will help each of the neurons to learn new features.

Types of weight initialization techniques:

Uniform distribution:

Here the weights are taken from a uniform distribution, weights are selected from a range and we know that range has a minimum and maximum value. So our target is to find the minimum and maximum value means creating the range. Then our system will pick a value from that range randomly.

W_ij=[(-1)/√Ni,1/√Ni]

Here the formula for minimum value is (-1)/√Ni and the formula of the maximum value is 1/√Ni. If we put the Ni value in the formula and solve the equation then we will get the range. Here Ni is the number of network connections that are coming out from a particular layer. Suppose we have 1 input layer and 1 hidden layer. In the input layer, we have 2 input nodes and in the hidden layer, we have 3 neurons. It means from each node three connections will go to the neurons. So for the 2 nodes total connection, we will get for three neurons is 6. So here Ni is 6. Now if we put these values in the equation we will get our minimum and maximum value means range.

Xavier/Glorat:

Here we can use two types of technique:

Xavier/Glorat normal:

The weight is selected from the normal distribution. The value is randomly selected from a range. Here the minimum value of the range is 0. We have to find the maximum value of the range.
The formula is :
Wid=ND(0,σ)

Here the formula of finding sigma is
σ=√{2/(Ni+N0)}

Here Ni is the number of network connections that are coming out from a particular layer and N0 is the number of network connections going out from a particular layer. Suppose we have 1 input layer and 1 hidden layer and 1 output layer. In the input layer, we have 2 input nodes and in the hidden layer we have 3 neurons and in the output layer, we have one neuron. For the hidden layer total, 6 connections are coming from the input layer and 3 connection is going out from the hidden layer to the output layer. So here Ni is 6 and N0 is 3. Now if we put these values in the equation we will get our minimum and maximum value means range.

Xavier/Glorat uniformal:

Weights are selected from a range and we know that range has a minimum and maximum value. So our target is to find the minimum and maximum value means creating the range then our system will pick value-form that range randomly.

The formula is :
W_ij=[(√-6)/Ni+N0,√6/√Ni+N0]

Here Ni is the number of network connections that are coming out from a particular layer and N0 is the number of network connections going out from a particular layer. Suppose we have 1 input layer and 1 hidden layer and 1 output layer. In the input layer, we have 2 input nodes and in the hidden layer we have 3 neurons and in the output layer, we have one neuron. For the hidden layer total, 6 connections are coming from the input layer and 3 connection is going out from the hidden layer to the output layer. SO here Ni is 6 and N0 is 3. Now if we put these values in the equation we will get our minimum and maximum value means range.

He init:

Here we can use two types of technique:

He init normal:

The weight is selected from the normal distribution. The value is randomly selected from a range. Here the minimum value of the range is 0. We have to find the maximum value of the range.

The formula is :
Wid=ND(0,σ)

Here the formula of find sigma is σ=√{2/Ni}

Here Ni is the number of network connections that are coming out from a particular layer Suppose we have 1 input layer and 1 hidden layer. In the input layer, we have 2 input nodes and in the hidden layer, we have 3 neurons. For the hidden layer total of 6 connections are coming from the input layer. So here Ni is 6. Now if we put these values in the equation we will get our minimum and maximum value means range.

He init uniform:

Weights are selected from a range and we know that range has a minimum and maximum value. So our target is to find the minimum and maximum value means creating the range then our system will pick value-form that range randomly.

The formula is :
Wij=[(√-6)/Ni,√6/√Ni]

Here Ni is the number of network connections that are coming out from a particular layer Suppose we have 1 input layer and 1 hidden layer. In the input layer, we have 2 input nodes and in the hidden layer, we have 3 neurons. For the hidden layer total of 6 connections are coming from the input layer. So here Ni is 6. Now if we put these values in the equation we will get our minimum and maximum value means range.