Deep learning working process of artificial neural network

Architecture of neural network

When we were kids and when we first time saw a cat or dog, we didn't know that which one is a cat and which one is a dog. When someone told us some features like the dog is taller than the cat, eye color, etc then we understood which one is dog and which one is cat. In the same way, we give various features in a neural network and it trains itself and then does predictions for us.

At first, the neural network has an input layer, then hidden layers, and then the output layer.

In neural network features are the input and the input layer takes input means features. So first pass the features to the neural network input layer. Here the features goes row wise. It means first row of the dataset, then second row then third row. If we have 5 features then each row will contain 5 different values. So, the input layer number of nodes or neurons are equal to the number of features.

Then all those information goes to the hidden layers and there some preprocessing or calculation happens. In hidden layers, there can be any number of neurons and hidden layers. Every node/neuron of the input layer is connected with all the neurons of first hidden layer. Suppose you have 2 input nodes and 3 neurons in the hidden layer, so the first input node will connect with three neurons of the hidden layer, and similarly, the second input node will also connect with all three neurons of the hidden layer. Now if you have second hidden layer, then all neurons or nodes of first hidden layer will be connected with the each node of second hidden layer.

Then you will see the output layers. In the output layer you can have single neuron and also multiple neurons. You can have single output layer and multiple output layer also. So the output of the last hidden layers goes to the output layer. Some calculation also happens in the output layer. From the output layers we get the output.

What is perceptron?

Perceptron is a neural network which contain one hidden and output layer where each layer contain single neuron. There can be multiple input nodes or neurons but can have only one hidden and output layer and inside the hidden and output layer, only one neuron is present. Now update methods have come and now we don't use this but it is the base of neural network architecture. If you don't understand how perceptron works then it will be tough for you to understand how neural network works.

How perceptron works?

Suppose you have 3 features X1, X2, X3. Because there are three features, so three nodes will be created in the input layer. Each node has a particular weight. So for three nodes, you will assign three different weights W1, W2, W3. Now each input node value and weight will goes to the neurons of hidden layer. All the calculation happens in the hidden layer and output layer.

Two types of operation happen inside a neuron.

Process happens in a neuron:
Step 1:
At first you will multiply weight and inputs. Then do summation of all the result of weight and input multiplication.
y=W₁*X₁ + W₂*X₂ + W₃*X₃ +....+ W_n*X_n
Here,
X1=input 1, W1=weight of X1.
X2=input 2, W2=weight of X2.
X3=input 3, W3=weight of X3.

Now add a bias parameter. You will learn about bias in the next lecture but understand that bias is a one kind of smaller value that gets added to the y value:
y=W₁*X₁ + W₂*X₂ + W₃*X₃ +....+ W_n*X_n + bias

Step 2:
After the calculation, you will run an activation function on the calculated value. Activation function and weight plays a very important role. You will learn about the activation function in the next lecture.

After this, the output of the hidden layer neuron will go to the output layer node. There also some work will happen. After the work, you will get the output from the output layer.

Let's see an example to understand the activation and weights working process
Suppose you put a hot object in your right hand. When you put it on the right hand then your right hand will give a reaction but the left hand will not give any reaction. The right hand gives a reaction because when you put a hot object, the neuron gets activated.
Why the neurons gets activated?
Because of the weights become higher of right hand neurons and then a activation function runs and decide that should it be activated or not.
Because the weights gets higher so the activation function will activate the right hand and the left hand will not because of no changes in weights.

So the weight activates the activation function and then the activation function decides that should it get activate or not.

What is multiple layer neural network?

In multiple neural networks, there can be multiple hidden layers, and multiple output layers, but you will always have only one input layer. In perceptron, you saw that there is only one hidden layer and only one neuron is present in the hidden layer, but in multiple neural networks there can be multiple hidden layers and in each hidden layer can have multiple neurons. You also saw in perceptron that there is only one output layer but in multiple neural networks, there can be multiple or single output layers and each layer can contain single or multiple output nodes.

How multiple neural network works?

Suppose You have four features X₁ ,X₂, X₃, X₄ and two hidden layers and one output layer. In the first hidden layer, you have three neurons, and in the second hidden layer, you have two neurons. You know that each node of the input layer is connected with each neuron of the first hidden layer. If there are multiple hidden layers then each hidden layer each neuron is also connected with other hidden layers neurons. Like first hidden layer neurons connected with second hidden layer neurons, second layer neurons are connected with third hidden layer neurons and this chain will go on till the output layer. The last hidden layer neurons are connected with the output layer neuron.

So when all nodes of input layer, X₁ ,X₂,X₃ and X₄ are connected with the first hidden layer, then each node gets a different weight.
So for first neuron the weights are W₁(1),W₁(2),W₁(3).
Here 1, 2, 3 in the bracket indicates the number of neurons of first hidden layer. W1 means the weight of the input layer first node and (1) means the first neuron of the first hidden layer. So W1(1) means input layer, first node weight for the first neuron of the first hidden layer. Similarly
First hidden layer weights are:
Weights for first node is: W₁(1),W₁(2),W₁(3)
Weights for second node is: W₂(1),W₂(2),W₂(3)
Weights for third node is: W₃(1),W₃(2),W₃(3)
Weights for fourth node is: W₄(1),W₄(2),W₄(3)
You know that all neurons and nodes are connected. So if input layers nodes are connected with each neuron of the first hidden layer neurons. It means that the first hidden layer neurons will be also connected with the second hidden layer neurons and when they are connected, it means that they also have weights. So the weights
For second hidden layer:
W₁₁(1),W₁₁(2)
W₂₂(1),W₂₂(2)
W₃₃(1),W₃₃(2)
Here W₁₁ means first hidden layer first neuron and (1) means second layer first neuron. Now the second layer is connected with output layer.
Output layers weights are:
W₁₁₁(1), W₂₂₂(1)
Here W₁₁₁ means second hidden layer first neuron and W₂₂₂ means second hidden layer second neuron and (1) means output layer first neuron. Here you have only one neuron but there can be multiple neurons in the output layer.

You know that some calculation happens
in neuron and we divide it into two steps:

1. Summation of all inputs nodes values,weights and bias:
Formula:y=W₁*X₁ + W₂*X₂ + W₃*X₃ +....+ W_n*X_n + bias
In this case:
For First hidden layer:
for first neuron:
y₁=X₁*(W₁₍₁₎)+X₂*(W₂₍₁₎)+X₃(W₃₍₁₎)+X₄*(W₄₍₁₎)+bias
For second neuron:
y₂=X₁*(W₁₍₂₎)+X₂*(W₂₍₂₎)+X₃(W₃₍₂₎)+X₄*(W₄₍₂₎)+bias
for third neuron:
y₃=X₁*(W₁₍₃₎)+X₂*(W₂₍₃₎)+X₃(W₃₍₃₎)+X₄*(W₄₍₃₎)+bias

2. Run an activaiton function.

These two steps happen in each neuron. Here we apply an activation function on the result that we get in step 1. There are a lot of activation functions and we will understand that in the future but in this case, we are using relu.
So its will look like :
relu(y₁), relu(y₂), relu(y₃).

After doing the calculation our work is done. Now, these steps will happen in each hidden layer neurons.
It means now these outputs and weights will go to the second hidden layer neurons and then again step 1 and step 2 will happen.
Then the output of the second hidden layer neurons and weights will go to our output layer node. The same steps will also happen in the output layer. After the calculation, we will get an output from the output layer.

Sometimes we get the wrong result or prediction and this is a problem. To get the right result we use loss or cost function. We use the loss function when we have one output and we use the cost function when we have multiple outputs. Loss and cost functions are nothing but finding the difference between actual value and predicted value. We will discuss loss functions later.

After getting the difference mean loss function value by applying the loss function, we try to reduce it and try to make it near to zero. To reduce we use an optimizer.

What the optimizer does is that it try to update all the weights in such a way that our predicted value should be equal to the actual value or near to the actual value. So this way we get a proper output and this way actually a neural network works.