Learn everything about TF-IDF in rnn

In this technique, we have two things one is TF(Term frequency) and the other is IDF(Inverse document frequency). We find TF and IDF separately and then multiply TF and IDF.

Formula of TF: TF= No of represent of words in sentence / No of words in sentence

Formula of IDF: IDF=log(no of sentences / No of sentences containing words)

Here we have three sentences.
Sentence 1--> He is a good boy
Sentence 2--> He is a good girl
Sentence 3-->Boy and girl are good.

At first, we have to convert the sentences words into lower case. Because in the first sentence we have 'boy' and in the third sentence we have 'Boy'. Now if we don't lower the sentence words then 'Boy' will consider as one word and 'boy' will consider as one word. It means two different words but both are same. So we have to convert the sentence words into lower case. Sometimes it creates problems like if we have 'USA' and 'us', then we can't perform this technique. That time what we can do is that, we can consider those words in a separate list, and later on we can just keep that particular words and remaining all words we can convert into lower case. Then we have to perform tokenization, stop words, lemmatization, or stemming. For example, we took three sentences, not a paragraph.

After performing our sentences will look like this:
Sentence 1--> good boy
Sentence 2--> good girl
Sentence 3-->Boy girl good

Now we will create a histogram:
In histogram we will find the frequency for each word. Frequency means the count of a word. Suppose first we take 'boy', now in the first sentence we have 'boy' one time and in the second sentence we don't have 'boy' and in the third sentence, we have 'boy' one time. So total we have 'boy' two times. So the frequency of 'boy' is 2. Similarly, the frequency of 'girl' is 2 and good is 3.
good--->3
girl--->2
boy---->2
We have to always sort this frequency in descending order.

Table for TF:

We will get the table using the TF formula.

Here in the first sentence we have two words and we have "good" one time so we wrote 1/2 and we have "boy" one time so we wrote 1/2 and we don't have "girl" in the first sentence so we wrote 0. Similar for other.

Table for IDF:

We will get the table using the IDF formula.

Here we have three sentences. Here three sentences contain "good" words. So we wrote log(3/3) and two sentences containing "girl" and "boy" so we wrote log(2/3).

TF-IDF we can see that those words which are not that important have value 0 and that word which is more important has different value than other words present in a sentence.

Code: