Here I shall implementing TFIDF from scratch in pure python without using sklearn or any other similar packages

First Some basics on TF-IDF

Tf-idf stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query.

One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.

Tf-idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.

Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:

Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as "is", "of", and "that", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following:

for numerical stabiltiy we will be changing this formula little bit

Example

Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.

Here I will implement a TFIDF vectorizer on a collection of text documents and then compare the results of my own implementation of TFIDF vectorizer with that of sklearns implemenation TFIDF vectorizer.

 Sklearn does few more tweaks in the implementation of its version of TFIDF vectorizer, so to replicate the exact results I would need to add following things to our custom implementation of tfidf vectorizer:

     - Sklearn has its vocabulary generated from idf sorted in alphabetical order
     
     - Sklearn formula of idf is different from the standard textbook formula. Here the constant "1" is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions.

• Sklearn applies L2-normalization on its output matrix. - The final output of sklearn tfidf vectorizer is a sparse matrix.

Steps to approach this problem :

• I would have to write both fit and transform methods for my custom implementation of tfidf vectorizer.

• Print out the alphabetically sorted voacb after I fit our data and check if its the same as that of the feature names from sklearn tfidf vectorizer.

• Print out the idf values from our implementation and check if its the same as that of sklearns tfidf vectorizer idf values.

•    Once I get our voacb and idf values to be same as that of sklearns implementation of tfidf vectorizer, proceed to the below steps.
  

•   Make sure the output of our implementation is a sparse matrix. Before generating the final output, I need to normalize our sparse matrix using L2 normalization. You can refer to this link https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html
  

•  After completing the above steps, print the output of our custom implementation and compare it with sklearns implementation of tfidf vectorizer.
  

•  To check the output of a single document in our collection of documents,  I can convert the sparse matrix related only to that document into dense matrix and print it.
  

The output of our custom implementation and that of sklearns implementation would match only with the collection of document strings I define below, as reference in this notebook. It would not match for strings that contain capital letters or punctuations, etc, because sklearn version of tfidf vectorizer deals with such strings in a different way.

fit() Method

• With this function, we will find all unique words in the data and assign a dimension-number to each unique word.

• So will define a python dictionary to store all the unique words, such that the key of dictionary represents a unique word and the corresponding value represent it's dimension-number.

• For example, if a review says, 'very good taste' - then I can represent each unique word with a dimension_number as,

   { 'very' : 1, 'good' : 2, 'taste' : 3}
  

• And remember our dataset is a list of string

In slightly more detail - In general whats exactly fit and transform method do in-scikit-learn

What fit() method does is create a model that extracts the various parameters from your training samples to do the neccessary transformation later on. transform() on the other hand is doing the actual transformation to the data itself returning a standardized or scaled form.

fit_transform() is just a faster way of doing the operations of fit() and transform() consequently.

Let us take an example for Scaling values in a dataset:

Here the fit method, when applied to the training dataset, learns the model parameters (for example, mean and standard deviation). We then need to apply the transform method on the training dataset to get the transformed (scaled) training dataset. We could also perform both of this steps in one step by applying fit_transform on the training dataset.

Hence, every sklearn's transform's fit() just calculates the relevant parameters (e.g. μ and σ in case of StandardScaler) and saves them as an internal object's state. Afterwards, you can call its transform() method to apply the transformation to any particular set of examples.

Then why do we need 2 separate methods - fit and transform ?

We use fit_transform() on the train data so that we learn the parameters of scaling on the train data and in the same time we scale the train data. We only use transform() on the test data because we use the scaling paramaters learned on the train data to scale the test data.

And here's why we do like that in detail

In practice we need to have a separate training and testing dataset and that is where having a separate fit and transform method helps. We apply fit on the training dataset and use the transform method on both - the training dataset and the test dataset. Thus the training as well as the test dataset are then transformed(scaled) using the model parameters that were learnt on applying the fit method the training dataset.

Important thing here is that when you divide your dataset into train and test sets what you are trying to achieve is somewhat simulate a real world application. In a real world scenario you will only have training data and you will develop a model according to that and predict unseen instances of similar data.

If you transform the entrire data with fit_transform() and then split to train test you violate that simulation approach and do the transformation according to the unseen examples as well. Which will inevatibly result in an optimistic model as you already somewhat prepared your model by the unseen samples metrics as well.

If you split the data to train test and apply fit_transform() to both you will also be mistaken as your first transformation of train data will be done by train splits metrics only and your second will be done by test metrics only.

The right way to do these preprocessings is to train any transformer with train data only and do the transformations to the test data. Because only then you can be sure that your resulting model represents a real world solution.

Example Code:

scaler = preprocessing.StandardScaler().fit(X_train)
scaler.transform(X_train)
scaler.transform(X_test)

Note that mean and std obtained from the training set are used for scaling all training dataset values. And we should not compute a separate mean and std on the test set to scale the test set values instead we have to use the ones obtained using fit on the training set. We have to ensure identical operation on test set.

The idea is, once we executed t.fit(train_data), t is fitted, so you can safely use

t.transform(test_data)

TFIDF - Sklearn Implementation

So from above we can see the dense-matrix printed are the same both for my custom function and from using scikit-learn's TfidfVectorizer

Implement max features functionality

Same as above implementation, but now ONLY consider top-50 words by top-50 IDF scores

As a part of this task I have to modify our fit and transform functions so that our vocab will contain only 50 terms with top idf scores.

This task is similar to our previous task, just that here our vocabulary is limited to only top 50 features names based on their idf values. Basically our output will have exactly 50 columns and the number of rows will depend on the number of documents I have in our corpus.

Here I will be give a pickle file, with file name cleaned_strings. You would have to load the corpus from this file and use it as input to our tfidf vectorizer.

Steps to approach this task:

     - You would have to write both fit and transform methods for our custom implementation of tfidf vectorizer, just like in the previous task. Additionally, here I have to limit the number of features generated to 50 as described above.
     
     - Now sort our vocab based in descending order of idf values and print out the words in the sorted voacb after I fit our data. Here I should be getting only 50 terms in our vocab. And make sure to print idf values for each term in our vocab.
     Make sure the output of our implementation is a sparse matrix.

     - Before generating the final output, I need to normalize our sparse matrix using L2 normalization. You can refer to [this link](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html)

     - Now check the output of a single document in our collection of documents,  I can convert the sparse matrix related only to that document into dense matrix and print it. And this dense matrix should contain 1 row and 50 columns.

As we can see above the dataframe has 1 row and 50 columns

Now will implement tf-idf with sklearn's TfidfVectorizer on the same Pickle dataset