How to Perform Sentiment Analysis and Text Classification with Transformers?

PyTorch is majorly used for its applications of deep learning models. It is an open-source framework for complex machine-learning tasks. Pytorch combines the functionality of complex Torch libraries with Python to run and test portions of complex codes. This framework supports 200 different mathematical operations and continues to rise in popularity due to its extensive application in Machine learning and Deep learning.

This article covers the following aspects:

What are Transformers in Machine Learning?

Machine learning is a diverse field and with every passing day, there are innovations that bring about recent changes to the current usage of Machine Learning. Transformers are capable of training huge datasets depending upon the hardware of the system, unlike the traditional algorithms that were slow and could not process volumes of data even with efficient hardware.

Transformers can write any type of text, e.g., poems, articles, blogs, etc., translating the text while considering the context, and even writing computer codes. Some of the popular applications of the transformers model are:

  • GPT-3
  • BERT
  • T-5 etc

How do Transformers Operate in Machine Learning?

Encoders and Decoders are the building blocks of transformers. The Encoders and Decoders are further composed of layers. Encoders are made up of two layers which are Self-Attention Layer and a Feed Forward Neural Network Layer. While decoder’s foundations are laid by self-attention layers, and one layers of feed-forward neural network . All the data is fed to the self-attention layer of the Encoders. Inside this layer, each word is compared to the other words in the sentences to interpret the pattern and meanings of the data. The data is then forwarded to the Feedforward neural network.

The inputs that are being fed to the encoder or the decoder are the raw inputs, and all are embedded. Embeddings are a means of representing this information in n-vector length in transformers. On the top of these embeddings are the positional encodings. The Positional encoding plays a key role in avoiding the recurrence pattern as observed in the traditional algorithms (RNN). The positional encoding adds a number to the words in the sentence, hence eliminating the need for the recurrence to memorize the word order.

Now for the outputs, transformers have a linear and SoftMax layer. These layers are responsible for creating human-readable outputs. A vector is generated whose length is equal to the words that we have in our vocabulary.

How to Perform Sentiment Analysis with Transformers?

To get started with Hugging face transformers, follow the below-mentioned steps:

Step 1: Install Transformers

To get started with transformers, provide the following command:

!pip install transformers

Step 2: Classification of Transformers

Here, in this section of the article, we will perform sentimental analysis on the text using transformers. The classification can either be positive or negative. For this purpose, provide the following code:

from transformers import pipeline

classifier("We did not enjoy the show")

In the above code, pipeline(“sentiment-analysis”) indicates what operation should be performed e.g., sentiment analysis, text-to-speech, etc.

The output will be like this. The transformer can determine the positive or negative sentences:

How to Use Specific Parameters with Pipeline Library?

To use specific parameters with Pipeline library, use the pipeline() function as in the following command:

!pip install transformers

We can implement various parameters with the pipeline() function. Here is the link to the Google Colab to run the following code:

from transformers import pipeline

gen_pipe = pipeline(model="facebook/wav2vec2-large-960h-lv60-self", return_timestamps="word")

How to Resolve the Runtime NVIDIA Error?

The runtime NVIDIA error occurs because by default Google Colab runs on the CPU. In case, you encounter the following error:

Click on the “RAM/Disk” button in the top-right corner and switch to the “T4 GPU” option:

After that, re-run all the commands:

There are many operations that we can perform with transformers i.e., text-to-speech, computer vision techniques, sentimental analysis, text generation, text classification, image segmentation, etc.

That is all from this guide.

Click on this Colab Link for the above code.


To perform sentiment analysis and text classification with transformers, install the transformer library using pip, and use pipeline() method. Hugging face offers a variety of NLP tasks that can be performed with transformers. This article provides a step-by-step guide to working with Hugging face transformers in Python.

About the author

Syed Minhal Abbas

I hold a master's degree in computer science and work as an academic researcher. I am eager to read about new technologies and share them with the rest of the world.