How to Use the Tokenize Module in Python

This article will cover a guide on using the “Tokenize” module in Python. The tokenize module can be used to segment or divide the text into small pieces in various ways. You can use these segments in Python applications that use machine learning, natural language processing, and artificial intelligence algorithms. All the code samples in this article are tested with Python 3.9.5 on Ubuntu 21.04.

About the Tokenize Module

As the name suggests, the tokenize module can be used to create “tokens” from a paragraph or a chunk of text. Each individual broken piece returned after the tokenization process is called a token. Once you tokenize a text, you can implement your own logic in your Python program to process the tokens according to your use case. The tokenize module provides some useful methods that can be used to create tokens. The usage of these methods can be best understood through examples. Some of them are explained below.

Tokenizing a Paragraph or Sentence

You can tokenize a paragraph or a sentence with space-separated words using the code sample explained below.

The first two statements import the necessary Python modules required for converting a piece of text into individual tokens. A variable called “text” contains an example string. Next, the “tokenize” method from the tokenize module is called. It uses the “readline” method as a mandatory argument. Since the text variable is of “str” type, using it directly will throw an error. The readline argument is a callable method that must return bytes instead of a string for the tokenize method to work correctly. So using the “BytesIO” class, the text is converted into a stream of bytes by specifying an encoding type.

The tokenize method generates a named tuple containing five types: type (type of the token), string (name of the token), start (starting position of the token), end (ending position of the token), and line (the line that was used for creating tokens). So after running the above code sample, you should get an output similar to this:

As you can see in the output above, the tokenize method generates a “TokenInfo” object with five types mentioned above. If you want to access these types individually, use dot notation (as shown in the code sample below).

After running the above code sample, you should get the following output:

Note that the “t.type” call returns a constant number for the token type. If you want a more human-readable token type, use the “token” module and the “tok_name” dictionary available in it.

By supplying “t.type” constant to “tok_name” dictionary, you can get a human readable name for the token type. After running the above code sample, you should get the following output:

A full list of all token types and their names is available here. Note that the first token is always the encoding type of input stream, and it doesn’t have a start and end value.

You can easily get a list of just token names using for loop statements or list comprehensions, as shown in the code sample below.

After running the above code sample, you should get the following output:

You can use the “generate_tokens” method available in the tokenize module if you want to tokenize a string without converting it to bytes. It still takes a callable readline method as the mandatory argument, but it only handles strings returned by the readline method and not bytes (unlike the tokenize method explained above). The code sample below illustrates the usage of the generate_tokens method. Instead of the BytesIO class, now the “StringIO” class is used.

After running the above code sample, you should get the following output:

Tokenizing Contents of a File

You can use the “with open” statement in “rb” mode to directly read the contents of a file and then tokenize it. The “r” in “rb” mode stands for read-only mode while “b” stands for binary mode. The code sample below opens a “sample.txt” file and tokenizes its contents using the tokenize and readline methods.

You can also use “open”, a convenience method available in the tokenize module, and then call generate_tokens and readline methods to create tokens from a file directly.

Assuming that the sample.txt file contains the same example string, you should get the following output after running the two code samples explained above.

Conclusion

The tokenize module in Python provides a useful way to tokenize chunks of text containing space-separated words. It also creates a map of starting and ending positions of tokens. If you want to tokenize each and every word of a text, the tokenize method is better than the “split” method as it also takes care of tokenizing punctuation characters / other symbols and also infers the token type.