Converting PySpark DataFrame to JSON

Transmitting a structured data using JSON is possible and also consumes low memory. When compared to PySpark RDD or PySpark DataFrame, JSON consumes low memory and serialization which is possible with JSON. We are able to convert the PySpark DataFrame to JSON using the pyspark.sql.DataFrameWriter.json() method. Apart from it, there are two other ways of converting the DataFrame to JSON.

Topic of Contents:

• PySpark DataFrame to JSON Using To_json() with ToPandas()
• PySpark DataFrame to JSON Using ToJSON()
• PySpark DataFrame to JSON Using Write.json()

Let’s consider a simple PySpark DataFrame in all the examples and convert it into JSON using the mentioned functions.

Required Module:

Install the PySpark library in your environment if it is not yet installed. You can refer to the following command to install it:

PySpark DataFrame to JSON Using To_json() with ToPandas()

The to_json() method is available in the Pandas module which converts the Pandas DataFrame to JSON. We can utilize this method if we convert our PySpark DataFrame to Pandas DataFrame. In order to convert the PySpark DataFrame to Pandas DataFrame, the toPandas() method is used. Let’s see the syntax of to_json() along with its parameters.

Syntax:

1. Orient is used to display the converted JSON as the desired format. It takes “records”, “table”, “values”, “columns”, “index”, “split”.
2. Index is used to include/remove the index from the converted JSON string. If it is set to “True”, the indices are displayed. Otherwise, the indices won’t be displayed if the orient is “split” or “table”.

Example 1: Orient as “Records”

Create a “skills_df” PySpark DataFrame with 3 rows and 4 columns. Convert this DataFrame to JSON by specifying the orient parameter as “records”.

Output:

We can see that the PySpark DataFrame is converted to the JSON array with a dictionary of values. Here, the keys represent the column name and the value represents the row/cell value in the PySpark DataFrame.

Example 2: Orient as “Split”

The JSON format that is returned by the “split” orient includes the column names which have a list of columns, list of index, and list of data. The following is the format of the “split” orient.

Output:

Example 3: Orient as “Index”

Here, each row from the PySpark DataFrame is retired in the form of a dictionary with the key as the column name. For each dictionary, the index position is specified as a key.

Output:

Example 4: Orient as “Columns”

Columns are the key for each record. Each column holds a dictionary that takes the column values with index numbers.

Output:

Example 5: Orient as “Values”

If you need only the values in JSON, you can go for the “values” orient. It displays each row in a list. Finally, all the lists are stored in a list. This JSON is of the nested list type.

Output:

Example 6: Orient as “Table”

The “table” orient returns the JSON that includes the schema with field names along with the column data types, the index as a primary key and the Pandas version. The column names with values are displayed as “data”.

Output:

Example 7: With Index Parameter

First, we pass the index parameter by setting it to “True”. You will see for each column value that the index position is returned as a key in a dictionary.

In the second output, only the column names (“columns”) and records (“data”) are returned without the index positions since the index is set to “False”.

Output:

PySpark DataFrame to JSON Using ToJSON()

The toJSON() method is used to convert the PySpark DataFrame to a JSON object. Basically, it returns a JSON string which is surrounded by a list. The [‘{column:value,…}’,…. ] is the format that is returned by this function. Here, each row from the PySpark DataFrame is returned as a dictionary with the column name as the key.

Syntax:

It can be possible to pass the parameters like the index, column labels, and data type.

Example:

Create a “skills_df” PySpark DataFrame with 5 rows and 4 columns. Convert this DataFrame to JSON using the toJSON() method.

Output:

There are 5 rows in the PySpark DataFrame. All these 5 rows are returned as a dictionary of strings which are separated by comma.

PySpark DataFrame to JSON Using Write.json()

The write.json() method is available in PySpark which writes/saves the PySpark DataFrame to a JSON file. It takes the file name/path as a parameter. Basically, it returns the JSON in multiple files (partitioned files). To merge all of them in a single file, we can use the coalesce() method.

Syntax:

1. Append Mode – dataframe_object.write.mode(‘append’).json(‘file_name’)
2. Overwrite Mode – dataframe_object.write.mode(‘overwrite’).json(‘file_name’)

It can be possible to append/overwrite the existing JSON. Using the write.mode(), we can append the data by passing “append” or overwrite the existing JSON data by passing “overwrite” to this function.

Example 1:

Create a “skills_df” PySpark DataFrame with 3 rows and 4 columns. Write this DataFrame to JSON.

JSON File:

We can see that the skills_data folder includes the partitioned JSON data.

Let’s open the JSON file. We can see that all the rows from the PySpark DataFrame are converted into JSON.

There are 5 rows in the PySpark DataFrame. All these 5 rows are returned as a dictionary of strings which are separated by comma.

Example 2:

Create a “skills2_df” PySpark DataFrame with one row. Append one row to the previous JSON file by specifying the mode as “append”.

JSON File:

We can see the partitioned JSON files. The first file holds the first DataFrame records and the second file holds the second DataFrame record.

Conclusion

There are three different ways to convert the PySpark DataFrame to JSON. First, we discussed the to_json() method which converts to JSON by converting the PySpark DataFrame to the Pandas DataFrame with different examples by considering different parameters. Next, we utilized the toJSON() method. Lastly, we learned how to use the write.json() function to write the PySpark DataFrame to JSON. Appending, and overwriting are possible with this function.