PySpark SelectExpr()

Using the selectExpr() function in PySpark, we can directly evaluate an expression without creating any TABLE or VIEW. This function is available in the pyspark.sql.DataFrame module which is similar to the select() method. With selectExpr(), we can display the columns, apply the functions on the columns, evaluate the expressions, perform the aggregations operations, etc. It is also possible to evaluate/specify multiple columns at a time.

Pyspark.sql.DataFrame.selectExpr()

The selectexpr() function takes the columns/set of expressions and returns the DataFrame based on the specified expressions/columns. Multiple expressions can be specified in this function which is separated by comma. To display the DataFrame, we can use the show()/collect() functions.

Syntax:

Here, the pyspark_DataFrame_object is the input PySpark DataFrame.

Scenario 1: Select the Columns

In this scenario, we will see how to select the particular columns from the PySpark DataFrame using the selectExpr() function.

The expression that is used is “existing_column as new_name”. Here, the existing_column is the column name that is present in the DataFrame and it is displayed as new_name (Aliasing).

Example:

Create a PySpark DataFrame named “agri_df” with 5 rows and columns. Get the “Soil_status” and “Soil_Type” columns as “STATUS” and “TYPE”.

Output:

Scenario 2: Specifying the Conditional Expressions

In this scenario, we will see how to evaluate the conditions within the selectExpr() function.

The expression that is used is “existing_column operator value”. Here, the existing_column is the column name that is present in the DataFrame and we compare each value in this column with the string/value.

Example 1:

Check whether the country is “USA” or not. The equalto (=) operator is used here.

Output:

Example 2:

Check whether the Soil_Type is NULL or not. The NULL keyword checks whether the value is NULL or not. If it is null, true is returned. Otherwise, false is returned. The final expression is “Soil_Type IS NULL”

Output:

Scenario 3: Evaluating the Expressions

In this scenario, we will see how to specify the mathematical expressions. The expression that is used is “existing_column mathematical_expression”.

Example:

1. Display the actual “Acres” column.
2. Add 100 to the “Acres” column.
3. Subtract 100 from the “Acres” column.
4. Multiply 100 with the “Acres” column.
5. Divide the “Acres” column by 100.

Output:

Scenario 4: Applying the Aggregate Functions

SUM(column_name) – It evaluates the total value in the specified column.

MEAN(column_name) – It evaluates the average value in the specified column.

MIN(column_name) – It returns the minimum element among all elements in the specified column.

MAX(column_name) – It returns the maximum element among all elements in the specified column.

Example:

1. Find the total, average, count, minimum, and maximum elements of “Acres”.
2. Find the minimum and maximum elements in the “Soil_status” column.

Output:

Conclusion

We discussed about the selectExpr() function which takes the columns/sets of expressions and returns the DataFrame based on the specified expressions/columns. As part of this, we learned the four major scenarios in which the selectExpr() is applicable. Multiple expressions can be specified in this function which are separated by comma. There is no need to create a TEMPORARY VIEW to use the selectExpr() function.