Category: PySpark

PySpark apply function to column in dataframe to get desired transformation as output. In this post, we will see 2 of the most common ways of applying function to column in PySpark. First is applying spark built-in functions to column […]

What is scientific notation or exponent number ? Recently I was working on PySpark process in which requirement was to apply some aggregation on big numbers. The result in output was accurate however it was in exponential format or scientific […]

In the previous post we saw how to create and run a very basic pyspark script in Hadoop environment. In this post, we will walkthrough a pyspark script template in detail. We will see different options while creating a pyspark […]

In this post, we will see how you can create your first PySpark script and then run it in batch mode. Many people I have seen use notebooks like Jupyter, Zeppelin however you may want to create pyspark script and […]

PySpark Filter is used to specify conditions and only the rows that satisfies those conditions are returned in the output. You can use WHERE or FILTER function in PySpark to apply conditional checks on the input rows and only the […]