Category: PySpark

In the previous post, we saw many common conversions from SQL to Dataframe in PySpark. In this post, we will see the strategy which you can follow to convert typical SQL query to dataframe in PySpark. If you have not […]

In this post, we will see how you can read parquet files using pyspark and will also see common options and challenges which you must consider while reading or writing parquet files. What is Parquet File Format ? Parquet is […]

Requirement: To change column names to upper case or lower case in PySpark Create a dummy dataframe Convert column names to uppercase in PySpark You can use “withColumnRenamed” function in FOR loop to change all the columns in PySpark dataframe […]

This is second part of PySpark Tutorial series. In this post, we will talk about : In the previous post, we covered following points and if you haven’t read it I will strongly recommend to read it first. Let’s begin […]

Introduction PySpark is becoming obvious choice for the enterprises when it comes to moving to Spark. As per my understanding , this is primarily for 2 reasons : The developers are people like me who are expert in SQL but […]