PySpark

In the previous post, we saw many common conversions from SQL to Dataframe in PySpark. In this post, we will see the strategy which you can follow to convert typical SQL query to dataframe in PySpark. If you have not checked previous post, I will strongly recommend to do it as we will refer to […]

In this post, we will see how you can read parquet files using pyspark and will also see common options and challenges which you must consider while reading or writing parquet files. What is Parquet File Format ? Parquet is a columnar file format and is becoming very popular because of the optimisations it brings

Requirement: To change column names to upper case or lower case in PySpark Create a dummy dataframe Convert column names to uppercase in PySpark You can use “withColumnRenamed” function in FOR loop to change all the columns in PySpark dataframe to uppercase by using “upper” function. Convert column names to lowercase in PySpark You can

This is second part of PySpark Tutorial series. In this post, we will talk about : Fetch unique values from dataframe in PySpark Use Filter to select few records from Dataframe in PySpark AND OR LIKE IN BETWEEN NULL How to SORT data on basis of one or more columns in ascending or descending order.

Introduction PySpark is becoming obvious choice for the enterprises when it comes to moving to Spark. As per my understanding , this is primarily for 2 reasons : The developers are people like me who are expert in SQL but not in programming language like Java, C#. So when it comes to picking one language