PySpark -Convert SQL queries to Dataframe
In PySpark, you can run dataframe commands or if you are comfortable with SQL then you can run SQL queries too. In this post, we will see how to run … Read More
Category: PySpark
PySpark RDD operations – Map, Filter, SortBy, reduceByKey, Joins
In the last post, we discussed about basic operations on RDD in PySpark. In this post, we will see other common operations one can perform on RDD in PySpark. Let’s … Read More
Basic RDD operations in PySpark
A Resilient Distributed Dataset (RDD) is the basic abstraction in Spark. In other words, we can say it is the most common structure that holds data in Spark. RDD is … Read More
PySpark Date Functions
In PySpark, you can do almost all the date operations you can think of using in-built functions. Let’s quickly jump to example and see it one by one. Create a … Read More
Teradata to PySpark – Replicate ACTIVITYCOUNT to Spark
Recently I was working on a project to convert Teradata BTEQ to PySpark code. Since it was mostly SQL queries, we were asked to typically transform into Spark SQL and … Read More
PySpark – zipWithIndex Example
One of the most common operation in any DATA Analytics environment is to generate sequences. There are multiple ways of generating SEQUENCE numbers however I find zipWithIndex as the best … Read More