PySpark -Convert SQL queries to Dataframe
In PySpark, you can run dataframe commands or if you are comfortable with SQL then you can run SQL queries too. In this post, we… Read More »PySpark -Convert SQL queries to Dataframe
PySpark
PySpark RDD operations – Map, Filter, SortBy, reduceByKey, Joins
In the last post, we discussed about basic operations on RDD in PySpark. In this post, we will see other common operations one can perform… Read More »PySpark RDD operations – Map, Filter, SortBy, reduceByKey, Joins
Basic RDD operations in PySpark
A Resilient Distributed Dataset (RDD) is the basic abstraction in Spark. In other words, we can say it is the most common structure that holds… Read More »Basic RDD operations in PySpark
PySpark Date Functions
In PySpark, you can do almost all the date operations you can think of using in-built functions. Let’s quickly jump to example and see it… Read More »PySpark Date Functions
Teradata to PySpark – Replicate ACTIVITYCOUNT to Spark
Recently I was working on a project to convert Teradata BTEQ to PySpark code. Since it was mostly SQL queries, we were asked to typically… Read More »Teradata to PySpark – Replicate ACTIVITYCOUNT to Spark
PySpark – zipWithIndex Example
One of the most common operation in any DATA Analytics environment is to generate sequences. There are multiple ways of generating SEQUENCE numbers however I… Read More »PySpark – zipWithIndex Example