Spark is distributed data processing engine which relies a lot on memory available for computation. Also if you have worked on spark, then you must… Read More »Spark Performance Tuning with help of Spark UI
In PySpark, you can run dataframe commands or if you are comfortable with SQL then you can run SQL queries too. In this post, we… Read More »PySpark -Convert SQL queries to Dataframe
If you migrate from any RDBMS platform to another, one technical challenge you may face is different Decimal Rounding on both the platforms. I was… Read More »Problem with Decimal Rounding & solution
Recently, I was working on one project where the ETL requirement was to have daily snapshot of the table. It was 15+ years old data… Read More »Never run INSERT OVERWRITE again – try Hadoop Distcp
If you are working on Hadoop or any other platform and storing structured data, I am sure you must have heard about columnar storage types.… Read More »Columnar Storage & why you must use it
In the last post, we discussed about basic operations on RDD in PySpark. In this post, we will see other common operations one can perform… Read More »PySpark RDD operations – Map, Filter, SortBy, reduceByKey, Joins