Spark is distributed data processing engine which relies a lot on memory available for computation. Also if you have worked on spark, then you must… Read More »Spark Performance Tuning with help of Spark UI
If you migrate from any RDBMS platform to another, one technical challenge you may face is different Decimal Rounding on both the platforms. I was… Read More »Problem with Decimal Rounding & solution
Recently, I was working on one project where the ETL requirement was to have daily snapshot of the table. It was 15+ years old data… Read More »Never run INSERT OVERWRITE again – try Hadoop Distcp
You may need to add new columns in the existing SPARK dataframe as per the requirement. This new column can be initialized with a default… Read More »Spark Dataframe add multiple columns with value
Repartition is the process of movement of data on the basis of some column or expression or random into required number of partitions. This depends… Read More »Spark Dataframe Repartition
Spark dataframe add row number is very common requirement especially if you are working on ELT in Spark. You can use monotonically_increasing_id method to generate… Read More »Spark Dataframe – monotonically_increasing_id