Spark Performance Tuning with help of Spark UI
Spark is distributed data processing engine which relies a lot on memory available for computation. Also if you have worked on spark, then you must have faced job/task/stage failures due … Read More
Category: Apache Spark
Problem with Decimal Rounding & solution
If you migrate from any RDBMS platform to another, one technical challenge you may face is different Decimal Rounding on both the platforms. I was recently working for a client … Read More
Never run INSERT OVERWRITE again – try Hadoop Distcp
Recently, I was working on one project where the ETL requirement was to have daily snapshot of the table. It was 15+ years old data model on which datawarehouse was … Read More
Spark Dataframe add multiple columns with value
You may need to add new columns in the existing SPARK dataframe as per the requirement. This new column can be initialized with a default value or you can assign … Read More
Spark Dataframe Repartition
Repartition is the process of movement of data on the basis of some column or expression or random into required number of partitions. This depends on the kind of value/s … Read More
Spark Dataframe – monotonically_increasing_id
Spark dataframe add row number is very common requirement especially if you are working on ELT in Spark. You can use monotonically_increasing_id method to generate incremental numbers. However the numbers … Read More
Spark Dataframe NULL values
In this post, we will see how to Handle NULL values in any given dataframe. Many people confuse it with BLANK or empty string however there is a difference. NULL … Read More
Spark Dataframe – Explode
In Spark, we can use “explode” method to convert single column values into multiple rows. Recently I was working on a task to convert Cobol VSAM file which often has … Read More
Spark Dataframe SHOW
In Spark Dataframe, SHOW method is used to display Dataframe records in readable tabular format. This method is used very often to check how the content inside Dataframe looks like. … Read More
Spark Dataframe Column list
Recently I was working on a task where I wanted Spark Dataframe Column List in a variable. This was required to do further processing depending on some technical columns present … Read More