I was recently working on a task where I have to read more than a Terabyte of data spread across multiple parquet files. Also some… Read More »Spark Case Study – optimise executor memory and cores per executor
The most common reason for namenode to go into safemode is due to under-replicated blocks. This is generally caused by storage issues on hdfs or… Read More »Namenode is in safe mode – Hadoop
This is second part of PySpark Tutorial series. In this post, we will talk about : Fetch unique values from dataframe in PySpark Use Filter… Read More »PySpark Tutorial – Day 2
PySpark is becoming obvious choice for the enterprises when it comes to moving to Spark. As per my understanding , this is primarily for 2… Read More »PySpark Tutorial – Day 1
I was recently working on a project to migrate some records from on-premises data warehouse to S3. The requirement was also to run MD5 check… Read More »PySpark-How to Generate MD5 of entire row with columns
I was recently working on EMR running some pyspark jobs and I encountered "No space left on device" error. Now the error seems to be… Read More »EMR – No space left on device [Solved]
