How to check Spark run logs in EMR
Situation: Someone in my team has executed Spark application in EMR and the job failed. The user is new to EMR and does not have… Read More »How to check Spark run logs in EMR
Amazon EMR
Run Spark Job in existing EMR using AIRFLOW
In this post, we will see how you can run Spark application on existing EMR cluster using Apache Airflow. The most basic way of scheduling… Read More »Run Spark Job in existing EMR using AIRFLOW
[EMR] 5 settings for better Spark environment
I have been working on Spark for many years now. Initially I started with working on on-premises Hadoop cluster using CDH or HDP. In the… Read More »[EMR] 5 settings for better Spark environment
Namenode is in safe mode – Hadoop
The most common reason for namenode to go into safemode is due to under-replicated blocks. This is generally caused by storage issues on hdfs or… Read More »Namenode is in safe mode – Hadoop
EMR – No space left on device [Solved]
I was recently working on EMR running some pyspark jobs and I encountered "No space left on device" error. Now the error seems to be… Read More »EMR – No space left on device [Solved]
