What is scientific notation or exponent number ? Recently I was working on PySpark process in which requirement was to apply some aggregation on big numbers. The result in output was accurate however it was in exponential format or scientific notation which definitely does not look ok in display. I am talking about numbers which […]
PySpark handle scientific number Read More »
In the previous post we saw how to create and run a very basic pyspark script in Hadoop environment. In this post, we will walkthrough a pyspark script template in detail. We will see different options while creating a pyspark script and also how to run a pyspark script with multiple configurations. In this post,
PySpark script example and how to run pyspark script Read More »
I have been working on Spark for many years now. Initially I started with working on on-premises Hadoop cluster using CDH or HDP. In the past few years, I have been working a lot on EMR primarily for Spark or PySpark tasks. In this post, I would like to share some of the general settings
[EMR] 5 settings for better Spark environment Read More »
In this post, we will see how you can create your first PySpark script and then run it in batch mode. Many people I have seen use notebooks like Jupyter, Zeppelin however you may want to create pyspark script and run it as per schedule. This is especially helpful if you want to run ETL
Your first PySpark Script – Create and Run Read More »
PySpark Filter is used to specify conditions and only the rows that satisfies those conditions are returned in the output. You can use WHERE or FILTER function in PySpark to apply conditional checks on the input rows and only the rows that pass all the mentioned checks will move to output result set. PySpark WHERE
PySpark Filter – 25 examples to teach you everything Read More »