Apache Spark – Page 7

SQL on Hadoop with Hive, Spark & PySpark on EMR & AWS Glue

SPARK Dataframe Alias AS

ALIAS is defined in order to make columns or tables name more readable or even shorter. If you wish to rename your columns while displaying it to the user or if you are using tables in joins then you may […]

Apache SparkBy Raj3 comments

Spark Dataframe Select

In this post, we will see how to fetch data from HIVE table into SPARK DataFrame and perform few SQL like “SELECT” operations on it. I have a table in HIVE database which has details of all the US Presidents […]

Apache SparkBy Raj1 comment

Raj on PySpark-How to Generate MD5 of entire row with columnsMarch 9, 2023
Done. Please check now.
Anand on PySpark-How to Generate MD5 of entire row with columnsFebruary 25, 2023
can you please make the video available to learn
Raj on Free Online SQL to PySpark ConverterAugust 9, 2022
Thank you for sharing this. I will give it a try as well.
John K-W on Free Online SQL to PySpark ConverterAugust 8, 2022
Might be interesting to add a PySpark dialect to SQLglot https://github.com/tobymao/sqlglot https://github.com/tobymao/sqlglot/tree/main/sqlglot/dialects
Meena M on Spark Dataframe WHEN caseJuly 28, 2022
try something like df.withColumn("type", when(col("flag1"), lit("type_1")).when(!col("flag1") && (col("flag2") || col("flag3") || col("flag4") || col("flag5")), lit("type2")).otherwise(lit("other")))