AWS Glue Archives | SQL & Hadoop

SQL on Hadoop with Hive, Spark & PySpark on EMR & AWS Glue

AWS Glue create dynamic frame

We can create AWS Glue dynamic frame using data present in S3 or tables that exists in Glue catalog. In addition to that we can create dynamic frames using custom connections as well. In this post, we will create new […]

AWS GlueBy Raj0 comments

AWS Glue read files from S3

You can use aws glue crawler to read file from S3 and create corresponding table in the Glue catalog. In this tutorial we will read few files present in S3 and will create corresponding tables in AWS Glue catalog. We […]

AWS GlueBy Raj0 comments

Raj on PySpark-How to Generate MD5 of entire row with columnsMarch 9, 2023
Done. Please check now.
Anand on PySpark-How to Generate MD5 of entire row with columnsFebruary 25, 2023
can you please make the video available to learn
Raj on Free Online SQL to PySpark ConverterAugust 9, 2022
Thank you for sharing this. I will give it a try as well.
John K-W on Free Online SQL to PySpark ConverterAugust 8, 2022
Might be interesting to add a PySpark dialect to SQLglot https://github.com/tobymao/sqlglot https://github.com/tobymao/sqlglot/tree/main/sqlglot/dialects
Meena M on Spark Dataframe WHEN caseJuly 28, 2022
try something like df.withColumn("type", when(col("flag1"), lit("type_1")).when(!col("flag1") && (col("flag2") || col("flag3") || col("flag4") || col("flag5")), lit("type2")).otherwise(lit("other")))