Apache Spark

We all know that UPDATING column value in a table is a pain in HIVE or SPARK SQL especially if you are dealing with non-ACID tables. However in Dataframe you can easily update column values. You can update a dataframe column value with value from another dataframe. For this purpose, we have to use JOINS […]

JOIN is used to retrieve data from two tables or dataframes. You will need “n” Join functions to fetch data from “n+1” dataframes. In order to join 2 dataframe you have to use “JOIN” function which requires 3 inputs – dataframe to join with, columns on which you want to join and type of join

Spark DISTINCT or spark drop duplicates is used to remove duplicate rows in the Dataframe. Row consists of columns, if you are selecting only one column then output will be unique values for that specific column. DISTINCT is very commonly used to identify possible values which exists in the dataframe for any given column. Spark

Recently I have received few queries regarding the query which we are passing to “load” function when using JDBC connection to connect to any RDBMS. The question is whether that query should be Spark SQL compliant or should be RDBMS specific. This is actually a very valid question because Spark SQL does not support all

In this post, we will see how to connect to 3 very popular RDBMS using Spark. We will create connection and will fetch some records via spark. The dataframe will hold data and we can use it as per requirement. We will talk about JAR files required for connection and JDBC connection string to fetch