Hi Guys. I have been using HDP2.5 for sometime now and few of my friends asked me that how can they select SPARK2 by default. In HDP2.5 we have Spark1.X & Spark2 both available. However when you will start SPARK-SHELL, it will show you a prompt and will select SPARK1.X as default. The answer to […]

Spark concatenate is used to merge two or more string into one string. In many scenarios, you may want to concatenate multiple strings into one. For example, you may want to concatenate “FIRST NAME” & “LAST NAME” of a customer to show his “FULL NAME”. In Spark SQL Dataframe, we can use concat function to

Replace String – TRANSLATE & REGEXP_REPLACE It is very common sql operation to replace a character in a string with other character or you may want to replace string with other string . This is possible in Spark SQL Dataframe easily using regexp_replace or translate function. Spark TRANSLATE function If we want to replace any

SORT is used to order resultset on the basis of values for any selected column. The syntax is to use sort function with column name inside it. We can also specify asending or descending order for sorting, default is ascending. In our dataframe, if we want to order the resultset on the basis of the

In SQL, if we have to check multiple conditions for any column value then we use case statement. In Spark SQL dataframes also we can replicate same functionality by using WHEN clause multiple times, once for each conditional check. No requirement to add CASE keyword though. So let’s see an example to see how to