In Spark, groupBy aggregate functions are used to group multiple rows into one and calculate measures by applying functions like MAX,SUM,COUNT etc. In Spark , you can perform aggregate operations on dataframe. This is similar to what we have in SQL like MAX, MIN, SUM etc. We can also perform aggregation on some specific columns…
PySpark Date Functions
In PySpark, you can do almost all the date operations you can think of using in-built functions. Let’s quickly jump to example and see it one by one. Create a dataframe with sample date values: Check dataframe info Now the problem I see here is that columns start_dt & end_dt are of type string and…
Spark Dataframe Update Column Value
We all know that UPDATING column value in a table is a pain in HIVE or SPARK SQL especially if you are dealing with non-ACID tables. However in Dataframe you can easily update column values. You can update a dataframe column value with value from another dataframe. For this purpose, we have to use JOINS…
Spark Dataframe JOINS – Only post you need to read
JOIN is used to retrieve data from two tables or dataframes. You will need “n” Join functions to fetch data from “n+1” dataframes. In order to join 2 dataframe you have to use “JOIN” function which requires 3 inputs – dataframe to join with, columns on which you want to join and type of join…
Hive Transactional Tables: Limitations and Considerations (Part 2)
In the previous post, we discussed about HIVE transactional tables; how to create it, properties and configurations required and example of HIVE transactional table DDL with INSERT/UPDATE/DELETE/MERGE statements on HIVE ACID table. In this post, we will talk about limitations with HIVE transactional tables and possible solutions if any. [Error 10302]: Updating values of bucketing…
