In Spark, groupBy aggregate functions are used to group multiple rows into one and calculate measures by applying functions like MAX,SUM,COUNT etc. In Spark , you can perform aggregate operations on dataframe. This is similar to what we have in SQL like MAX, MIN, SUM etc. We can also perform aggregation on some specific columns […]
Spark Dataframe groupBy Aggregate Functions Read More »
In PySpark, you can do almost all the date operations you can think of using in-built functions. Let’s quickly jump to example and see it one by one. Create a dataframe with sample date values: Check dataframe info Now the problem I see here is that columns start_dt & end_dt are of type string and
PySpark Date Functions Read More »
We all know that UPDATING column value in a table is a pain in HIVE or SPARK SQL especially if you are dealing with non-ACID tables. However in Dataframe you can easily update column values. You can update a dataframe column value with value from another dataframe. For this purpose, we have to use JOINS
Spark Dataframe Update Column Value Read More »
JOIN is used to retrieve data from two tables or dataframes. You will need “n” Join functions to fetch data from “n+1” dataframes. In order to join 2 dataframe you have to use “JOIN” function which requires 3 inputs – dataframe to join with, columns on which you want to join and type of join
Spark Dataframe JOINS – Only post you need to read Read More »
In the previous post, we discussed about HIVE transactional tables; how to create it, properties and configurations required and example of HIVE transactional table DDL with INSERT/UPDATE/DELETE/MERGE statements on HIVE ACID table. In this post, we will talk about limitations with HIVE transactional tables and possible solutions if any. [Error 10302]: Updating values of bucketing
Hive Transactional Tables: Limitations and Considerations (Part 2) Read More »