UDAF stands for User Defined Aggregate functions. Aggregate functions are used to perform a calculation on a set of values and return a single value. It is difficult to write an aggregate function compared to writing a User Defined Functions(UDF) as we need to aggregate on multiple rows and columns. Apache Spark UDAF operates on more than one row or Column while returning a single value results
With the introduction of Apache Hive 3, Apache Hadoop has introduced different new features to address the growing needs of enterprise data warehouse systems. This blog post talks about several…
Apache Hadoop 3 incorporated a number of enhancements over the Hadoop-2.x. We will talk about the important enhancement that was implemented as part of Hadoop 3 over Hadoop 2 in…
An Option is a data type in Scala that indicates the presence or absence of some data. It can be an instance of either case class called Some or Singleton object called None.An instance of Some can store data of any type . An instance of None object represents absence of data.
Metadata is the information that describes other data or, simply, it is data about the data. It is the descriptive, administrative, and structural data that defines a firm’s data assets. It…
Managing metadata is an integral part of the overall data governance standard. An efficient way to do this is to establish data stewardship for metadata. It will ensure that data…
An exit code, or sometimes known as a return code, is the number between 0 and 255 returned to a parent process by an executable or command when it returns control to its parent process. We use the special variable called $? in/Linux and Unix bash scripts to check the exit code.
A Case class in Scala programming language is a class with a case modifier. It is used to conveniently store and match the contents of a class. case class Person…
date command in Linux can be used to display current time and date.It is one of the most useful tool when working with bash script as it is available on all Linux Systems.It allows you to display and configure the current date and time in different format as needed.
Vi editor, which is short for Visual Editor, is the default Text Editor that comes with majority of Linux distributions.This blog post will show you the most useful Vi Text Editor Commands.