Pig Quick notes

PIG QUICK NOTES:
Pig latin – is the language used to analyze data in Hadoop using Apache Pig.
A RELATION is outermost structure of Pig Latin data model. and it is bag where-
-A bag is collection of Tuples
-A tuple is an ordered set of fields
-A field is a piece of data

Pig Latin –Statements
While processing data using Pig Latin, statements are the basic constructs.
1. These statements work with relations. They include expressions and schemas.
2. Every statement ends with a semicolon (;).
3. We will perform various operations using operators provided by Pig Latin, through
statements.
4. Except LOAD and STORE, while performing all other operations, Pig Latin
statements take a relation as input and produce another relation as output.
5. As soon as you enter a Load statement in the Grunt shell, its semantic checking
will be carried out. To see the contents of the schema, you need to use the Dump
operator. Only after performing the dump operation, the MapReduce job for
loading the data into the file system will be carried out.

load statement ex-

student_data = LOAD ‘/home/student.txt’ USING PigStorage(‘,’) AS  (id:int, firstname:chararray, lastname:chararray, phone:int, city:chararray);

DATA TYPES:

Complex datatype

Loading and Storing

Combining and Splitting

Load Operator:

You can load data into Apache Pig from the file system (HDFS/ Local) using LOAD operator of Pig Latin.

Syntax

The load statement consists of two parts divided by the “=” operator. On the left-hand side, we need to mention the name of the relation where we want to store the data, and on the right-hand side, we have to define how we store the data. Given below is the syntax of the Load operator.

Relation_name = LOAD ‘Input file path’ USING function as schema;

Where,

–relation_name – We have to mention the relation in which we want to store the data.

– Input file path – We have to mention the HDFS directory where the file is stored. (In MapReduce mode)

function – We have to choose a function from the set of load functions provided by Apache Pig (BinStorage, JsonLoader, PigStorage, TextLoader). Or, we can define our own load function.

–Schema – We have to define the schema of the data. We can define the required schema as follows:

(column1 : data type, column2 : data type, column3 : data type);

Note: We load the data without specifying the schema. In that case, the columns will be addressed as $01, $02, etc… (check).

grunt> student = LOAD ‘hdfs://localhost:9000/pig_data/student_data.txt’ USING PigStorage(‘,’)as ( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );

Storing Data:

STORE relation_name INTO ‘required path’ [Using function];

Grunt> STORE student into ‘ hdfs://localhost:9000/pig_Output/ ‘ USING PigStorage(‘,’);

Diagnostic Operator:

DUMP : used to print the  relation name.

DESCRIBE : used to view the schema of a relation.

                                DESCRIBE student:

                                grunt> student: { id: int,firstname: chararray,lastname: chararray,phone: chararray,city: chararray }