Setting up Pigeon on Pig and Hadoop

From Pig to Pigeon

Pigeon

Pig is a framework that allows developers to express their MapReduce programs in a nice and easy-to-use high level language, termed Pig Latin. Pigeon builds on top of that by providing a set of user-defined functions (UDFs) that can manipulate spatial data. In this blog post, I'll describe in a easy steps how to install and run Pigeon on an existing Hadoop cluster running Pig.

Preliminaries

Before you work with Pigeon, you need to have an installed Hadoop distribution with Pig configured to run with it. You can still use Pigeon on a single machine running Pig in local mode, however, it is supposed to be for testing only.
How to set up Hadoop
How to set up Pig

Install Pigeon

Pigeon ships as a set of JARs that can be easily integrated with your existing Pig Latin scripts. The primary JAR file that contains Pigeon UDFs can be downloaded here.
You will also need third part JAR files that are required by Pigeon. You need the JTS and ESRI Geometry API files. If  you are running Pigeon on SpatialHadoop, you would already have these JAR files. Otherwise, you can download them here and here.
And that's everything. You're ready to go with your first example now.

Example

In this example, we will process a dataset of sports areas around the world, extracted from OpenStreetMap, and compute the minimum bounding rectangle (MBR) around each object. The dataset can be download from this direct link or at the SpatialHadoop datasets page.

1. Download the file to your local dirve:
   wget https://s3.amazonaws.com/planet-datasets/sports.bz2/sports.bz2
2. Start Pig in local mode
   Pig -x local
3. Load the JAR files
   REGISTER pigeon-0.2.1.jar;
   REGISTER jts-1.8.jar;
   REGISTER esri-geometry-api-1.2.1.jar;
4. Load the dataset
   sports = LOAD 'sports.bz2' AS (id: int, geom, tags: map[chararray]);
5. Calculate the envelope (MBR)
   sports_mbr = FOREACH sports GENERATE id, edu.umn.cs.pigeon.Envelope(geom) AS geom_mbr, tags;
6. Store the result back to a file
   STORE sports_mbr INTO 'sports_mbr';

Use Short Function Names

To avoid writing the full function name (package + class name), you can use the 'pigeon_import.pig' file that creates a short name for all functions. You can download the file here and place it next to your Pigeon script. To load this file into your script, issue the following line to the beginning of your script.
IMPORT 'pigeon_import.pig';
After that, you can write the envelope function as ST_Envelope instead of edu.umn.cs.pigeon.Envelope

Compile Pigeon from source

You can also compile Pigeon from source in case you want to get the latest updates. Pigeon uses Maven to manage its source and dependencies.

1. Download the source code using git.
   git clone https://github.com/aseldawy/pigeon.git
   Or download a ZIP archive with the source code at
   https://github.com/aseldawy/pigeon/archive/master.zip
2. Compile the source code by issuing the command
   mvn package
3. The generated JAR file will be located under 'target/' directory.

Further Resources

• Around the world in one hour!