Friday, January 30, 2015

Installing SpatialHadoop on an existing Hadoop cluster

I occasionally get a question about how to install SpatialHadoop on an existing cluster that runs Hadoop. So, decided to write this blog post to describe the different ways to setup SpatialHadoop on an existing cluster.
In this blog post, I'll describe two techniques to install SpatialHadoop on an existing cluster. The first techniques requires an administrator access to Hadoop, not necessarily to the while system. The second technique is less efficient but can work even if you cannot restart the cluster or manage it.

The first techniques

In this technique, all you need to do is extract the binaries of SpatialHadoop on every node in your cluster. This technique is only tested with Hadoop 1.x but it can also with with Hadoop 2.x, at least in concept. The binary archive of SpatialHadoop matches this of an Apache Hadoop 1.x installation. Basically, it installs the required libraries in the lib folder. Once the required libraries are in place on all machines, you need to restart the cluster to ensure that the libraries are loaded. After that, your cluster is ready to use.

Hadoop 2.x

Although not officially supported, you can use the same technique to install SpatialHadoop on Apache Hadoop 2.x. To do that, you first need to grab the source code of SpatialHadoop and build the binary package, then you can install it in your Hadoop distribution.
To grab the latest source code
git clone https://github.com/aseldawy/spatialhadoop2.git
ant dist2
The created package can be installed in a similar way on an Apache Hadoop 2.x

The second technique

In this technique, we assume that you don't have administrator access to the cluster so you can't install the libraries in Hadoop nodes or restart the cluster. Therefore, we compile SpatialHadoop libraries along with all required libraries into one jar which you can run using 'hadoop jar' command.
To create that jar, you need to grab the latest source code from github and then create the jar using the ant command.
git clone https://github.com/aseldawy/spatialhadoop2.git
ant emr-jar1
Once you create the jar file, you can run it using the command hadoop jar.
Similarly, if you're going to run the created jar on Hadoop 2.x, you should use the ant target emr-jar2 instead of emr-jar1

46 comments:

  1. Wow that's a wonderfull blog having all details & helpful. Hadoop cluster NJ

    ReplyDelete
  2. Thanks for providing this informative information. it is very useful you may also refer-http://www.s4techno.com/blog/2016/07/11/hadoop-distributed-file-system-hdfs-for-big-data/

    ReplyDelete
  3. Thanks for providing this informative information…..
    You may also refer-
    http://www.s4techno.com/blog/category/hadoop/

    ReplyDelete
  4. http://www.s4techno.com/blog/2016/08/13/installing-a-storm-cluster/

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. This comment has been removed by a blog administrator.

    ReplyDelete
  7. Hello, I have a cluster installed with Spatial hadoop. Synthetic data of rectangles is generated & operations are working fine. But when I try to copy the csv files of real datasets from the Spatial Hadoop website, it doesn't work. Giving errors in all the steps.
    Any body care to list the steps to do this.
    Thanks in anticipation.

    ReplyDelete
    Replies
    1. Did the file get copied successfully to HDFS? What is the error that you get?

      Delete
  8. This comment has been removed by the author.

    ReplyDelete
  9. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work, Amazon Web Services (AWS) is a dynamic, growing business unit within Amazon.com. AWS is designed to allow application providers, ISVs, and vendors to quickly and securely host your applications. Credo Systemz provides the best AWS Training to get your certifications easily.


    aws training in velachery|aws training in OMR|aws certification training in Chennai|aws course in chennai what is the qualification

    ReplyDelete
  10. Hey, would you mind if I share your blog with my twitter group? There’s a lot of folks that I think would enjoy your content. Please let me know. Thank you.
    Java Training in Chennai | J2EE Training in Chennai | Advanced Java Training in Chennai | Core Java Training in Chennai | Java Training institute in Chennai

    ReplyDelete
  11. Very impressive post,thanks for sharing.Very clear and good content.Keep posting more.
    Data science training institute in btm layout

    ReplyDelete
  12. Fabulous post, you have denoted out some fantastic points, I likewise think this s a very wonderful website. I will visit again for more quality contents and also, recommend this site to all. Thanks. i need some more info
    Ai & Artificial Intelligence Course in Chennai
    PHP Training in Chennai
    Ethical Hacking Course in Chennai Blue Prism Training in Chennai
    UiPath Training in Chennai

    ReplyDelete
  13. Thanks For Sharing The Information The information shared Is Very Valuable Please Keep Updating Us Time just went On reading The article
    Java Training in Chennai

    Java Training in Velachery

    Java Training inTambaram

    Java Training in Porur

    Java Training in Omr

    Java Training in Annanagar

    ReplyDelete
  14. very informative blogs posting thanks you so much sharing me information.
    German language course In gurgaon

    ReplyDelete
  15. This is just the information I am finding everywhere. Thanks for your blog, I just subscribe your blog. This is a nice blog.
    Zirconia Crown Manufacturer In India

    ReplyDelete
  16. very informative blogs posting thanks you so much sharing me information.

    GERMAN LANGUAGE COURSE IN DELHI

    ReplyDelete
  17. This post is so interactive and informative.keep update more information...
    Tally Course in Velachery
    Tally course in Chennai

    ReplyDelete
  18. This post is so attractive and informative.keep updating more information...
    Full Stack Programmer
    Full Stack Certification

    ReplyDelete
  19. thanks for sharing valid information and a blog post.

    Study In Germany

    ReplyDelete
  20. Your article completely stunned me. Also visit our link for best zirconia crowns in india

    ReplyDelete
  21. thanks for sharing a valid Information and a blog post.


    German Language Institute In Chandigarh

    ReplyDelete
  22. Hotspot Shield Elite 5.20.11 Latest Release is a free tool with which we will be safe and completely anonymous on the Internet. This IP hider and changer. Hotspot Shield License Key

    ReplyDelete
  23. days ago — Sketchup pro 2023 Free Download is a subtle three dimensional building programmer designed for a large vary of sketching applications.Sketchup Crack For Windows

    ReplyDelete