Installation Steps for Spark Tools (DEPRECATED)

Java

Install a Java Development Kit according to your operating system.

Sample install command for Debian, Ubuntu, etc.

sudo apt-get install openjdk-8-jdk

Sample install command for Fedora, Oracle Linux, Red Hat Enterprise Linux, etc.

su -c "yum install java-1.8.0-openjdk-devel"

Sample install commands for MacOS

conda install -c anaconda openjdk

Hadoop

Download the Hadoop binary file:

curl -L --retry 3 -o /tmp/hadoop-2.7.3.tar.gz \
    "https://archive.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz"

Put the Hadoop folder inside the /usr/local/ directory:

sudo tar -xzf /tmp/hadoop-2.7.3.tar.gz -C /usr/local/
cd /usr/local/
sudo mv hadoop-2.7.3/ hadoop/
rm -rf /tmp/hadoop-2.7.3.tar.gz

Spark

Download the Spark binary file:

curl -L --retry 3 -o /tmp/spark-2.4.0-bin-hadoop2.7.tgz \
    "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"

Put the Spark folder inside the /usr/local/ directory:

sudo tar -xf /tmp/spark-2.4.0-bin-hadoop2.7.tgz -C /usr/local/
cd /usr/local/
sudo mv spark-2.4.0-bin-hadoop2.7/ spark/
rm -rf /tmp/spark-2.4.0-bin-hadoop2.7.tgz

Livy

Download the Livy binary file:

curl -L --retry 3 -o /tmp/livy-0.5.0-incubating-bin.zip \
    "https://www-us.apache.org/dist/incubator/livy/0.5.0-incubating/livy-0.5.0-incubating-bin.zip"

Put the Livy folder in the home folder:

unzip -q /tmp/livy-0.5.0-incubating-bin.zip -d ~/
mv ~/livy-0.5.0-incubating-bin ~/livy
mkdir -p ~/livy/logs
rm -rf /tmp/livy-0.5.0-incubating-bin.zip