Spark：v2.4
一、基础软件
1. R-lang
sudo yum -y install epel-release
sudo yum install R
注意：如果Java没有安装，会附带着安装OpenJDK。
2. Scala
wget https://downloads.lightbend.com/scala/2.12.8/scala-2.12.8.tgz
tar xvf scala-2.12.8.tgz
sudo mv scala-2.12.8 /usr/lib
sudo ln -s /usr/lib/scala-2.12.8 /usr/lib/scala
1) add path
$vi ~/.bashrc
PATH=$PATH:/usr/lib/scala/bin
二、Spark
1. down & unzip
mkdir -p ~/app/spark
tar zxvf spark-2.4.1-bin-hadoop2.7.tgz -C ~/app/spark
mv ~/app/spark/spark-2.4.1-bin-hadoop2.7 ~/app/spark/2.4.1
2. test
./bin/spark-shell --master local[2]
input:
for (i <- 1 to 3; j <- 1 to 3 if i != j) print(10 * i + j + "\t")
input:(exit)
:q
三、开启服务
1. 添加系统变量
echo 'export SPARK_HOME=$HOME/app/spark/2.4.1' >> .bash_profile
echo 'export PATH=$PATH:$SPARK_HOME/bin' >> .bash_profile
$source ~/.bash_profile
2. open port
sudo firewall-cmd --permanent --zone=public --add-port=6066/tcp
sudo firewall-cmd --permanent --zone=public --add-port=7077/tcp
sudo firewall-cmd --permanent --zone=public --add-port=8080-8088/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --reload
3. exec
$SPARK_HOME/sbin/start-master.sh
http://192.168.42.101:8080/
