Value for Hadoop_Conf_Dir from Cluster
I Have Setup a Cluster(Yarn) Using Ambari with 3 Vms as Hosts. Where I Can Find the Value for Hadoop_Conf_Dir? # Run on a Yarn Cluster Export...
I have setup a cluster(YARN) using Ambari with 3 VMs as hosts.
Where I can find the value for HADOOP_CONF_DIR ?
# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-cluster \ # can also be `yarn-client` for client mode
--executor-memory 20G \
--num-executors 50 \
/path/to/examples.jar \
1000
2 Answers
Install Hadoop as well. In my case I've installed it in /usr/local/hadoop
Setup Hadoop Environment Variables
export HADOOP_INSTALL=/usr/local/hadoop
Then set the conf directory
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
From /etc/spark/conf/spark-env.sh:
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}
David Miller brings 15 years of experience in global economics, personal finance strategy, and market dynamics. He specializes in turning complex economic trends into actionable insights for everyday readers.
