Spark-Shell Startup Errors

I have downloaded Spark binaries for Windows and tried running spark-shell batch file and ran into some issues. Following are the issues and the respective solutions:

Problem 1

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
at org.apache.hadoop.util.Shell.(Shell.java:387)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2327)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:365)
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:105)

Solution

Download the winutils.exe from http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe and copy it to some directory say c:/hadoop/bin

Define the environmental variable HADOOP_HOME as C:/hadoop

Problem 2

ERROR SparkContext: Error initializing SparkContext.
java.lang.AssertionError: assertion failed: Expected hostname
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.util.Utils$.checkHost(Utils.scala:931)
at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:31)

Solution

Set environmental variable SPARK_LOCAL_HOSTNAME as localhost

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s