How to select SPARK2 as default spark version

Spark1 in HDP

Hi Guys. I have been using HDP2.5 for sometime now and few of my friends asked me that how can they select SPARK2 by default. In HDP2.5 we have Spark1.X & Spark2 both available. However when you will start SPARK-SHELL, it will show you a prompt and will select SPARK1.X as default.

Spark1 in HDP
Spark1 in HDP

The answer to the question is present in the prompt itself. You can see it displays on screen that SPARK_MAJOR_VERSION is not set hence taking SPARK1 as default. All you have to do is set this parameter before calling SPARK-SHELL and it will select proper SPARK version. Run below command before calling spark-shell
export SPARK_MAJOR_VERSION=2
Now call SPARK-SHELL and you can see that SPARK2 is selected.

Spark-2 in HDP
Spark-2 in HDP

There are other ways too like you can call the spark by giving absolute path in which you can select SPARK2 in place of SPARK. However I prefer this method. Hope this helps.

Leave a Reply

Your email address will not be published. Required fields are marked *