Spark Java.lang.verifyerror

I get the following error when I try to call I use python client for the spark. lines = sc.textFile(hdfs://...) lines.take(10) I suspect that spark and hadoop versions might not

Solution 1:

check the pom.xml where you compiled

search for protobuf version. It might solve the problem.

Or the problem might be something else as mentioned in this Jira thread.

https://issues.apache.org/jira/browse/SPARK-7238

Solution 2:

You need to check the py4j jar version that is needed for this hadoop version. Download that and place it in the lib folder of spark installed directory. And check bashrc for path reference. It will fix this error