Ufdjrw

What exactly is the purpose of connection links straped between the rocket and the launch pad

Making a sword in the stone, in a medieval world without magic

Replacing Windows 7 security updates with anti-virus?

Time travel short story where dinosaur doesn't taste like chicken

What is the difference between "shut" and "close"?

What has been your most complicated TikZ drawing?

"One can do his homework in the library"

When were linguistics departments first established

Force user to remove USB token

How can I discourage/prevent PCs from using door choke-points?

What Happens when Passenger Refuses to Fly Boeing 737 Max?

what does the apostrophe mean in this notation?

Is King K. Rool's down throw to up-special a true combo?

It's a yearly task, alright

How could a female member of a species produce eggs unto death?

If the Captain's screens are out, does he switch seats with the co-pilot?

Potentiometer like component

Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?

When two POV characters meet

Best approach to update all entries in a list that is paginated?

Good allowance savings plan?

What happens with multiple copies of Humility and Glorious Anthem on the battlefield?

Provisioning profile doesn't include the application-identifier and keychain-access-groups entitlements

Straight line with arrows and dots

Scala RDD piping to Python Code throws IllegalStateException on Linux Server while locally on windows runs fine

0

I am trying to execute Python code inside Scala program .Everything works fine locally on windows machine, but when I created jar and moved to server its throwing this error

java.lang.IllegalStateException: Subprocess exited with status 2. Command 
ran: python /home/folder2/scores.py someusername password /folder1

I followed this question:

Piping Scala RDD to Python code fails

but still its throwing same error.

This is my code from where I am calling Python Script:

import org.apache.spark.SparkFiles
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession


class Test(spark: SparkSession) 

 val sparkContext = spark.sparkContext

 def runPython = 

 val path = "/folder1"
 val userName = "someusername"
 val password = "password"
 val params = s"$userName $password $path
 val rdd = sparkContext.parallelize(params)
 val pythonFile = "/home/folder2/scores.py"
 val command = s"python $pythonFile $params"
 val res = rdd.pipe(command)
 res.collect()

python scala apache-spark rdd

share|improve this question

edited Mar 7 at 2:47

Jimmy Maguel

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117

add a comment | 

0

I am trying to execute Python code inside Scala program .Everything works fine locally on windows machine, but when I created jar and moved to server its throwing this error

java.lang.IllegalStateException: Subprocess exited with status 2. Command 
ran: python /home/folder2/scores.py someusername password /folder1

I followed this question:

Piping Scala RDD to Python code fails

but still its throwing same error.

This is my code from where I am calling Python Script:

import org.apache.spark.SparkFiles
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession


class Test(spark: SparkSession) 

 val sparkContext = spark.sparkContext

 def runPython = 

 val path = "/folder1"
 val userName = "someusername"
 val password = "password"
 val params = s"$userName $password $path
 val rdd = sparkContext.parallelize(params)
 val pythonFile = "/home/folder2/scores.py"
 val command = s"python $pythonFile $params"
 val res = rdd.pipe(command)
 res.collect()

python scala apache-spark rdd

share|improve this question

edited Mar 7 at 2:47

Jimmy Maguel

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117

add a comment | 

I am trying to execute Python code inside Scala program .Everything works fine locally on windows machine, but when I created jar and moved to server its throwing this error

java.lang.IllegalStateException: Subprocess exited with status 2. Command 
ran: python /home/folder2/scores.py someusername password /folder1

I followed this question:

Piping Scala RDD to Python code fails

but still its throwing same error.

This is my code from where I am calling Python Script:

import org.apache.spark.SparkFiles
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession


class Test(spark: SparkSession) 

 val sparkContext = spark.sparkContext

 def runPython = 

 val path = "/folder1"
 val userName = "someusername"
 val password = "password"
 val params = s"$userName $password $path
 val rdd = sparkContext.parallelize(params)
 val pythonFile = "/home/folder2/scores.py"
 val command = s"python $pythonFile $params"
 val res = rdd.pipe(command)
 res.collect()

python scala apache-spark rdd

share|improve this question

edited Mar 7 at 2:47

Jimmy Maguel

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117

java.lang.IllegalStateException: Subprocess exited with status 2. Command 
ran: python /home/folder2/scores.py someusername password /folder1

import org.apache.spark.SparkFiles
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession


class Test(spark: SparkSession) 

 val sparkContext = spark.sparkContext

 def runPython = 

 val path = "/folder1"
 val userName = "someusername"
 val password = "password"
 val params = s"$userName $password $path
 val rdd = sparkContext.parallelize(params)
 val pythonFile = "/home/folder2/scores.py"
 val command = s"python $pythonFile $params"
 val res = rdd.pipe(command)
 res.collect()

share|improve this question

edited Mar 7 at 2:47

Jimmy Maguel

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117

share|improve this question

edited Mar 7 at 2:47

Jimmy Maguel

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117

asked Mar 6 at 17:41

Jimmy MaguelJimmy Maguel

117