Ufdjrw

Student evaluations of teaching assistants

Coordinate position not precise

What to do with wrong results in talks?

The plural of 'stomach"

Is it correct to write "is not focus on"?

is this a spam?

Tiptoe or tiphoof? Adjusting words to better fit fantasy races

What is difference between behavior and behaviour

Is HostGator storing my password in plaintext?

Stereotypical names

How will losing mobility of one hand affect my career as a programmer?

What's the purpose of "true" in bash "if sudo true; then"

At which point does a character regain all their Hit Dice?

Your magic is very sketchy

What't the meaning of this extra silence?

(Bedrock Edition) Loading more than six chunks at once

Is expanding the research of a group into machine learning as a PhD student risky?

There is only s̶i̶x̶t̶y one place he can be

Is there any reason not to eat food that's been dropped on the surface of the moon?

How do I rename a LINUX host without needing to reboot for the rename to take effect?

Applicability of Single Responsibility Principle

How does it work when somebody invests in my business?

Failed to fetch jessie backports repository

Opposite of a diet

Spark Streaming textFileStream not picking new files

spark ssc.textFileStream is not streamining new filesSpark Streaming not detecting new HDFS filesSpark streaming does not read files moved from HDFS to HDFSSpark Streaming: java.io.FileNotFoundException: File does not exist: <input_filename>._COPYING_Counting with Spark StreamingSpark Streaming : textFileStream doesn't monitor the folder or stream filesSpark textFileStream not reading filessubmitting spark word count job using hidden REST is creating output hdfs directory with only temporary fileSparkStreaming textFileStreaming doesn't give outputSpark streaming from local file to hdfs. textFileStream

0

I am trying to execute spark streaming word count application using spark-submit command. I started the spark program and when I copy new files to the local folder path, the file is not recognized and no operation is performed. In the log, I find error stating,

19/03/03 17:22:17 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@87b5b49/static/streaming,null,AVAILABLE,@Spark
19/03/03 17:22:17 INFO streaming.StreamingContext: StreamingContext started
19/03/03 17:22:21 WARN dstream.FileInputDStream: Error finding new files java.lang.NullPointerException

I use below command to submit spark job,

spark-submit --class com.company.stream.WordCounter --master local[4] /home/workspace/spark/SparkWordCounter.jar

Below is the source code,

val lines = ssc.textFileStream("/home/workspace/spark/data")
val words = lines.flatMap(line => line.split(","))
val pairs = words.map(word => (word, 1))
val wordCount = pairs.reduceByKey(_ + _)
wordCount.print()
ssc.start()
ssc.awaitTermination()

Interestingly, it works seemlessly in Windows and the file I place is picked and program working fine.

Even, if I poll HDFS folder path "hdfs:///home/workspace/data", the program picks the file and read perfectly. But only not happening in CentOs local folder path.

scala apache-spark spark-streaming

share|improve this question

edited Mar 8 at 10:49

Saravanan Ponnaiah

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214

add a comment | 

0

I am trying to execute spark streaming word count application using spark-submit command. I started the spark program and when I copy new files to the local folder path, the file is not recognized and no operation is performed. In the log, I find error stating,

19/03/03 17:22:17 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@87b5b49/static/streaming,null,AVAILABLE,@Spark
19/03/03 17:22:17 INFO streaming.StreamingContext: StreamingContext started
19/03/03 17:22:21 WARN dstream.FileInputDStream: Error finding new files java.lang.NullPointerException

I use below command to submit spark job,

spark-submit --class com.company.stream.WordCounter --master local[4] /home/workspace/spark/SparkWordCounter.jar

Below is the source code,

val lines = ssc.textFileStream("/home/workspace/spark/data")
val words = lines.flatMap(line => line.split(","))
val pairs = words.map(word => (word, 1))
val wordCount = pairs.reduceByKey(_ + _)
wordCount.print()
ssc.start()
ssc.awaitTermination()

Interestingly, it works seemlessly in Windows and the file I place is picked and program working fine.

Even, if I poll HDFS folder path "hdfs:///home/workspace/data", the program picks the file and read perfectly. But only not happening in CentOs local folder path.

scala apache-spark spark-streaming

share|improve this question

edited Mar 8 at 10:49

Saravanan Ponnaiah

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214

add a comment | 

I am trying to execute spark streaming word count application using spark-submit command. I started the spark program and when I copy new files to the local folder path, the file is not recognized and no operation is performed. In the log, I find error stating,

19/03/03 17:22:17 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@87b5b49/static/streaming,null,AVAILABLE,@Spark
19/03/03 17:22:17 INFO streaming.StreamingContext: StreamingContext started
19/03/03 17:22:21 WARN dstream.FileInputDStream: Error finding new files java.lang.NullPointerException

I use below command to submit spark job,

spark-submit --class com.company.stream.WordCounter --master local[4] /home/workspace/spark/SparkWordCounter.jar

Below is the source code,

val lines = ssc.textFileStream("/home/workspace/spark/data")
val words = lines.flatMap(line => line.split(","))
val pairs = words.map(word => (word, 1))
val wordCount = pairs.reduceByKey(_ + _)
wordCount.print()
ssc.start()
ssc.awaitTermination()

Interestingly, it works seemlessly in Windows and the file I place is picked and program working fine.

Even, if I poll HDFS folder path "hdfs:///home/workspace/data", the program picks the file and read perfectly. But only not happening in CentOs local folder path.

scala apache-spark spark-streaming

share|improve this question

edited Mar 8 at 10:49

Saravanan Ponnaiah

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214

val lines = ssc.textFileStream("/home/workspace/spark/data")
val words = lines.flatMap(line => line.split(","))
val pairs = words.map(word => (word, 1))
val wordCount = pairs.reduceByKey(_ + _)
wordCount.print()
ssc.start()
ssc.awaitTermination()

share|improve this question

edited Mar 8 at 10:49

Saravanan Ponnaiah

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214

share|improve this question

edited Mar 8 at 10:49

Saravanan Ponnaiah

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214

asked Mar 7 at 11:58

Saravanan PonnaiahSaravanan Ponnaiah

214