Spark streaming read old files and new as well Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Spark Streaming: HDFSSpark stream unable to read files created from flume in hdfsSpark Streaming not detecting new HDFS filesSpark streaming does not read files moved from HDFS to HDFSSpark Streaming: java.io.FileNotFoundException: File does not exist: <input_filename>._COPYING_SaveAsTextFile: Empty file causes Spark streaming exceptionRead new s3 file paths from spark streamingHow to create a stop condition on Spark streaming?In Spark Streaming how to process old data and delete processed Datassc.filestream unable to read already existing files in directory
Do wooden building fires get hotter than 600°C?
Why weren't discrete x86 CPUs ever used in game hardware?
How long can equipment go unused before powering up runs the risk of damage?
Is CEO the "profession" with the most psychopaths?
Flight departed from the gate 5 min before scheduled departure time. Refund options
Karn the great creator - 'card from outside the game' in sealed
Maximum summed subsequences with non-adjacent items
Project Euler #1 in C++
How many morphisms from 1 to 1+1 can there be?
Why is it faster to reheat something than it is to cook it?
Why does it sometimes sound good to play a grace note as a lead in to a note in a melody?
What order were files/directories output in dir?
What's the meaning of "fortified infraction restraint"?
Crossing US/Canada Border for less than 24 hours
Should a wizard buy fine inks every time he want to copy spells into his spellbook?
macOS: Name for app shortcut screen found by pinching with thumb and three fingers
The test team as an enemy of development? And how can this be avoided?
Is there hard evidence that the grant peer review system performs significantly better than random?
What is the chair depicted in Cesare Maccari's 1889 painting "Cicerone denuncia Catilina"?
Semigroups with no morphisms between them
How does Belgium enforce obligatory attendance in elections?
How can I prevent/balance waiting and turtling as a response to cooldown mechanics
How many time has Arya actually used Needle?
If Windows 7 doesn't support WSL, then what is "Subsystem for UNIX-based Applications"?
Spark streaming read old files and new as well
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
Data science time! April 2019 and salary with experience
The Ask Question Wizard is Live!Spark Streaming: HDFSSpark stream unable to read files created from flume in hdfsSpark Streaming not detecting new HDFS filesSpark streaming does not read files moved from HDFS to HDFSSpark Streaming: java.io.FileNotFoundException: File does not exist: <input_filename>._COPYING_SaveAsTextFile: Empty file causes Spark streaming exceptionRead new s3 file paths from spark streamingHow to create a stop condition on Spark streaming?In Spark Streaming how to process old data and delete processed Datassc.filestream unable to read already existing files in directory
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I am trying to read files using spark streaming. I am using textFileStream to read from hdfs like this:
val logStream=ssc.textFileStream(args(0))
Its able to get the new files created but do not read the files already present in the hdfs directory when the job is started. Is there any way this can be achieved?
apache-spark hadoop hdfs spark-streaming
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
add a comment |
I am trying to read files using spark streaming. I am using textFileStream to read from hdfs like this:
val logStream=ssc.textFileStream(args(0))
Its able to get the new files created but do not read the files already present in the hdfs directory when the job is started. Is there any way this can be achieved?
apache-spark hadoop hdfs spark-streaming
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
add a comment |
I am trying to read files using spark streaming. I am using textFileStream to read from hdfs like this:
val logStream=ssc.textFileStream(args(0))
Its able to get the new files created but do not read the files already present in the hdfs directory when the job is started. Is there any way this can be achieved?
apache-spark hadoop hdfs spark-streaming
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
I am trying to read files using spark streaming. I am using textFileStream to read from hdfs like this:
val logStream=ssc.textFileStream(args(0))
Its able to get the new files created but do not read the files already present in the hdfs directory when the job is started. Is there any way this can be achieved?
apache-spark hadoop hdfs spark-streaming
apache-spark hadoop hdfs spark-streaming
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
asked Mar 8 at 22:10
Y0gesh GuptaY0gesh Gupta
88722646
88722646
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55071702%2fspark-streaming-read-old-files-and-new-as-well%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55071702%2fspark-streaming-read-old-files-and-new-as-well%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
