Where is the mapGroupsWithState State stored? Docker and Local instance behaving differently The Next CEO of Stack OverflowHow is Docker different from a virtual machine?Where are Docker images stored on the host machine?What is the difference between “expose” and “publish” in Docker?What is the difference between a Docker image and a container?NoSuchMethodError for Scala Seq line in SparkStructured Streaming set checkpointLocation offsets replication factorHow to get rid of NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe error in Spark Streaming + KafkaWhich spark-sql-kafka to use for Apche Kafka 1.0.0.?Rewind Offset Spark Structured Streaming from KafkaCustom state store provider for Apache Spark on Mesos
A Man With a Stainless Steel Endoskeleton (like The Terminator) Fighting Cloaked Aliens Only He Can See
Is a distribution that is normal, but highly skewed considered Gaussian?
What was the first Unix version to run on a microcomputer?
Does soap repel water?
Make solar eclipses exceedingly rare, but still have new moons
RigExpert AA-35 - Interpreting The Information
Is it ever safe to open a suspicious HTML file (e.g. email attachment)?
What did we know about the Kessel run before the prequels?
If Nick Fury and Coulson already knew about aliens (Kree and Skrull) why did they wait until Thor's appearance to start making weapons?
Bartok - Syncopation (1): Meaning of notes in between Grand Staff
Unclear about dynamic binding
Prepend last line of stdin to entire stdin
Why is the US ranked as #45 in Press Freedom ratings, despite its extremely permissive free speech laws?
Can you be charged for obstruction for refusing to answer questions?
Why didn't Khan get resurrected in the Genesis Explosion?
Where do students learn to solve polynomial equations these days?
Is it possible to use a NPN BJT as switch, from single power source?
Flying from Cape Town to England and return to another province
Axiom Schema vs Axiom
Does Germany produce more waste than the US?
How to install OpenCV on Raspbian Stretch?
Why the difference in type-inference over the as-pattern in two similar function definitions?
The past simple of "gaslight" – "gaslighted" or "gaslit"?
Proper way to express "He disappeared them"
Where is the mapGroupsWithState State stored? Docker and Local instance behaving differently
The Next CEO of Stack OverflowHow is Docker different from a virtual machine?Where are Docker images stored on the host machine?What is the difference between “expose” and “publish” in Docker?What is the difference between a Docker image and a container?NoSuchMethodError for Scala Seq line in SparkStructured Streaming set checkpointLocation offsets replication factorHow to get rid of NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe error in Spark Streaming + KafkaWhich spark-sql-kafka to use for Apche Kafka 1.0.0.?Rewind Offset Spark Structured Streaming from KafkaCustom state store provider for Apache Spark on Mesos
I have a project running Spark 2.2.1 Structured Streaming, with a calculation using mapGroupsWithState.
Running the project locally with
spark-submit --class com.project.DataEnrichment --master local[4] target/scala-2.11/assembly-project.jar
the local checkpointLocation contains the following folders:
- commits
- offsets
- sources
- state
But in our docker environment the checkpointLocation is missing the state folder. With the exact same application running.
I'm trying to figure out a way to keep the state outside of Docker so that it is possible to update the application without resetting the state, but first I have to locate it.
The docker environment is using the spark images from gettyimages/spark:2.2.1-hadoop-2.7
Is there any logical reason why the docker environment is not storing the State within the checkpoint Location? And is this configurable?
docker apache-spark spark-structured-streaming
asked Mar 7 at 16:38
MartinMartin
142213
add a comment |
I have a project running Spark 2.2.1 Structured Streaming, with a calculation using mapGroupsWithState.
Running the project locally with
spark-submit --class com.project.DataEnrichment --master local[4] target/scala-2.11/assembly-project.jar
the local checkpointLocation contains the following folders:
- commits
- offsets
- sources
- state
But in our docker environment the checkpointLocation is missing the state folder. With the exact same application running.
I'm trying to figure out a way to keep the state outside of Docker so that it is possible to update the application without resetting the state, but first I have to locate it.
The docker environment is using the spark images from gettyimages/spark:2.2.1-hadoop-2.7
Is there any logical reason why the docker environment is not storing the State within the checkpoint Location? And is this configurable?
docker apache-spark spark-structured-streaming
asked Mar 7 at 16:38
MartinMartin
142213
add a comment |
I have a project running Spark 2.2.1 Structured Streaming, with a calculation using mapGroupsWithState.
Running the project locally with
spark-submit --class com.project.DataEnrichment --master local[4] target/scala-2.11/assembly-project.jar
the local checkpointLocation contains the following folders:
- commits
- offsets
- sources
- state
But in our docker environment the checkpointLocation is missing the state folder. With the exact same application running.
I'm trying to figure out a way to keep the state outside of Docker so that it is possible to update the application without resetting the state, but first I have to locate it.
The docker environment is using the spark images from gettyimages/spark:2.2.1-hadoop-2.7
Is there any logical reason why the docker environment is not storing the State within the checkpoint Location? And is this configurable?
docker apache-spark spark-structured-streaming
asked Mar 7 at 16:38
MartinMartin
142213
I have a project running Spark 2.2.1 Structured Streaming, with a calculation using mapGroupsWithState.
Running the project locally with
spark-submit --class com.project.DataEnrichment --master local[4] target/scala-2.11/assembly-project.jar
the local checkpointLocation contains the following folders:
- commits
- offsets
- sources
- state
But in our docker environment the checkpointLocation is missing the state folder. With the exact same application running.
I'm trying to figure out a way to keep the state outside of Docker so that it is possible to update the application without resetting the state, but first I have to locate it.
The docker environment is using the spark images from gettyimages/spark:2.2.1-hadoop-2.7
Is there any logical reason why the docker environment is not storing the State within the checkpoint Location? And is this configurable?
docker apache-spark spark-structured-streaming
docker apache-spark spark-structured-streaming
asked Mar 7 at 16:38
MartinMartin
142213
asked Mar 7 at 16:38
MartinMartin
142213
asked Mar 7 at 16:38
MartinMartin
142213
asked Mar 7 at 16:38
MartinMartin
142213
asked Mar 7 at 16:38
MartinMartin
142213
142213
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55048788%2fwhere-is-the-mapgroupswithstate-state-stored-docker-and-local-instance-behaving%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55048788%2fwhere-is-the-mapgroupswithstate-state-stored-docker-and-local-instance-behaving%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
