Continuous processing mode and python udf Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Why did two spark streaming jobs pull messages from the same Kafka topic with same group id not balancing load but getting same messages?Spark Streaming Kafka createDirectStream - Spark UI shows input event size as zeroFIFO processing using Spark Streaming?Spark streaming applications subscribing to same kafka topicSpark Structured Streaming Multiple Kafka Topics With Unique Message SchemasProducer/Consumer Kafka Spark StreamingMessage issue with Producer/Consumer Kafka Spark StreamingSpark continuous processing mode does not read all kafka topic partitionRestarting Spark Structured Streaming Job consumes Millions of Kafka messages and diesHow to build a dataframe thats a join of two kafka streams with “key(s)” column and remaining columns being the latest values
How does light 'choose' between wave and particle behaviour?
Time evolution of a Gaussian wave packet, why convert to k-space?
What initially awakened the Balrog?
Sum letters are not two different
How can I prevent/balance waiting and turtling as a response to cooldown mechanics
Semigroups with no morphisms between them
Is there public access to the Meteor Crater in Arizona?
How can I set the aperture on my DSLR when it's attached to a telescope instead of a lens?
How could we fake a moon landing now?
How many time has Arya actually used Needle?
A term for a woman complaining about things/begging in a cute/childish way
Do I really need to have a message in a novel to appeal to readers?
Did any compiler fully use 80-bit floating point?
What is an "asse" in Elizabethan English?
What is the difference between a "ranged attack" and a "ranged weapon attack"?
Project Euler #1 in C++
Do wooden building fires get hotter than 600°C?
Is the IBM 5153 color display compatible with the Tandy 1000 16 color modes?
How does Belgium enforce obligatory attendance in elections?
Has negative voting ever been officially implemented in elections, or seriously proposed, or even studied?
Significance of Cersei's obsession with elephants?
How do living politicians protect their readily obtainable signatures from misuse?
Karn the great creator - 'card from outside the game' in sealed
How were pictures turned from film to a big picture in a picture frame before digital scanning?
Continuous processing mode and python udf
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
Data science time! April 2019 and salary with experience
The Ask Question Wizard is Live!Why did two spark streaming jobs pull messages from the same Kafka topic with same group id not balancing load but getting same messages?Spark Streaming Kafka createDirectStream - Spark UI shows input event size as zeroFIFO processing using Spark Streaming?Spark streaming applications subscribing to same kafka topicSpark Structured Streaming Multiple Kafka Topics With Unique Message SchemasProducer/Consumer Kafka Spark StreamingMessage issue with Producer/Consumer Kafka Spark StreamingSpark continuous processing mode does not read all kafka topic partitionRestarting Spark Structured Streaming Job consumes Millions of Kafka messages and diesHow to build a dataframe thats a join of two kafka streams with “key(s)” column and remaining columns being the latest values
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
does Spark 2.4.0 support Python UDFs with Continuous Processing mode?
In my simple code i'm consuming from a kafka topic, doing some trivial processing per-row (basically add a dummy field to the json messages) and write out to another topic.
However i dont see any messages in the output topic when i trigger using continuous mode. If i dont use the python udf, then it works fine.
EDIT: This bug report (unrelated to this post) reported lack of support.
apache-spark spark-streaming spark-structured-streaming spark-streaming-kafka
asked Mar 8 at 22:09
VenkiVenki
6914
add a comment |
does Spark 2.4.0 support Python UDFs with Continuous Processing mode?
In my simple code i'm consuming from a kafka topic, doing some trivial processing per-row (basically add a dummy field to the json messages) and write out to another topic.
However i dont see any messages in the output topic when i trigger using continuous mode. If i dont use the python udf, then it works fine.
EDIT: This bug report (unrelated to this post) reported lack of support.
apache-spark spark-streaming spark-structured-streaming spark-streaming-kafka
asked Mar 8 at 22:09
VenkiVenki
6914
I'll post sample code on Monday
– Venki
Mar 8 at 22:10
add a comment |
does Spark 2.4.0 support Python UDFs with Continuous Processing mode?
In my simple code i'm consuming from a kafka topic, doing some trivial processing per-row (basically add a dummy field to the json messages) and write out to another topic.
However i dont see any messages in the output topic when i trigger using continuous mode. If i dont use the python udf, then it works fine.
EDIT: This bug report (unrelated to this post) reported lack of support.
apache-spark spark-streaming spark-structured-streaming spark-streaming-kafka
asked Mar 8 at 22:09
VenkiVenki
6914
does Spark 2.4.0 support Python UDFs with Continuous Processing mode?
In my simple code i'm consuming from a kafka topic, doing some trivial processing per-row (basically add a dummy field to the json messages) and write out to another topic.
However i dont see any messages in the output topic when i trigger using continuous mode. If i dont use the python udf, then it works fine.
EDIT: This bug report (unrelated to this post) reported lack of support.
apache-spark spark-streaming spark-structured-streaming spark-streaming-kafka
apache-spark spark-streaming spark-structured-streaming spark-streaming-kafka
asked Mar 8 at 22:09
VenkiVenki
6914
asked Mar 8 at 22:09
VenkiVenki
6914
edited Apr 5 at 17:08
Venki
asked Mar 8 at 22:09
VenkiVenki
6914
asked Mar 8 at 22:09
VenkiVenki
6914
asked Mar 8 at 22:09
VenkiVenki
6914
6914
I'll post sample code on Monday
– Venki
Mar 8 at 22:10
add a comment |
I'll post sample code on Monday
– Venki
Mar 8 at 22:10
I'll post sample code on Monday
– Venki
Mar 8 at 22:10
I'll post sample code on Monday
– Venki
Mar 8 at 22:10
add a comment |
1 Answer
1
active
oldest
votes
Although I have not tried myself, I believe not. The documentation lists the supported operations; udfs are not on the list.
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55071690%2fcontinuous-processing-mode-and-python-udf%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Although I have not tried myself, I believe not. The documentation lists the supported operations; udfs are not on the list.
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
add a comment |
Although I have not tried myself, I believe not. The documentation lists the supported operations; udfs are not on the list.
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
add a comment |
Although I have not tried myself, I believe not. The documentation lists the supported operations; udfs are not on the list.
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
Although I have not tried myself, I believe not. The documentation lists the supported operations; udfs are not on the list.
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
answered Apr 16 at 7:00
Matthew JacksonMatthew Jackson
94211
94211
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55071690%2fcontinuous-processing-mode-and-python-udf%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I'll post sample code on Monday
– Venki
Mar 8 at 22:10