How to store data into specific directories after reading it from kafka consumer?2019 Community Moderator ElectionData Modeling with Kafka? Topics and PartitionsHow to read data using Kafka Consumer API from beginning?Kafka distributing messages from a partition among consumersSpark Streaming from Kafka ConsumerHow should I process Kafka data?How to fetch ConsumerOffset (stored in kafka) for a consumer group in Java?How does Kafka store offsets for each topic?How to authenticate/authorize a consumer in Kafka for a topic before it consumes the messageKafka 0.10 Java consumer not reading message from topicKafka Consumer implementation for reading messages from multiple topics and producer
How much attack damage does the AC boost from a shield prevent on average?
Why don't MCU characters ever seem to have language issues?
How did the power source of Mar-Vell's aircraft end up with her?
Solving "Resistance between two nodes on a grid" problem in Mathematica
Is there an elementary proof that there are infinitely many primes that are *not* completely split in an abelian extension?
Why doesn't this Google Translate ad use the word "Translation" instead of "Translate"?
Why is Beresheet doing a only a one-way trip?
What Happens when Passenger Refuses to Fly Boeing 737 Max?
Low budget alien movie about the Earth being cooked
If the Captain's screens are out, does he switch seats with the co-pilot?
Virginia employer terminated employee and wants signing bonus returned
Things to avoid when using voltage regulators?
How to clip a background including nodes according to an arbitrary shape?
Why is there a voltage between the mains ground and my radiator?
Reverse string, can I make it faster?
How do you like my writing?
Is "history" a male-biased word ("his+story")?
Is there any way to damage Intellect Devourer(s) when already within a creature's skull?
The bar has been raised
Is it possible to have an Abelian group under two different binary operations but the binary operations are not distributive?
How do I locate a classical quotation?
Who deserves to be first and second author? PhD student who collected data, research associate who wrote the paper or supervisor?
Peter's Strange Word
Examples of a statistic that is not independent of sample's distribution?
How to store data into specific directories after reading it from kafka consumer?
2019 Community Moderator ElectionData Modeling with Kafka? Topics and PartitionsHow to read data using Kafka Consumer API from beginning?Kafka distributing messages from a partition among consumersSpark Streaming from Kafka ConsumerHow should I process Kafka data?How to fetch ConsumerOffset (stored in kafka) for a consumer group in Java?How does Kafka store offsets for each topic?How to authenticate/authorize a consumer in Kafka for a topic before it consumes the messageKafka 0.10 Java consumer not reading message from topicKafka Consumer implementation for reading messages from multiple topics and producer
I have a Kafka topic from which I need to read JSON data using consumer, but based on specific tags in the data I need to store it in different directories in the drive.
How do I achieve this? Can the data be directly parsed before storing it?
apache-kafka kafka-consumer-api apache-kafka-connect
edited Mar 6 at 19:29
cricket_007
83k1145112
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I have a Kafka topic from which I need to read JSON data using consumer, but based on specific tags in the data I need to store it in different directories in the drive.
How do I achieve this? Can the data be directly parsed before storing it?
apache-kafka kafka-consumer-api apache-kafka-connect
edited Mar 6 at 19:29
cricket_007
83k1145112
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I have a Kafka topic from which I need to read JSON data using consumer, but based on specific tags in the data I need to store it in different directories in the drive.
How do I achieve this? Can the data be directly parsed before storing it?
apache-kafka kafka-consumer-api apache-kafka-connect
edited Mar 6 at 19:29
cricket_007
83k1145112
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have a Kafka topic from which I need to read JSON data using consumer, but based on specific tags in the data I need to store it in different directories in the drive.
How do I achieve this? Can the data be directly parsed before storing it?
apache-kafka kafka-consumer-api apache-kafka-connect
apache-kafka kafka-consumer-api apache-kafka-connect
edited Mar 6 at 19:29
cricket_007
83k1145112
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Mar 6 at 19:29
cricket_007
83k1145112
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Mar 6 at 19:29
cricket_007
83k1145112
edited Mar 6 at 19:29
cricket_007
83k1145112
edited Mar 6 at 19:29
cricket_007
83k1145112
83k1145112
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
asked Mar 6 at 16:14
Dijesh NairDijesh Nair
104
104
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Dijesh Nair is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Not clear where those directories will exist, but you can try using the HDFS Kafka Connect (which will work with an HDFS-Compatible filesystem, including local disk) with the FieldPartitioner to designate which fields within the Kafka records are to be made as directories in the format of /topic-dir/topic-name/field1=value/field2=value
To write to local disk, set store.url=file:///some/path
Otherwise, Spark, Flink, regular Kafka consumer etc, you can consume that data and construct directories on your own.
Can the data be directly parsed before storing it?
Depends on how you consume the data...
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Dijesh Nair is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55027602%2fhow-to-store-data-into-specific-directories-after-reading-it-from-kafka-consumer%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Not clear where those directories will exist, but you can try using the HDFS Kafka Connect (which will work with an HDFS-Compatible filesystem, including local disk) with the FieldPartitioner to designate which fields within the Kafka records are to be made as directories in the format of /topic-dir/topic-name/field1=value/field2=value
To write to local disk, set store.url=file:///some/path
Otherwise, Spark, Flink, regular Kafka consumer etc, you can consume that data and construct directories on your own.
Can the data be directly parsed before storing it?
Depends on how you consume the data...
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
add a comment |
Not clear where those directories will exist, but you can try using the HDFS Kafka Connect (which will work with an HDFS-Compatible filesystem, including local disk) with the FieldPartitioner to designate which fields within the Kafka records are to be made as directories in the format of /topic-dir/topic-name/field1=value/field2=value
To write to local disk, set store.url=file:///some/path
Otherwise, Spark, Flink, regular Kafka consumer etc, you can consume that data and construct directories on your own.
Can the data be directly parsed before storing it?
Depends on how you consume the data...
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
add a comment |
Not clear where those directories will exist, but you can try using the HDFS Kafka Connect (which will work with an HDFS-Compatible filesystem, including local disk) with the FieldPartitioner to designate which fields within the Kafka records are to be made as directories in the format of /topic-dir/topic-name/field1=value/field2=value
To write to local disk, set store.url=file:///some/path
Otherwise, Spark, Flink, regular Kafka consumer etc, you can consume that data and construct directories on your own.
Can the data be directly parsed before storing it?
Depends on how you consume the data...
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
Not clear where those directories will exist, but you can try using the HDFS Kafka Connect (which will work with an HDFS-Compatible filesystem, including local disk) with the FieldPartitioner to designate which fields within the Kafka records are to be made as directories in the format of /topic-dir/topic-name/field1=value/field2=value
To write to local disk, set store.url=file:///some/path
Otherwise, Spark, Flink, regular Kafka consumer etc, you can consume that data and construct directories on your own.
Can the data be directly parsed before storing it?
Depends on how you consume the data...
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
answered Mar 6 at 19:31
cricket_007cricket_007
83k1145112
83k1145112
add a comment |
add a comment |
Dijesh Nair is a new contributor. Be nice, and check out our Code of Conduct.
Dijesh Nair is a new contributor. Be nice, and check out our Code of Conduct.
Dijesh Nair is a new contributor. Be nice, and check out our Code of Conduct.
Dijesh Nair is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55027602%2fhow-to-store-data-into-specific-directories-after-reading-it-from-kafka-consumer%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
