Ufdjrw

Is it legal for company to use my work email to pretend I still work there?

Why does Arabsat 6A need a Falcon Heavy to launch

How to take photos in burst mode, without vibration?

Why is the 'in' operator throwing an error with a string literal instead of logging false?

Watching something be written to a file live with tail

Memorizing the Keyboard

Infinite Abelian subgroup of infinite non Abelian group example

How much of data wrangling is a data scientist's job?

How do I write bicross product symbols in latex?

How can I make my BBEG immortal short of making them a Lich or Vampire?

SSH "lag" in LAN on some machines, mixed distros

Why can't we play rap on piano?

Facing a paradox: Earnshaw's theorem in one dimension

What about the virus in 12 Monkeys?

Theorems that impeded progress

How can I prevent hyper evolved versions of regular creatures from wiping out their cousins?

Neighboring nodes in the network

What do you call someone who asks many questions?

In Romance of the Three Kingdoms why do people still use bamboo sticks when papers are already invented?

Is there a hemisphere-neutral way of specifying a season?

Could gravitational lensing be used to protect a spaceship from a laser?

Does a druid starting with a bow start with no arrows?

1960's book about a plague that kills all white people

What to put in ESTA if staying in US for a few days before going on to Canada

Database operations or http calls using kafka consumer

Which equals operator (== vs ===) should be used in JavaScript comparisons?What is the !! (not not) operator in JavaScript?What is the difference between call and apply?How do I return the response from an asynchronous call?Kafka consuming the latest message again when I rerun the Flink consumerUnderstanding Kafka Topics and PartitionsCan i consume based on specific condition in Kafka?How to replicate microservices when consuming same kafka topic?kafka consumer API consumer.poll() doen't work with no wrong, no exception and just blockMaking Kafka producer and Consumer synchronous

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

I am new to Apache Kafka. I have a few questions?
if you produce topic using Kafka producer and the Kafka consumer received the message

consumer.on('message', function (message) 
)

so for example, if I want to send data to some DB or to call in API to update my DB, Is it better to do all the codding inside consumer.on method

 consumer.on('message', function (message) 
 let data = await getPayload();//get payload
 //console.log(data,'data')
 //pause(topic,0); //pause until the prev request is completed
 request(
 url: process.env.API + '/some-api',
 method: 'POST',
 json: true,
 body: data
 , function(err, response, body) 
 if (err) 
 console.error('got error', err);
 //resume(topic,0); resume when err/response recived 
 else 
 // console.log('response', response);
 console.log('body', body);
 //resume(topic,0);
 
 );
 )

Is there any alternative or recomended methods?

javascript node.js apache-kafka

share|improve this question

asked Mar 8 at 0:01

SohailSohail

139111

add a comment | 

0

I am new to Apache Kafka. I have a few questions?
if you produce topic using Kafka producer and the Kafka consumer received the message

consumer.on('message', function (message) 
)

so for example, if I want to send data to some DB or to call in API to update my DB, Is it better to do all the codding inside consumer.on method

 consumer.on('message', function (message) 
 let data = await getPayload();//get payload
 //console.log(data,'data')
 //pause(topic,0); //pause until the prev request is completed
 request(
 url: process.env.API + '/some-api',
 method: 'POST',
 json: true,
 body: data
 , function(err, response, body) 
 if (err) 
 console.error('got error', err);
 //resume(topic,0); resume when err/response recived 
 else 
 // console.log('response', response);
 console.log('body', body);
 //resume(topic,0);
 
 );
 )

Is there any alternative or recomended methods?

javascript node.js apache-kafka

share|improve this question

asked Mar 8 at 0:01

SohailSohail

139111

add a comment | 

I am new to Apache Kafka. I have a few questions?
if you produce topic using Kafka producer and the Kafka consumer received the message

consumer.on('message', function (message) 
)

so for example, if I want to send data to some DB or to call in API to update my DB, Is it better to do all the codding inside consumer.on method

 consumer.on('message', function (message) 
 let data = await getPayload();//get payload
 //console.log(data,'data')
 //pause(topic,0); //pause until the prev request is completed
 request(
 url: process.env.API + '/some-api',
 method: 'POST',
 json: true,
 body: data
 , function(err, response, body) 
 if (err) 
 console.error('got error', err);
 //resume(topic,0); resume when err/response recived 
 else 
 // console.log('response', response);
 console.log('body', body);
 //resume(topic,0);
 
 );
 )

Is there any alternative or recomended methods?

javascript node.js apache-kafka

share|improve this question

asked Mar 8 at 0:01

SohailSohail

139111

consumer.on('message', function (message) 
)

 consumer.on('message', function (message) 
 let data = await getPayload();//get payload
 //console.log(data,'data')
 //pause(topic,0); //pause until the prev request is completed
 request(
 url: process.env.API + '/some-api',
 method: 'POST',
 json: true,
 body: data
 , function(err, response, body) 
 if (err) 
 console.error('got error', err);
 //resume(topic,0); resume when err/response recived 
 else 
 // console.log('response', response);
 console.log('body', body);
 //resume(topic,0);
 
 );
 )

share|improve this question

asked Mar 8 at 0:01

SohailSohail

139111

share|improve this question

asked Mar 8 at 0:01

SohailSohail

139111

asked Mar 8 at 0:01

SohailSohail

139111

asked Mar 8 at 0:01

SohailSohail

139111

1 Answer
1

active

oldest

votes

1

While consuming from kafka and loading it to any sink like DB in your case , you need to be aware handling any failures, exactly one semantics etc.
There is already a database sink connector available (link below), you can check if it suffice you requirements

kafka-connect-jdbc

share|improve this answer

answered Mar 8 at 2:45

Liju JohnLiju John

980912

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  what about API call?
  
  – Sohail
  Mar 8 at 14:28
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55054760%2fdatabase-operations-or-http-calls-using-kafka-consumer%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

While consuming from kafka and loading it to any sink like DB in your case , you need to be aware handling any failures, exactly one semantics etc.
There is already a database sink connector available (link below), you can check if it suffice you requirements

kafka-connect-jdbc

share|improve this answer

answered Mar 8 at 2:45

Liju JohnLiju John

980912

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  what about API call?
  
  – Sohail
  Mar 8 at 14:28
  

  
  
  
  

add a comment | 

1

While consuming from kafka and loading it to any sink like DB in your case , you need to be aware handling any failures, exactly one semantics etc.
There is already a database sink connector available (link below), you can check if it suffice you requirements

kafka-connect-jdbc

share|improve this answer

answered Mar 8 at 2:45

Liju JohnLiju John

980912

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  what about API call?
  
  – Sohail
  Mar 8 at 14:28
  

  
  
  
  

add a comment | 

While consuming from kafka and loading it to any sink like DB in your case , you need to be aware handling any failures, exactly one semantics etc.
There is already a database sink connector available (link below), you can check if it suffice you requirements

kafka-connect-jdbc

share|improve this answer

answered Mar 8 at 2:45

Liju JohnLiju John

980912

share|improve this answer

answered Mar 8 at 2:45

Liju JohnLiju John

980912

answered Mar 8 at 2:45

Liju JohnLiju John

980912

answered Mar 8 at 2:45

Liju JohnLiju John

980912