Saving Strategy/design for high frequency incoming data [closed]2019 Community Moderator ElectionHow to find high frequency words in a book in an environment low on memory?Recognition of a short high frequency sound in low frequency noise (objc/c)Include zero frequencies in frequency table for Likert dataFrequency filtered image is black after performing inverse FFTHow to calculate quantile data for table of frequencies in SAS?frequency of discrete data in excelhow to extract high frequency phrase in pythonSave frequency data of histogram in MatlabAutocorrelation algorithm fails in high frequenciesGroup frequency data in R
Use of プラトニック in this sentence?
Can anyone tell me why this program fails?
Humanity loses the vast majority of its technology, information, and population in the year 2122. How long does it take to rebuild itself?
RegionDifference for Cylinder and Cuboid
What is IP squat space
What is the greatest age difference between a married couple in Tanach?
Latest web browser compatible with Windows 98
Replacing Windows 7 security updates with anti-virus?
Why are the outputs of printf and std::cout different
Is it normal that my co-workers at a fitness company criticize my food choices?
Calculus II Professor will not accept my correct integral evaluation that uses a different method, should I bring this up further?
Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?
My adviser wants to be the first author
Does this AnyDice function accurately calculate the number of ogres you make unconcious with three 4th-level castings of Sleep?
How to write cleanly even if my character uses expletive language?
Meaning of "SEVERA INDEOVI VAS" from 3rd Century slab
How to simplify this time periods definition interface?
At what level can a dragon innately cast its spells?
Why do Australian milk farmers need to protest supermarkets' milk price?
Theorems like the Lovász Local Lemma?
How to explain that I do not want to visit a country due to personal safety concern?
Is having access to past exams cheating and, if yes, could it be proven just by a good grade?
PTIJ: Who should pay for Uber rides: the child or the parent?
Where is the 1/8 CR apprentice in Volo's Guide to Monsters?
Saving Strategy/design for high frequency incoming data [closed]
2019 Community Moderator ElectionHow to find high frequency words in a book in an environment low on memory?Recognition of a short high frequency sound in low frequency noise (objc/c)Include zero frequencies in frequency table for Likert dataFrequency filtered image is black after performing inverse FFTHow to calculate quantile data for table of frequencies in SAS?frequency of discrete data in excelhow to extract high frequency phrase in pythonSave frequency data of histogram in MatlabAutocorrelation algorithm fails in high frequenciesGroup frequency data in R
What is the best way to save high frequency incoming data IOT system?
Should I be using NoSql or mysql should be enough? In case I use MYSQL with spring jpa, should i collect data and do batch create or spring data jpa is capable of handling the scenario?
Thanks
frequency
asked Mar 6 at 18:52
ZeyaulZeyaul
1
closed as primarily opinion-based by Sam Hanley, user590028, Marvin, gdlmx, Dale Burrell Mar 7 at 5:00
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
What is the best way to save high frequency incoming data IOT system?
Should I be using NoSql or mysql should be enough? In case I use MYSQL with spring jpa, should i collect data and do batch create or spring data jpa is capable of handling the scenario?
Thanks
frequency
asked Mar 6 at 18:52
ZeyaulZeyaul
1
closed as primarily opinion-based by Sam Hanley, user590028, Marvin, gdlmx, Dale Burrell Mar 7 at 5:00
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
What is the best way to save high frequency incoming data IOT system?
Should I be using NoSql or mysql should be enough? In case I use MYSQL with spring jpa, should i collect data and do batch create or spring data jpa is capable of handling the scenario?
Thanks
frequency
asked Mar 6 at 18:52
ZeyaulZeyaul
1
What is the best way to save high frequency incoming data IOT system?
Should I be using NoSql or mysql should be enough? In case I use MYSQL with spring jpa, should i collect data and do batch create or spring data jpa is capable of handling the scenario?
Thanks
frequency
frequency
asked Mar 6 at 18:52
ZeyaulZeyaul
1
asked Mar 6 at 18:52
ZeyaulZeyaul
1
asked Mar 6 at 18:52
ZeyaulZeyaul
1
asked Mar 6 at 18:52
ZeyaulZeyaul
1
asked Mar 6 at 18:52
ZeyaulZeyaul
1
1
closed as primarily opinion-based by Sam Hanley, user590028, Marvin, gdlmx, Dale Burrell Mar 7 at 5:00
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
closed as primarily opinion-based by Sam Hanley, user590028, Marvin, gdlmx, Dale Burrell Mar 7 at 5:00
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Use a pub-sub model. Your IOT data should be placed in the queue and your batch should listen to the queue continuously and update to any database of your choice. Spring batch has a JDBCTemplate which can do batch inserts and practically they are faster than JPA's
answered Mar 6 at 20:23
Tech GuyTech Guy
497
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Use a pub-sub model. Your IOT data should be placed in the queue and your batch should listen to the queue continuously and update to any database of your choice. Spring batch has a JDBCTemplate which can do batch inserts and practically they are faster than JPA's
answered Mar 6 at 20:23
Tech GuyTech Guy
497
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
add a comment |
Use a pub-sub model. Your IOT data should be placed in the queue and your batch should listen to the queue continuously and update to any database of your choice. Spring batch has a JDBCTemplate which can do batch inserts and practically they are faster than JPA's
answered Mar 6 at 20:23
Tech GuyTech Guy
497
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
add a comment |
Use a pub-sub model. Your IOT data should be placed in the queue and your batch should listen to the queue continuously and update to any database of your choice. Spring batch has a JDBCTemplate which can do batch inserts and practically they are faster than JPA's
answered Mar 6 at 20:23
Tech GuyTech Guy
497
Use a pub-sub model. Your IOT data should be placed in the queue and your batch should listen to the queue continuously and update to any database of your choice. Spring batch has a JDBCTemplate which can do batch inserts and practically they are faster than JPA's
answered Mar 6 at 20:23
Tech GuyTech Guy
497
answered Mar 6 at 20:23
Tech GuyTech Guy
497
answered Mar 6 at 20:23
Tech GuyTech Guy
497
answered Mar 6 at 20:23
Tech GuyTech Guy
497
497
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
add a comment |
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
Sorry if my question is not clear but I am not asking about design from reading point. what I am asking is i have data reached to a broker or some listen maintain queue. But now onward should I use NoSql or RDBMS. If RDBMS, then should I wait for 2 minutes and then use use JDBCTemplate for batch insert as u suggested or hit it as data received and jdbctemplate will handle caching and batch insert themself.
– Zeyaul
Mar 6 at 23:57
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
@Zeyaul It's better you use NoSQL. While you are fetching data from IOT systems, it must be some raw data. So it's better to have ELK pipeline around that, this approach will be faster and robust. As your data increases, for you can increase your clusters efficiently.
– AshwinK
Mar 7 at 7:16
add a comment |
