Amazon and websocket recording best practices Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) Data science time! April 2019 and salary with experience Should we burninate the [wrap] tag? The Ask Question Wizard is Live!Commonly accepted best practices around code organization in JavaScriptTrying to SSH into an Amazon Ec2 instance - permission errorWhat is the simplest way to process WebSockets messages on AWS?Amazon S3 + Lambda + DynamoDB Website HostingAWS “Serverless” architecture for real time client-server messengingHow to execute Amazon Lambda functions on dedicated EC2 server?Is it possible to subscribe to a WebSocket in a serverless fashion using AWS?How would I configure an EC2 instance To Download From MYSQL and Process The Data Based On A Script?Schedule to start an EC2 instance and run a python script within itUDP/TCP server in AWS
How can I make names more distinctive without making them longer?
What is a non-alternating simple group with big order, but relatively few conjugacy classes?
What is the role of the transistor and diode in a soft start circuit?
Why did the rest of the Eastern Bloc not invade Yugoslavia?
Storing hydrofluoric acid before the invention of plastics
Why was the term "discrete" used in discrete logarithm?
What does the word "veer" mean here?
How to align text above triangle figure
How to call a function with default parameter through a pointer to function that is the return of another function?
How to run gsettings for another user Ubuntu 18.04.2 LTS
Identifying polygons that intersect with another layer using QGIS?
Why are there no cargo aircraft with "flying wing" design?
English words in a non-english sci-fi novel
A coin, having probability p of landing heads and probability of q=(1-p) of landing on heads.
How would the world control an invulnerable immortal mass murderer?
List *all* the tuples!
What is Arya's weapon design?
Is there a (better) way to access $wpdb results?
Short Story with Cinderella as a Voo-doo Witch
List of Python versions
Sci-Fi book where patients in a coma ward all live in a subconscious world linked together
Bete Noir -- no dairy
Coloring maths inside a tcolorbox
How to find all the available tools in macOS terminal?
Amazon and websocket recording best practices
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)
Data science time! April 2019 and salary with experience
Should we burninate the [wrap] tag?
The Ask Question Wizard is Live!Commonly accepted best practices around code organization in JavaScriptTrying to SSH into an Amazon Ec2 instance - permission errorWhat is the simplest way to process WebSockets messages on AWS?Amazon S3 + Lambda + DynamoDB Website HostingAWS “Serverless” architecture for real time client-server messengingHow to execute Amazon Lambda functions on dedicated EC2 server?Is it possible to subscribe to a WebSocket in a serverless fashion using AWS?How would I configure an EC2 instance To Download From MYSQL and Process The Data Based On A Script?Schedule to start an EC2 instance and run a python script within itUDP/TCP server in AWS
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
add a comment |
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
add a comment |
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
13
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55068004%2famazon-and-websocket-recording-best-practices%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55068004%2famazon-and-websocket-recording-best-practices%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
