DynamoDB, method to paginate 100,000 documents with offset/limit? The Next CEO of Stack OverflowAWS DynamoDB get multiple items by String an range with JavaAWS DynamoBD NoSQL: many-to-many with composite keysWhat is the sense of Query in AWS DynamoDBHow do I get started with DynamoDBdyanmodb batchGetItem and Partition Key and Sort KeyHow should I store this in DynamoDB if I want to search by these fields?2 items added to DynamoDB when I run putItemHow to design key schema to have only one DynamoDB table per application?Need advice on DynamoDB schemaHow to use pagination on dynamoDB
Getting Stale Gas Out of a Gas Tank w/out Dropping the Tank
Help understanding this unsettling image of Titan, Epimetheus, and Saturn's rings?
Can this note be analyzed as a non-chord tone?
Help! I cannot understand this game’s notations!
Audio Conversion With ADS1243
Is there a way to save my career from absolute disaster?
Is French Guiana a (hard) EU border?
Easy to read palindrome checker
TikZ: How to fill area with a special pattern?
Is there a reasonable and studied concept of reduction between regular languages?
Man transported from Alternate World into ours by a Neutrino Detector
Lucky Feat: How can "more than one creature spend a luck point to influence the outcome of a roll"?
How to find image of a complex function with given constraints?
Aggressive Under-Indexing and no data for missing index
Could a dragon use its wings to swim?
What steps are necessary to read a Modern SSD in Medieval Europe?
Film where the government was corrupt with aliens, people sent to kill aliens are given rigged visors not showing the right aliens
IC has pull-down resistors on SMBus lines?
free fall ellipse or parabola?
A question about free fall, velocity, and the height of an object.
From jafe to El-Guest
Expectation in a stochastic differential equation
(How) Could a medieval fantasy world survive a magic-induced "nuclear winter"?
Players Circumventing the limitations of Wish
DynamoDB, method to paginate 100,000 documents with offset/limit?
The Next CEO of Stack OverflowAWS DynamoDB get multiple items by String an range with JavaAWS DynamoBD NoSQL: many-to-many with composite keysWhat is the sense of Query in AWS DynamoDBHow do I get started with DynamoDBdyanmodb batchGetItem and Partition Key and Sort KeyHow should I store this in DynamoDB if I want to search by these fields?2 items added to DynamoDB when I run putItemHow to design key schema to have only one DynamoDB table per application?Need advice on DynamoDB schemaHow to use pagination on dynamoDB
I have 100,000 documents successfully stored in a table. Currently, I'm just using a primary key (not hash/sort combo). There's no good way to split these into useful partitions for reads, because primarily customers will just be initially pulling the entire database, and then just pulling whatever items have been updated. Additionally, I would like to return results in a paginated fashion using an offset/limit method.
I'm wondering what the best way to do this is. An example item that I have stored in the table is like (id is the primary key):
"id": 11299,
"name": "plugin1",
"attributes":
"plugin_version": "1.30",
"exploit_available": false,
"in_the_news": false,
"exploited_by_malware": false,
"exploited_by_nessus": false,
"risk_factor": "Medium",
"plugin_type": "remote",
"exploitability_ease": "No known exploits are available",
"plugin_publication_date": "2003-03-01T00:00:00Z",
"plugin_modification_date": "2018-07-16T00:00:00Z",
"vuln_publication_date": "2003-01-23T00:00:00Z"
"cvss_temporal_vector":
"raw": "E:U/RL:OF/RC:C",
"ReportConfidence": "Confirmed",
"Exploitability": "Unproven",
"RemediationLevel": "Official-fix"
I also need to filter on plugin_modification_date, so not sure if it would be helpful to make that a sort key. What's been frustrating when investigating this so far is that everything seems to rely on using the partition key somehow, where it is basically useless when you have a solitary primary key which is unique for all items.
java kotlin amazon-dynamodb dynamodb-queries
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
add a comment |
I have 100,000 documents successfully stored in a table. Currently, I'm just using a primary key (not hash/sort combo). There's no good way to split these into useful partitions for reads, because primarily customers will just be initially pulling the entire database, and then just pulling whatever items have been updated. Additionally, I would like to return results in a paginated fashion using an offset/limit method.
I'm wondering what the best way to do this is. An example item that I have stored in the table is like (id is the primary key):
"id": 11299,
"name": "plugin1",
"attributes":
"plugin_version": "1.30",
"exploit_available": false,
"in_the_news": false,
"exploited_by_malware": false,
"exploited_by_nessus": false,
"risk_factor": "Medium",
"plugin_type": "remote",
"exploitability_ease": "No known exploits are available",
"plugin_publication_date": "2003-03-01T00:00:00Z",
"plugin_modification_date": "2018-07-16T00:00:00Z",
"vuln_publication_date": "2003-01-23T00:00:00Z"
"cvss_temporal_vector":
"raw": "E:U/RL:OF/RC:C",
"ReportConfidence": "Confirmed",
"Exploitability": "Unproven",
"RemediationLevel": "Official-fix"
I also need to filter on plugin_modification_date, so not sure if it would be helpful to make that a sort key. What's been frustrating when investigating this so far is that everything seems to rely on using the partition key somehow, where it is basically useless when you have a solitary primary key which is unique for all items.
java kotlin amazon-dynamodb dynamodb-queries
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
1
DynamoDB isn’t well suited for offset/limit pagination because either you need to give the items unique sequential numbers (and maintain the numbers as items are added and removed) or else you need to read items 0..N-1 just to know which item is the Nth item. Furthermore, the ordering of a scan is not guaranteed to be consistent between calls.
– Matthew Pope
Mar 7 at 22:22
add a comment |
I have 100,000 documents successfully stored in a table. Currently, I'm just using a primary key (not hash/sort combo). There's no good way to split these into useful partitions for reads, because primarily customers will just be initially pulling the entire database, and then just pulling whatever items have been updated. Additionally, I would like to return results in a paginated fashion using an offset/limit method.
I'm wondering what the best way to do this is. An example item that I have stored in the table is like (id is the primary key):
"id": 11299,
"name": "plugin1",
"attributes":
"plugin_version": "1.30",
"exploit_available": false,
"in_the_news": false,
"exploited_by_malware": false,
"exploited_by_nessus": false,
"risk_factor": "Medium",
"plugin_type": "remote",
"exploitability_ease": "No known exploits are available",
"plugin_publication_date": "2003-03-01T00:00:00Z",
"plugin_modification_date": "2018-07-16T00:00:00Z",
"vuln_publication_date": "2003-01-23T00:00:00Z"
"cvss_temporal_vector":
"raw": "E:U/RL:OF/RC:C",
"ReportConfidence": "Confirmed",
"Exploitability": "Unproven",
"RemediationLevel": "Official-fix"
I also need to filter on plugin_modification_date, so not sure if it would be helpful to make that a sort key. What's been frustrating when investigating this so far is that everything seems to rely on using the partition key somehow, where it is basically useless when you have a solitary primary key which is unique for all items.
java kotlin amazon-dynamodb dynamodb-queries
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
I have 100,000 documents successfully stored in a table. Currently, I'm just using a primary key (not hash/sort combo). There's no good way to split these into useful partitions for reads, because primarily customers will just be initially pulling the entire database, and then just pulling whatever items have been updated. Additionally, I would like to return results in a paginated fashion using an offset/limit method.
I'm wondering what the best way to do this is. An example item that I have stored in the table is like (id is the primary key):
"id": 11299,
"name": "plugin1",
"attributes":
"plugin_version": "1.30",
"exploit_available": false,
"in_the_news": false,
"exploited_by_malware": false,
"exploited_by_nessus": false,
"risk_factor": "Medium",
"plugin_type": "remote",
"exploitability_ease": "No known exploits are available",
"plugin_publication_date": "2003-03-01T00:00:00Z",
"plugin_modification_date": "2018-07-16T00:00:00Z",
"vuln_publication_date": "2003-01-23T00:00:00Z"
"cvss_temporal_vector":
"raw": "E:U/RL:OF/RC:C",
"ReportConfidence": "Confirmed",
"Exploitability": "Unproven",
"RemediationLevel": "Official-fix"
I also need to filter on plugin_modification_date, so not sure if it would be helpful to make that a sort key. What's been frustrating when investigating this so far is that everything seems to rely on using the partition key somehow, where it is basically useless when you have a solitary primary key which is unique for all items.
java kotlin amazon-dynamodb dynamodb-queries
java kotlin amazon-dynamodb dynamodb-queries
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
asked Mar 7 at 17:55
Matt TakaoMatt Takao
253
253
1
DynamoDB isn’t well suited for offset/limit pagination because either you need to give the items unique sequential numbers (and maintain the numbers as items are added and removed) or else you need to read items 0..N-1 just to know which item is the Nth item. Furthermore, the ordering of a scan is not guaranteed to be consistent between calls.
– Matthew Pope
Mar 7 at 22:22
add a comment |
1
DynamoDB isn’t well suited for offset/limit pagination because either you need to give the items unique sequential numbers (and maintain the numbers as items are added and removed) or else you need to read items 0..N-1 just to know which item is the Nth item. Furthermore, the ordering of a scan is not guaranteed to be consistent between calls.
– Matthew Pope
Mar 7 at 22:22
1
1
DynamoDB isn’t well suited for offset/limit pagination because either you need to give the items unique sequential numbers (and maintain the numbers as items are added and removed) or else you need to read items 0..N-1 just to know which item is the Nth item. Furthermore, the ordering of a scan is not guaranteed to be consistent between calls.
– Matthew Pope
Mar 7 at 22:22
DynamoDB isn’t well suited for offset/limit pagination because either you need to give the items unique sequential numbers (and maintain the numbers as items are added and removed) or else you need to read items 0..N-1 just to know which item is the Nth item. Furthermore, the ordering of a scan is not guaranteed to be consistent between calls.
– Matthew Pope
Mar 7 at 22:22
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55050081%2fdynamodb-method-to-paginate-100-000-documents-with-offset-limit%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55050081%2fdynamodb-method-to-paginate-100-000-documents-with-offset-limit%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
DynamoDB isn’t well suited for offset/limit pagination because either you need to give the items unique sequential numbers (and maintain the numbers as items are added and removed) or else you need to read items 0..N-1 just to know which item is the Nth item. Furthermore, the ordering of a scan is not guaranteed to be consistent between calls.
– Matthew Pope
Mar 7 at 22:22