Parsing datastore Entity in Python efficiently The 2019 Stack Overflow Developer Survey Results Are InAppengine datastore phantom entity - inconsistent state?Best practice to query large number of ndb entities from datastoreappengine: How to query entities by key, in entities with ancester?Deleted entities keep returning using datastore adminGAE datastore admin copy failing on MapReduce model to JSON conversionHow to avoid “safety” over quota panic when accessing datastore ? (billing is enabled)Querying immediately for saved entity returns null sometimes - datastoregoogle datastore: breaking change re: anonymous struct fields?How to filter a parent entity using properties of child entity in datastoreGoogle Cloud Datastore Indexes for count queries

Understanding the implication of what "well-defined" means for the operation in quotient group

Why isn't airport relocation done gradually?

Lethal sonic weapons

aging parents with no investments

How to deal with fear of taking dependencies

Why is my p-value correlated to difference between means in two sample tests?

Why is it "Tumoren" and not "Tumore"?

What are the motivations for publishing new editions of an existing textbook, beyond new discoveries in a field?

What is the meaning of Triage in Cybersec world?

Is this food a bread or a loaf?

Which Sci-Fi work first showed weapon of galactic-scale mass destruction?

What do the Banks children have against barley water?

Why do UK politicians seemingly ignore opinion polls on Brexit?

What can other administrators access on my machine?

What does Linus Torvalds mean when he says that Git "never ever" tracks a file?

Where does the "burst of radiance" from Holy Weapon originate?

How to make payment on the internet without leaving a money trail?

Does duplicating a spell with Wish count as casting that spell?

What is the best strategy for white in this position?

Is flight data recorder erased after every flight?

Does light intensity oscillate really fast since it is a wave?

"What time...?" or "At what time...?" - what is more grammatically correct?

What does "rabbited" mean/imply in this sentence?

Why don't Unix/Linux systems traverse through directories until they find the required version of a linked library?



Parsing datastore Entity in Python efficiently



The 2019 Stack Overflow Developer Survey Results Are InAppengine datastore phantom entity - inconsistent state?Best practice to query large number of ndb entities from datastoreappengine: How to query entities by key, in entities with ancester?Deleted entities keep returning using datastore adminGAE datastore admin copy failing on MapReduce model to JSON conversionHow to avoid “safety” over quota panic when accessing datastore ? (billing is enabled)Querying immediately for saved entity returns null sometimes - datastoregoogle datastore: breaking change re: anonymous struct fields?How to filter a parent entity using properties of child entity in datastoreGoogle Cloud Datastore Indexes for count queries



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








1















So right now, datastore ds.query(kind='users') returns me a response in the form:



<Entity(u'users', 5633378543992832L) u'username': u'xyz', u'user_ip': '127.0.0.1', u'name': u'xyz', u'password': 'something', u'register_date': datetime.datetime(2019, 3, 8, 5, 50, 1, 443212, tzinfo=<UTC>)>


And although I can iterate it like this:



result = 
for oneItem in query.fetch():
# oneItem is Entity iterable as shown above
for oneProp in oneItem:
result[oneProp] = oneItem[oneProp]


And access any property by something = result['password']



Which works fine but it is grossly inefficient. Is there any way I can directly access any specific property without using for loops or another data structure? Similar to accessing a value in JSON.



I am using from google.cloud import datastore










share|improve this question
























  • What are you trying to illustrate in your example? Your for-loop keeps overwriting the values in result with the next entity. You are effectively just doing this: result = query.fetch()[-1]

    – Alex
    Mar 8 at 16:52











  • ahh yes, in my case it would only ever have one entity which again I can't, or don't know, how to access directly. query.fetch() doesn't support indexing. I wouldn't be posting this question if it did.

    – Zain Qasmi
    Mar 9 at 0:16

















1















So right now, datastore ds.query(kind='users') returns me a response in the form:



<Entity(u'users', 5633378543992832L) u'username': u'xyz', u'user_ip': '127.0.0.1', u'name': u'xyz', u'password': 'something', u'register_date': datetime.datetime(2019, 3, 8, 5, 50, 1, 443212, tzinfo=<UTC>)>


And although I can iterate it like this:



result = 
for oneItem in query.fetch():
# oneItem is Entity iterable as shown above
for oneProp in oneItem:
result[oneProp] = oneItem[oneProp]


And access any property by something = result['password']



Which works fine but it is grossly inefficient. Is there any way I can directly access any specific property without using for loops or another data structure? Similar to accessing a value in JSON.



I am using from google.cloud import datastore










share|improve this question
























  • What are you trying to illustrate in your example? Your for-loop keeps overwriting the values in result with the next entity. You are effectively just doing this: result = query.fetch()[-1]

    – Alex
    Mar 8 at 16:52











  • ahh yes, in my case it would only ever have one entity which again I can't, or don't know, how to access directly. query.fetch() doesn't support indexing. I wouldn't be posting this question if it did.

    – Zain Qasmi
    Mar 9 at 0:16













1












1








1








So right now, datastore ds.query(kind='users') returns me a response in the form:



<Entity(u'users', 5633378543992832L) u'username': u'xyz', u'user_ip': '127.0.0.1', u'name': u'xyz', u'password': 'something', u'register_date': datetime.datetime(2019, 3, 8, 5, 50, 1, 443212, tzinfo=<UTC>)>


And although I can iterate it like this:



result = 
for oneItem in query.fetch():
# oneItem is Entity iterable as shown above
for oneProp in oneItem:
result[oneProp] = oneItem[oneProp]


And access any property by something = result['password']



Which works fine but it is grossly inefficient. Is there any way I can directly access any specific property without using for loops or another data structure? Similar to accessing a value in JSON.



I am using from google.cloud import datastore










share|improve this question
















So right now, datastore ds.query(kind='users') returns me a response in the form:



<Entity(u'users', 5633378543992832L) u'username': u'xyz', u'user_ip': '127.0.0.1', u'name': u'xyz', u'password': 'something', u'register_date': datetime.datetime(2019, 3, 8, 5, 50, 1, 443212, tzinfo=<UTC>)>


And although I can iterate it like this:



result = 
for oneItem in query.fetch():
# oneItem is Entity iterable as shown above
for oneProp in oneItem:
result[oneProp] = oneItem[oneProp]


And access any property by something = result['password']



Which works fine but it is grossly inefficient. Is there any way I can directly access any specific property without using for loops or another data structure? Similar to accessing a value in JSON.



I am using from google.cloud import datastore







firebase nosql google-cloud-datastore datastore






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 9 at 1:02







Zain Qasmi

















asked Mar 8 at 8:26









Zain QasmiZain Qasmi

387




387












  • What are you trying to illustrate in your example? Your for-loop keeps overwriting the values in result with the next entity. You are effectively just doing this: result = query.fetch()[-1]

    – Alex
    Mar 8 at 16:52











  • ahh yes, in my case it would only ever have one entity which again I can't, or don't know, how to access directly. query.fetch() doesn't support indexing. I wouldn't be posting this question if it did.

    – Zain Qasmi
    Mar 9 at 0:16

















  • What are you trying to illustrate in your example? Your for-loop keeps overwriting the values in result with the next entity. You are effectively just doing this: result = query.fetch()[-1]

    – Alex
    Mar 8 at 16:52











  • ahh yes, in my case it would only ever have one entity which again I can't, or don't know, how to access directly. query.fetch() doesn't support indexing. I wouldn't be posting this question if it did.

    – Zain Qasmi
    Mar 9 at 0:16
















What are you trying to illustrate in your example? Your for-loop keeps overwriting the values in result with the next entity. You are effectively just doing this: result = query.fetch()[-1]

– Alex
Mar 8 at 16:52





What are you trying to illustrate in your example? Your for-loop keeps overwriting the values in result with the next entity. You are effectively just doing this: result = query.fetch()[-1]

– Alex
Mar 8 at 16:52













ahh yes, in my case it would only ever have one entity which again I can't, or don't know, how to access directly. query.fetch() doesn't support indexing. I wouldn't be posting this question if it did.

– Zain Qasmi
Mar 9 at 0:16





ahh yes, in my case it would only ever have one entity which again I can't, or don't know, how to access directly. query.fetch() doesn't support indexing. I wouldn't be posting this question if it did.

– Zain Qasmi
Mar 9 at 0:16












1 Answer
1






active

oldest

votes


















1














Ideally you'd be retrieving the desired object by key, instead of loading up the whole table.



It's a little challenging to give you exact code since I can't tell which library you are using to access the datastore. I've always used ndb but it seems like you are using this one:



https://googleapis.github.io/google-cloud-python/latest/datastore/index.html



To fetch by key you'd do:



from google.cloud import datastore
from google.cloud.datastore.key import Key
ds = datastore.Client()
oneItem = ds.get(Key(u'users', 5633378543992832L, project=project))


and then at this point, just interact with it oneItem['password']



The id 5633378543992832L should be supplied to you from the current session. So you'd only need to query during session creation. Something like this:



def create_session(username, raw_password):
client = datastore.Client()
query = client.query(kind=u'users')
query.add_filter('username', '=', username)
query.add_filter('password', '=', _your_password_hash_function(raw_password))
results = query.fetch(1)
if results:
return _create_session_for_user(results[0])
raise Exception("Invalid username/password")


You'll need add an index for the above query to work. It seems like your not using app engine, so you probably have to add indices through the web console






share|improve this answer























  • Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

    – Zain Qasmi
    Mar 10 at 23:42











Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55059285%2fparsing-datastore-entity-in-python-efficiently%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














Ideally you'd be retrieving the desired object by key, instead of loading up the whole table.



It's a little challenging to give you exact code since I can't tell which library you are using to access the datastore. I've always used ndb but it seems like you are using this one:



https://googleapis.github.io/google-cloud-python/latest/datastore/index.html



To fetch by key you'd do:



from google.cloud import datastore
from google.cloud.datastore.key import Key
ds = datastore.Client()
oneItem = ds.get(Key(u'users', 5633378543992832L, project=project))


and then at this point, just interact with it oneItem['password']



The id 5633378543992832L should be supplied to you from the current session. So you'd only need to query during session creation. Something like this:



def create_session(username, raw_password):
client = datastore.Client()
query = client.query(kind=u'users')
query.add_filter('username', '=', username)
query.add_filter('password', '=', _your_password_hash_function(raw_password))
results = query.fetch(1)
if results:
return _create_session_for_user(results[0])
raise Exception("Invalid username/password")


You'll need add an index for the above query to work. It seems like your not using app engine, so you probably have to add indices through the web console






share|improve this answer























  • Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

    – Zain Qasmi
    Mar 10 at 23:42















1














Ideally you'd be retrieving the desired object by key, instead of loading up the whole table.



It's a little challenging to give you exact code since I can't tell which library you are using to access the datastore. I've always used ndb but it seems like you are using this one:



https://googleapis.github.io/google-cloud-python/latest/datastore/index.html



To fetch by key you'd do:



from google.cloud import datastore
from google.cloud.datastore.key import Key
ds = datastore.Client()
oneItem = ds.get(Key(u'users', 5633378543992832L, project=project))


and then at this point, just interact with it oneItem['password']



The id 5633378543992832L should be supplied to you from the current session. So you'd only need to query during session creation. Something like this:



def create_session(username, raw_password):
client = datastore.Client()
query = client.query(kind=u'users')
query.add_filter('username', '=', username)
query.add_filter('password', '=', _your_password_hash_function(raw_password))
results = query.fetch(1)
if results:
return _create_session_for_user(results[0])
raise Exception("Invalid username/password")


You'll need add an index for the above query to work. It seems like your not using app engine, so you probably have to add indices through the web console






share|improve this answer























  • Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

    – Zain Qasmi
    Mar 10 at 23:42













1












1








1







Ideally you'd be retrieving the desired object by key, instead of loading up the whole table.



It's a little challenging to give you exact code since I can't tell which library you are using to access the datastore. I've always used ndb but it seems like you are using this one:



https://googleapis.github.io/google-cloud-python/latest/datastore/index.html



To fetch by key you'd do:



from google.cloud import datastore
from google.cloud.datastore.key import Key
ds = datastore.Client()
oneItem = ds.get(Key(u'users', 5633378543992832L, project=project))


and then at this point, just interact with it oneItem['password']



The id 5633378543992832L should be supplied to you from the current session. So you'd only need to query during session creation. Something like this:



def create_session(username, raw_password):
client = datastore.Client()
query = client.query(kind=u'users')
query.add_filter('username', '=', username)
query.add_filter('password', '=', _your_password_hash_function(raw_password))
results = query.fetch(1)
if results:
return _create_session_for_user(results[0])
raise Exception("Invalid username/password")


You'll need add an index for the above query to work. It seems like your not using app engine, so you probably have to add indices through the web console






share|improve this answer













Ideally you'd be retrieving the desired object by key, instead of loading up the whole table.



It's a little challenging to give you exact code since I can't tell which library you are using to access the datastore. I've always used ndb but it seems like you are using this one:



https://googleapis.github.io/google-cloud-python/latest/datastore/index.html



To fetch by key you'd do:



from google.cloud import datastore
from google.cloud.datastore.key import Key
ds = datastore.Client()
oneItem = ds.get(Key(u'users', 5633378543992832L, project=project))


and then at this point, just interact with it oneItem['password']



The id 5633378543992832L should be supplied to you from the current session. So you'd only need to query during session creation. Something like this:



def create_session(username, raw_password):
client = datastore.Client()
query = client.query(kind=u'users')
query.add_filter('username', '=', username)
query.add_filter('password', '=', _your_password_hash_function(raw_password))
results = query.fetch(1)
if results:
return _create_session_for_user(results[0])
raise Exception("Invalid username/password")


You'll need add an index for the above query to work. It seems like your not using app engine, so you probably have to add indices through the web console







share|improve this answer












share|improve this answer



share|improve this answer










answered Mar 9 at 0:59









AlexAlex

2,159415




2,159415












  • Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

    – Zain Qasmi
    Mar 10 at 23:42

















  • Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

    – Zain Qasmi
    Mar 10 at 23:42
















Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

– Zain Qasmi
Mar 10 at 23:42





Thanks. It worked...locally. But turns out GAE only supports ndb in the standard env which seems better among the two anyway.

– Zain Qasmi
Mar 10 at 23:42



















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55059285%2fparsing-datastore-entity-in-python-efficiently%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Save data to MySQL database using ExtJS and PHP [closed]2019 Community Moderator ElectionHow can I prevent SQL injection in PHP?Which MySQL data type to use for storing boolean valuesPHP: Delete an element from an arrayHow do I connect to a MySQL Database in Python?Should I use the datetime or timestamp data type in MySQL?How to get a list of MySQL user accountsHow Do You Parse and Process HTML/XML in PHP?Reference — What does this symbol mean in PHP?How does PHP 'foreach' actually work?Why shouldn't I use mysql_* functions in PHP?

Compiling GNU Global with universal-ctags support Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Tags for Emacs: Relationship between etags, ebrowse, cscope, GNU Global and exuberant ctagsVim and Ctags tips and trickscscope or ctags why choose one over the other?scons and ctagsctags cannot open option file “.ctags”Adding tag scopes in universal-ctagsShould I use Universal-ctags?Universal ctags on WindowsHow do I install GNU Global with universal ctags support using Homebrew?Universal ctags with emacsHow to highlight ctags generated by Universal Ctags in Vim?

Add ONERROR event to image from jsp tldHow to add an image to a JPanel?Saving image from PHP URLHTML img scalingCheck if an image is loaded (no errors) with jQueryHow to force an <img> to take up width, even if the image is not loadedHow do I populate hidden form field with a value set in Spring ControllerStyling Raw elements Generated from JSP tagds with Jquery MobileLimit resizing of images with explicitly set width and height attributeserror TLD use in a jsp fileJsp tld files cannot be resolved