memory issues with google_matrix Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Very large matrices using Python and NumPyDifference between numpy.array shape (R, 1) and (R,)Python NetworkX creating graph from incidence matrixScaling NetworkX nodes and edges proportional to adjacency matrixPython networkx: grouping nodes based on a node attributeHow shall I resolve this “Memory Error” in numpy?calculation of degree from a weighted adjacency matrix in networkxDistinguish classes with different colors NetworkxEffective algorithms to find k clique communitieshow do I compress matrix to avoid memory error?
Why can't I install Tomboy in Ubuntu Mate 19.04?
Co-worker has annoying ringtone
How to write capital alpha?
In musical terms, what properties are varied by the human voice to produce different words / syllables?
1-probability to calculate two events in a row
macOS: Name for app shortcut screen found by pinching with thumb and three fingers
Why do early math courses focus on the cross sections of a cone and not on other 3D objects?
Is multiple magic items in one inherently imbalanced?
What makes a man succeed?
How does Belgium enforce obligatory attendance in elections?
Why are vacuum tubes still used in amateur radios?
Do I really need to have a message in a novel to appeal to readers?
Putting class ranking in CV, but against dept guidelines
Is it possible for SQL statements to execute concurrently within a single session in SQL Server?
Flash light on something
Deconstruction is ambiguous
Project Euler #1 in C++
Does the Mueller report show a conspiracy between Russia and the Trump Campaign?
Amount of permutations on an NxNxN Rubik's Cube
What is the difference between a "ranged attack" and a "ranged weapon attack"?
Can a sorcerer use careful spell on himself?
Did Mueller's report provide an evidentiary basis for the claim of Russian govt election interference via social media?
Did any compiler fully use 80-bit floating point?
What's the point of the test set?
memory issues with google_matrix
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
Data science time! April 2019 and salary with experience
The Ask Question Wizard is Live!Very large matrices using Python and NumPyDifference between numpy.array shape (R, 1) and (R,)Python NetworkX creating graph from incidence matrixScaling NetworkX nodes and edges proportional to adjacency matrixPython networkx: grouping nodes based on a node attributeHow shall I resolve this “Memory Error” in numpy?calculation of degree from a weighted adjacency matrix in networkxDistinguish classes with different colors NetworkxEffective algorithms to find k clique communitieshow do I compress matrix to avoid memory error?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I am trying to infer construct a transition matrix with rows 956104. This is a huge matrix.
Number of Nodes: 303413
Number of Edges : 875191
I tried numpy way first creating a zero matrix using (np.zeros), this obviously creates memory issues (956104*956104).
I also tried networkX google_matrix which is purely a numpy based. This also creates memory Error. I have 16 GB of memory. Is there a way I could construct transition matrix?
python-3.x numpy networkx
asked Mar 8 at 21:54
lptlpt
347111
add a comment |
I am trying to infer construct a transition matrix with rows 956104. This is a huge matrix.
Number of Nodes: 303413
Number of Edges : 875191
I tried numpy way first creating a zero matrix using (np.zeros), this obviously creates memory issues (956104*956104).
I also tried networkX google_matrix which is purely a numpy based. This also creates memory Error. I have 16 GB of memory. Is there a way I could construct transition matrix?
python-3.x numpy networkx
asked Mar 8 at 21:54
lptlpt
347111
numpy.memmap() try this....
– Justice_Lords
Mar 9 at 12:52
add a comment |
I am trying to infer construct a transition matrix with rows 956104. This is a huge matrix.
Number of Nodes: 303413
Number of Edges : 875191
I tried numpy way first creating a zero matrix using (np.zeros), this obviously creates memory issues (956104*956104).
I also tried networkX google_matrix which is purely a numpy based. This also creates memory Error. I have 16 GB of memory. Is there a way I could construct transition matrix?
python-3.x numpy networkx
asked Mar 8 at 21:54
lptlpt
347111
I am trying to infer construct a transition matrix with rows 956104. This is a huge matrix.
Number of Nodes: 303413
Number of Edges : 875191
I tried numpy way first creating a zero matrix using (np.zeros), this obviously creates memory issues (956104*956104).
I also tried networkX google_matrix which is purely a numpy based. This also creates memory Error. I have 16 GB of memory. Is there a way I could construct transition matrix?
python-3.x numpy networkx
python-3.x numpy networkx
asked Mar 8 at 21:54
lptlpt
347111
asked Mar 8 at 21:54
lptlpt
347111
asked Mar 8 at 21:54
lptlpt
347111
asked Mar 8 at 21:54
lptlpt
347111
asked Mar 8 at 21:54
lptlpt
347111
347111
numpy.memmap() try this....
– Justice_Lords
Mar 9 at 12:52
add a comment |
numpy.memmap() try this....
– Justice_Lords
Mar 9 at 12:52
numpy.memmap() try this....
– Justice_Lords
Mar 9 at 12:52
numpy.memmap() try this....
– Justice_Lords
Mar 9 at 12:52
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55071528%2fmemory-issues-with-google-matrix%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55071528%2fmemory-issues-with-google-matrix%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
numpy.memmap() try this....
– Justice_Lords
Mar 9 at 12:52