Ufdjrw

A Journey Through Space and Time

Could a US political party gain complete control over the government by removing checks & balances?

DOS, create pipe for stdin/stdout of command.com(or 4dos.com) in C or Batch?

Why was the small council so happy for Tyrion to become the Master of Coin?

How old can references or sources in a thesis be?

Why did the Germans forbid the possession of pet pigeons in Rostov-on-Don in 1941?

least quadratic residue under GRH: an EXPLICIT bound

What is GPS' 19 year rollover and does it present a cybersecurity issue?

New order #4: World

Extreme, but not acceptable situation and I can't start the work tomorrow morning

Can an x86 CPU running in real mode be considered to be basically an 8086 CPU?

My colleague's body is amazing

Circuitry of TV splitters

Are white and non-white police officers equally likely to kill black suspects?

What is the command to reset a PC without deleting any files

I see my dog run

I’m planning on buying a laser printer but concerned about the life cycle of toner in the machine

Are tax years 2016 & 2017 back taxes deductible for tax year 2018?

How do you conduct xenoanthropology after first contact?

Infinite past with a beginning?

How did the USSR manage to innovate in an environment characterized by government censorship and high bureaucracy?

Can you lasso down a wizard who is using the Levitate spell?

What Brexit solution does the DUP want?

N.B. ligature in Latex

Julia pmap: How to effectively send helper functions to other workers?

Julia: use of pmap with matricesJulia parallel programming - Making existing function available to all workersSending data to workersJulia - Sending GLPK.Prob to workerJulia: Variable names inside of the function need to match the names outside of the function when distributing an array among workers?Julia - Worker can't find functionJulia codegeneration with global functionJulia 0.6 pmap functionJulia - Transferring data between workersJulia pmap speed - parallel processing - dynamic programming

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

4

1

When using pmap function, an error would occur if the function is not defined on any of the worker processes. However, when the function calls other functions or using other functions inside another .jl file, to use @everywhere macro on every related functions is certainly not a good solution.

Is there a neat way to effectively send a function along with its helpers to all available workers?

parallel-processing julia

share|improve this question

edited Mar 8 at 22:25

hckr

2,4091023

asked Mar 8 at 6:16

Jay RenJay Ren

211

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Come on #parallel-processing folks — this is a perfectly fine question with a good answer. Must we put it on hold?
  
  – Matt B.
  Mar 8 at 23:06
  

  
  
  
  

add a comment | 

4

1

When using pmap function, an error would occur if the function is not defined on any of the worker processes. However, when the function calls other functions or using other functions inside another .jl file, to use @everywhere macro on every related functions is certainly not a good solution.

Is there a neat way to effectively send a function along with its helpers to all available workers?

parallel-processing julia

share|improve this question

edited Mar 8 at 22:25

hckr

2,4091023

asked Mar 8 at 6:16

Jay RenJay Ren

211

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Come on #parallel-processing folks — this is a perfectly fine question with a good answer. Must we put it on hold?
  
  – Matt B.
  Mar 8 at 23:06
  

  
  
  
  

add a comment | 

When using pmap function, an error would occur if the function is not defined on any of the worker processes. However, when the function calls other functions or using other functions inside another .jl file, to use @everywhere macro on every related functions is certainly not a good solution.

Is there a neat way to effectively send a function along with its helpers to all available workers?

parallel-processing julia

share|improve this question

edited Mar 8 at 22:25

hckr

2,4091023

asked Mar 8 at 6:16

Jay RenJay Ren

211

share|improve this question

edited Mar 8 at 22:25

hckr

2,4091023

asked Mar 8 at 6:16

Jay RenJay Ren

211

share|improve this question

edited Mar 8 at 22:25

hckr

2,4091023

asked Mar 8 at 6:16

Jay RenJay Ren

211

edited Mar 8 at 22:25

hckr

2,4091023

edited Mar 8 at 22:25

hckr

2,4091023

asked Mar 8 at 6:16

Jay RenJay Ren

211

asked Mar 8 at 6:16

Jay RenJay Ren

211

1 Answer
1

active

oldest

votes

3

I do not think there is a macro that can be used with a function definition to send the definitions of all its helper functions to all the worker processes.

However, there are better ways to send all the functions you need than putting an @everywhere before each of them.

You can put all these functions in a file and include it everywhere with @everywhere include("mynewfile.jl"). If your functions use other functions inside another .jl file, put that include statement for the other file in mynewfile.jl as well. If you are using modules from the other file, put the using or import statements inside mynewfile.jl

In a similar way, instead of a file, you can use @everywhere begin...end block. Put all these functions, using or import statements, includes etc. into a begin...end block and put an @everywhere before the begin. This is especially useful if you are working with IJulia notebooks.

julia> @everywhere begin
 g(x) = x^2
 f(x) = g(x)*2
 end

julia> pmap(f, 1:5)
5-element ArrayInt64,1:
 2
 8
 18
 32
 50

You can also create modules/packages and just use a single @eveywhere using MyNewModule. If you are using modules outside a package, you should also include the definitions/file of that module everywhere.

You might find it useful to read the relevant manual entry.

share|improve this answer

edited Mar 8 at 22:29

answered Mar 8 at 7:38

hckrhckr

2,4091023

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55057714%2fjulia-pmap-how-to-effectively-send-helper-functions-to-other-workers%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

3

I do not think there is a macro that can be used with a function definition to send the definitions of all its helper functions to all the worker processes.

However, there are better ways to send all the functions you need than putting an @everywhere before each of them.

You can put all these functions in a file and include it everywhere with @everywhere include("mynewfile.jl"). If your functions use other functions inside another .jl file, put that include statement for the other file in mynewfile.jl as well. If you are using modules from the other file, put the using or import statements inside mynewfile.jl

In a similar way, instead of a file, you can use @everywhere begin...end block. Put all these functions, using or import statements, includes etc. into a begin...end block and put an @everywhere before the begin. This is especially useful if you are working with IJulia notebooks.

julia> @everywhere begin
 g(x) = x^2
 f(x) = g(x)*2
 end

julia> pmap(f, 1:5)
5-element ArrayInt64,1:
 2
 8
 18
 32
 50

You can also create modules/packages and just use a single @eveywhere using MyNewModule. If you are using modules outside a package, you should also include the definitions/file of that module everywhere.

You might find it useful to read the relevant manual entry.

share|improve this answer

edited Mar 8 at 22:29

answered Mar 8 at 7:38

hckrhckr

2,4091023

add a comment | 

3

I do not think there is a macro that can be used with a function definition to send the definitions of all its helper functions to all the worker processes.

However, there are better ways to send all the functions you need than putting an @everywhere before each of them.

You can put all these functions in a file and include it everywhere with @everywhere include("mynewfile.jl"). If your functions use other functions inside another .jl file, put that include statement for the other file in mynewfile.jl as well. If you are using modules from the other file, put the using or import statements inside mynewfile.jl

In a similar way, instead of a file, you can use @everywhere begin...end block. Put all these functions, using or import statements, includes etc. into a begin...end block and put an @everywhere before the begin. This is especially useful if you are working with IJulia notebooks.

julia> @everywhere begin
 g(x) = x^2
 f(x) = g(x)*2
 end

julia> pmap(f, 1:5)
5-element ArrayInt64,1:
 2
 8
 18
 32
 50

You can also create modules/packages and just use a single @eveywhere using MyNewModule. If you are using modules outside a package, you should also include the definitions/file of that module everywhere.

You might find it useful to read the relevant manual entry.

share|improve this answer

edited Mar 8 at 22:29

answered Mar 8 at 7:38

hckrhckr

2,4091023

add a comment | 

I do not think there is a macro that can be used with a function definition to send the definitions of all its helper functions to all the worker processes.

However, there are better ways to send all the functions you need than putting an @everywhere before each of them.

You can put all these functions in a file and include it everywhere with @everywhere include("mynewfile.jl"). If your functions use other functions inside another .jl file, put that include statement for the other file in mynewfile.jl as well. If you are using modules from the other file, put the using or import statements inside mynewfile.jl

In a similar way, instead of a file, you can use @everywhere begin...end block. Put all these functions, using or import statements, includes etc. into a begin...end block and put an @everywhere before the begin. This is especially useful if you are working with IJulia notebooks.

julia> @everywhere begin
 g(x) = x^2
 f(x) = g(x)*2
 end

julia> pmap(f, 1:5)
5-element ArrayInt64,1:
 2
 8
 18
 32
 50

You can also create modules/packages and just use a single @eveywhere using MyNewModule. If you are using modules outside a package, you should also include the definitions/file of that module everywhere.

You might find it useful to read the relevant manual entry.

share|improve this answer

edited Mar 8 at 22:29

answered Mar 8 at 7:38

hckrhckr

2,4091023

julia> @everywhere begin
 g(x) = x^2
 f(x) = g(x)*2
 end

julia> pmap(f, 1:5)
5-element ArrayInt64,1:
 2
 8
 18
 32
 50

share|improve this answer

edited Mar 8 at 22:29

answered Mar 8 at 7:38

hckrhckr

2,4091023

answered Mar 8 at 7:38

hckrhckr

2,4091023

answered Mar 8 at 7:38

hckrhckr

2,4091023