Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!
Get Started for Free

distribute_load

Distribute load


Description

Utility function for distributing computations among a pool of workers for parallel processing.

Usage

distribute_load(x, n = 1)

Arguments

x

integer number of item to process.

n

integer number of threads.

Details

This function returns a list containing an element for each worker. Each element contains a integer vector specifying the indices that the worker should process.

Value

list object.

Examples

## Not run: 

# imagine that we have 10 jobs that need processing. For simplicity,
# our jobs will involve adding 1 to each element in 1:10.
values <- 1:10

# we could complete this processing using the following vectorized code
result <- 1 + 1:10
print(result)

# however, if our jobs were complex then we would be better off using
# functionals
result <- lapply(1:10, function(x) x + 1)
print(result)

# we could do one better, and use the "plyr" package to handle the
# processing
result <- plyr::llply(1:10, function(x) x + 1)
print(result)

# we could also use the parallel processing options available through "plyr"
# to use more computation resources to complete the jobs (note that since
# these jobs are very quick to process this is actually slower).
cl <- parallel::makeCluster(2, "PSOCK")
doParallel::registerDoParallel(cl)
result <- plyr::llply(1:10, function(x) x + 1, .parallel = TRUE)
cl <- parallel::stopCluster(cl)
print(result)

# however this approach iterates over each element individually, we could
# use the distribute_load function to split the N jobs up into K super
# jobs, and evaluate each super job using vectorized code.
x <- 1:10
cl <- parallel::makeCluster(2, "PSOCK")
parallel::clusterExport(cl, 'x', envir = environment())
doParallel::registerDoParallel(cl)
l <- distribute_load(length(x), n = 2)
result <- plyr::llply(l, function(i) x[i] + 1, .parallel = TRUE)
cl <- parallel::stopCluster(cl)
print(result)

## End(Not run)

prioritizr

Systematic Conservation Prioritization in R

v7.0.1
GPL-3
Authors
Jeffrey O Hanson [aut] (<https://orcid.org/0000-0002-4716-6134>), Richard Schuster [aut, cre] (<https://orcid.org/0000-0003-3191-7869>), Nina Morrell [aut], Matthew Strimas-Mackey [aut] (<https://orcid.org/0000-0001-8929-7776>), Matthew E Watts [aut], Peter Arcese [aut] (<https://orcid.org/0000-0002-8097-482X>), Joseph Bennett [aut] (<https://orcid.org/0000-0002-3901-9513>), Hugh P Possingham [aut] (<https://orcid.org/0000-0001-7755-996X>)
Initial release

We don't support your browser anymore

Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.