I often use the %dopar% operator in from the foreach package to run code in parallel. However, the only other use on the box seemingly installed a few items and suddenly %dopar% will use far more than the number of cores I am specifying and seems to load balance between all of them. The issue with this is that it doesn't actually seem to perform non-trivial tasks at all anymore.
This is an example of testing code I've been using for testing the %dopar% loop.
Code: Select all
library(iterators)
library(foreach)
library(doParallel)
library(Parallel)
nCores <- 4
cl <- makeCluster(nCores)
registerDoParallel(cl)
trials = 100000
x <- iris[which(iris[,5] != "setosa"),c(1,5)]
t2 <- system.time({
r2 <- foreach(icount(trials), .combine=cbind) %dopar% {
ind <- sample(100,100,replace= TRUE)
results1 <- glm(x[ind,2]~x[ind,1],family=binomial(logit))
coefficients(results1)
}
})[3]
stopCluster(cl)
I have checked the system logs and can't seem to find what's been changed so it's unclear to me exactly how to proceed. It is almost reminiscent of a change in the BLAS library but this behavior persists after recompiling R to not use the shared BLAS lib.
here is the output from lsof -p 23618 | grep 'blas\|lapack'
Code: Select all
R 39781 mem REG 8,17 3576576 1055038 /usr/local/lib/R/lib/libRlapack.so
R 39781 mem REG 8,17 655336 1572936 /usr/lib/libblas/libblas.so.3.0
Code: Select all
linux-vdso.so.1 (0x00007ffca59fd000)
libR.so => /usr/local/lib/R/lib/libR.so (0x00007f15fdc30000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f15fd9f7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f15fd7da000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15fd431000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f15fd190000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f15fce72000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15fcb71000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f15fc933000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f15fc6e9000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f15fc4c6000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f15fc2bd000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f15fc0b9000)
libicuuc.so.48 => /usr/lib/x86_64-linux-gnu/libicuuc.so.48 (0x00007f15fbd4a000)
libicui18n.so.48 => /usr/lib/x86_64-linux-gnu/libicui18n.so.48 (0x00007f15fb97e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f15fe1cf000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f15fb768000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f15fb53d000)
libicudata.so.48 => /usr/lib/x86_64-linux-gnu/libicudata.so.48 (0x00007f15fa1cd000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15f9ec2000)
Is it possible to rebuild R without any shared library's? I am open to any suggestions at this point.
Thanks for your help guys!