R mclapply return serialization object size limit using version 4.3.0 on Apple M1 aarch64-apple-darwin20

Question

The mclapply help page says:

"Prior to R 3.4.0 and on a 32-bit platform, the serialized result from each forked process is limited to 2^31 - 1 bytes."

I have this problem even with version 4.3.0 on a 64 bit Apple M1 Pro laptop using a 64-bit build of R.

The following example demonstrates the fail when the return object crosses over the integer overflow size. Note it seems to work when the sent objects are over this limit but fails when the return objects are over the limit.

library(parallel)

## just change numeric -> integer to reduce object size for testing

test_function=function(input) {return(as.integer(input))}  


## This code works using mclapply and 2 cores

test_object=rep(1.0,5.3e8)
as.numeric(object.size(test_object))/1e9

## input object size is 4.24e9 Bytes (well above 2^31)

out=mclapply(list(test_object,test_object),test_function,mc.cores=2)

as.numeric(object.size(out[[1]]))/1e9

## each output object size is 2.12e9 Bytes (just smaller than integer overflow)


## And this code also works using mclapply and 1 core (it is really using lapply since only 1 core)

test_object=rep(1.0,5.4e8)  #slight increase in size
as.numeric(object.size(test_object))/1e9

## input object size is 4.32e9 Bytes (well above 2^31)

out=mclapply(list(test_object,test_object),test_function,mc.cores=1)

as.numeric(object.size(out[[1]]))/1e9

## each output object size is 2.16e9 Bytes (just larger than integer overflow)


## But this code fails using mclapply and 2 cores

test_object=rep(1.0,5.4e8)  #slight increase in size
as.numeric(object.size(test_object))/1e9

## input object size is 4.32e9 Bytes (well above 2^31)

out=mclapply(list(test_object,test_object),test_function,mc.cores=2)

## Warning message:
## In mclapply(list(test_obect, test_obect), test_function, mc.cores = 2) :
##   scheduled cores 1, 2 did not deliver results, all values of the jobs will be affected