CODEWITHSUNDEEP
arraysjuliamemory-mapping
Eliassj
Eliassj

Reputation: 1

Julia mmap: Fast(er) way of reading columnwise?

Disclaimer: I'm new to both Julia and memory mapping.

I'm mapping large files (~80Gb) consisting of recordings from ~400 channels using Mmap.mmap. Since the values are saved in channel order 1 timepoint at a time the obvious way to shape the memory map is (n channels, n timepoints). However, reading an entire channel from this map requires reading a single row and 100000000+ columns. I only have to do this for one channel but reading in this way is way slower compared to if I could read rowwise.

Structure of current memory map:

ch1: t1 t2 t3 t4 t5 ...
ch2: t1 t2 t3 t4 t5 ...
ch3: t1 t2 t3 t4 t5 ...
ch4: t1 t2 t3 t4 t5 ...
.
.
.

I have gotten similar performance from the following:

Reading each value in for loops:

ch = channels to read from
len = Time window (tmin:tmax)
res::Matrix{Int16} = Matrix{Int16}(undef, length(len), length(ch))

for (nch, c) in enumerate(ch)
        for (ntim, t) in enumerate(len)
            res[ntim, nch] = memorymap[c, t]
        end
    end

Using slicing:

ch = channels to read from
len = Time window (tmin:tmax)
res::Union{Matrix{Int16}, Vector{Int16}} = memorymap[ch, len]

Both methods are inside a function whose other parts are creating the memory map(using another function) and checking so the channels to read from and tmin/tmax exists. I will probably try multithreading but a more efficient method would be good/better.

EDIT: Switched place of rowwise/columnwise. Reading rowwise is obviously faster.

Upvotes: 0

Views: 113

Answers (0)

Related Questions