CODEWITHSUNDEEP
javaniommap
marcin_koss
marcin_koss

Reputation: 5882

Reading data from memory mapped file in parallel?

If mapped file data is fully resident in physical memory will there be any benefit of reading data in parallel for example by defining number of sections with start/end byte and have separate thread working each section? The goal is to allow for frequent quick reads of data from big binary file.

I've been doing some tests (Java NIO) where each thread (testing with 4 threads) has access to reference of mmap but since each thread changes internal pointer in mmaped file to read next set of bytes this doesn't seem safe. I'm thinking about splitting a file into 4 mmaped chunks for each thread?

UPDATE: To give more context ultimately what I'm going after is having a data structure that will have a reference to number of mmaped files so then those references can be provided to some function that will do a loop scan testing for values and putting them into byte buffer.

UPDATE: This is for read-only files.

Upvotes: 3

Views: 1639

Answers (1)

Francisco Hernandez
Francisco Hernandez

Reputation: 2468

You can create different FileChannel for each thread. Each Thread will read a different part.

As documentation says, FileChannels are thread-safe.

Your code would be something like this

package nio;

import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;

public class HughTest {

    public static void main(String[] args) {

        try (FileChannel inChannel = new RandomAccessFile("file_Path", "r").getChannel()){

            // TODO Change in each thread the chunk size to read
            long fileSize = inChannel.size();
            ByteBuffer buffer = ByteBuffer.allocate((int) fileSize);
            inChannel.read(buffer);
            buffer.flip();
            // Do what you want

            inChannel.close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

This code reads a file in a unique thread, you would have adapt the code inside a runnable class and pass the FileChannel size in constructor or elsewhere to read the entire file in parallel as described in this question: Can I seek a file from different threads independently using FileChannel?

UPDATE

Unfortunately MappedByteBuffer is not thread safe as it is a subclass of Buffer, as you can see here: Does memory mapped file support concurrent get/put? so you have to use a synchronize mechanism in order to do it in parallel.

One approach would be copy the entire file in a temporal one (this way you ensure that the file will never be modified), and then use a runnable implementation like this

   private class ThreadFileRead implements Runnable {

        private final long ini;
        private final long end;

        public ThreadFileRead(long ini, long end) {
            this.ini = ini;
            this.end = end;
        }

        @Override
        public void run() {
            MappedByteBuffer out = null;

            try {
                out = new RandomAccessFile("FILEPATH", "r").
                        getChannel().map(FileChannel.MapMode.READ_ONLY, ini, end);

                for (long i = ini; i < end; i++)
                {
                    // do work
                }


            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

        }

    }

Upvotes: 2

Related Questions