OS level file I/O locks on Qt platforms

Question

Following from this question I have decided to see whether I could implement proper asynchronous file I/O using (multiple) QFiles. The idea is to use a "pool" of QFile objects operating on a single file and dispatch requests via QtConcurrent API to be executed with dedicated QFile object each. After the task would finish the result would be emitted (in case of reads) and QFile object returned to the pool. My initial tests seem to indicate that this is a valid approach and in fact does allow concurrent read/write operations (e.g. read while writing) and also that it can further help with performance (read can finish in between a write).

The obvious issue is reading and writing the same segment of the file. To see what happens I used the above mentioned approach to set up the situation and just let it write and read frantically over the same part of the file. To spot the possible "corruption" I am increasing a number at the beginning of the segment and at the end of it in the writes. The idea being that if the read ever reads different numbers at the start or at the end it can in real situation read corrupted data because it did read partially written data in such a case.

The reads and writes were overlapping a lot so I knew they were happening asynchronously and yet not a single time the output was "wrong". It basically means that the read will never read partially written data. At least on Windows. Using QIODevice::Unbuffered flag did not change it.

I assume that some kind of locking is done on the OS level to prevent this (or caching possibly?), please correct me if this assumption is wrong. I base this on a fact that a read that started after write started could finish before a write finished. Since I plan to deploy the application on other platforms as well I was wondering whether I can count on this being the case for all platforms supported by Qt (mainly those based on POSIX and Android) or I need to actually implement a locking mechanism myself for these situations - to defer reading from a segment that is being written to.

Answer 1

There's nothing in the implementation of QFile that guarantees atomicity of writes. So the idea of using multiple QFile objects to access the same sections of the same underlying file won't ever work right. Your tests on Windows are not indicative of there not being a problem, they are merely insufficient: had they been sufficient, they'd have produced the problem you're expecting.

For highly performant file access in small, possibly overlapping chunks, you have to:

1. Map the file to memory.
2. Serialize access to the memory, perhaps using multiple mutexes to improve concurrency.
3. Access memory concurrently, and don't hold the mutex while the data is paged in.

This is done by first prefetching - either reading from every page in the range of bytes to be accessed, and discarding the results, or using a platform-specific API. Then you lock the mutex and copy the data either out of the file or into it. The OS does the rest.

class FileAccess : public QObject {
  Q_OBJECT
  QFile m_file;
  QMutex m_mutex;
  uchar * m_area = nullptr;
  void prefetch(qint64 pos, qint64 size);
public:
  FileAccess(const QString & name) : m_file{name} {}
  bool open() {
    if (m_file.open(QIODevice::ReadWrite)) {
      m_area = m_file.map(0, m_file.size());
      if (! m_area) m_file.close();
    }
    return m_area != nullptr;
  }
  void readReq(qint64 pos, qint64 size);
  Q_SIGNAL readInd(const QByteArray & data, qint64 pos);
  void write(const QByteArray & data, qint64 pos);
};

void FileAccess:prefetch(qint64 pos, qint64 size) {
  const qint64 pageSize = 4096;
  const qint64 pageMask = ~pageSize;
  for (qint64 offset = pos & pageMask; offset < size; offset += pageSize) {
    volatile uchar * p = m_area+offset;
    (void)(*p);
  }
}

void FileAccess:readReq(qint64 pos, qint64 size) {
  QtConcurrent::run([=]{
    QByteArray result{size, Qt::Uninitialized};
    prefetch(pos, size);
    QMutexLocker lock{&m_mutex};
    memcpy(result.data(), m_area+pos, result.size());
    lock.unlock();
    emit readInd(result, pos);
  });
}

void FileAccess::write(const QByteArray & data, qint64 pos) {
  QtConcurrent::run([=]{
    prefetch(pos, data.size());
    QMutexLocker lock{&m_mutex};
    memcpy(m_area+pos, data.constData(), data.size());
  });
}