CODEWITHSUNDEEP
pythonsqliteconcurrencyshardinglustre
Xophmeister
Xophmeister

Reputation: 9219

SQLite "disk I/O error" with multiple readers on Lustre filesystem

I'm aware that SQLite is not ideal on a shared filesystem with multiple clients. However, the documentation implies that multiple readers should be fine.

My SQLite database resides on a Lustre volume and the database is "partitioned" -- albeit as a VIEW, rather than explicit tables, using the modulo operator on the ID to define partitions -- such that the multiple readers are mutually exclusive. This works, but in (anecdotally) about 40% of the time, I get a "disk I/O error" when reading the DB (using Python 3.7.4 with SQLite 3.30.0, using the Python stdlib sqlite3).

Is it safe to assume that the assumptions of the standard Unix VFS do not apply to Lustre filesystems (as they don't with other networked filesystems)? If that's the case, why does it matter for readers?

Assuming that SQLite on Lustre is all I have to work with, is there anything I can do to mitigate this? For example, physically separating data into multiple SQLite shards, which will only ever have one client, and perhaps ATTACHing them to a central source?

Upvotes: 0

Views: 814

Answers (1)

LustreOne
LustreOne

Reputation: 349

You didn't mention what version of Lustre you are using, which is important to know. Assuming your server version is at least 2.5 or later, I would recommend to use Lustre 2.12.3 (or later) for the client, as this fixes a number of performance and functional issues. In particular, it enables flock by default, which is often a requirement for sharing files between applications in userspace (otherwise, mount the client with the -o flock mount option). The flock functionality is often needed to ensure that multiple reads/writes of the same file are consistent.

Lustre itself is fully cache coherent between clients, and almost totally POSIX compliant (the same as a local ext4 filesystem, with a few exceptions like atime updates), which is one of the reasons that it is widely used for parallel applications. Unlike NFS, it doesn't have consistency problems for multiple writers/readers on the same file.

As for the IO error, the first thing to ask is what kind of IO is the application doing (buffered or O_DIRECT)? For O_DIRECT on non-ancient kernels read/write need to be sized/memory aligned to the disk's logical block size (might be 4096 bytes or an even higher power-of-two), so it can't do 512-byte sized O_DIRECT. It is definitely possible to have concurrent write and read operations on the same file with Lustre, but that will cause higher latency if the clients are contending on the same blocks.

Secondly, are you getting any errors on the console of the client or server when SQLite reports an error? Check the output of dmesg or /var/log/messages for LustreError: lines around the time you are running your application.

Upvotes: 1

Related Questions