CODEWITHSUNDEEP
linuxcachingiofilesystemslvm
Ahhmyface
Ahhmyface

Reputation: 61

Is it necessary to invalidate/drop cache to read when using O_DIRECT to write?

I've encountered a race condition with LVM and a disk driver I'm working on. It looks like things like vgcreate and lvcreate do their IO in O_DIRECT mode. I discovered this when running those commands with -vvv.

Clearing start of logical volume "test"
    /dev/Finance-PG-vg/test: Added to device cache
    Opened /dev/Finance-PG-vg/test RW O_DIRECT
    Wiping /dev/Finance-PG-vg/test at sector 0 length 8 sectors
    /dev/Finance-PG-vg/test: block size is 4096 bytes
    Closed /dev/Finance-PG-vg/test

Specifically, I suspect that our reads are hitting the cache, and not getting the latest disk contents.

If something is written with O_DIRECT, my understanding is that this bypasses the cache. Therefore any reads to that sector are going to recieve the old data from cache, at least until the cache is invalidated. So if I want to read whatever O_DIRECT just wrote within a few seconds, I should be dropping the cache first?

Correct?

Upvotes: 2

Views: 256

Answers (1)

Radu
Radu

Reputation: 1128

There are several confusions here:

  • The tools you mention mostly likely use O_DIRECT to make sure the new LVM configuration is persistent. The LVM metadata is actually stored in a specific location on all the physical disks/partitions you provide.

  • Writing to LVM devices does not use by default O_DIRECT (although you can pass this flag when you open a file).

  • Bypassing the cache with O_DIRECT does not mean that you get stale data. Lets assume that you open a file, write to it, close it, then open it again with O_DIRECT, and read the file. The read is guaranteed to return the latest changes to the file. There is no stale data ever returned. There is no need to drop caches when using O_DIRECT.

Upvotes: 1

Related Questions