CODEWITHSUNDEEP
amazon-s3cachingdirectorys3fs
user14389292
user14389292

Reputation:

S3Fs directory listings slow - caching somehow possible?

I'm wondering if there is any way to practically speedup directory listings of a s3fs mount? I have a WebDAV server, only for read operations, that basically access my s3fs mount. The problem is that listing directories is slow, while transfer speed is fine.

So I started to look a bit around the web a stumbled across "JuiceFS", sadly this was also not an option for several reasons. Then I tried "vmtouch" to index the mounted s3 storage to local memory, this is also not working as it's a shared resourced managed by the fuse kernel extension.

Even using S3FS built-in cache does not solve the issue, instead it makes it even worse as the file first getting downloaded from s3 into the cache locally and then served via WebDav ...

Is there no way to just speedup directory listing using S3? Basically, this is all I need in the end and no fancy POSIX compatible Block Device like JuiceFS which basically creates its own logic on top of your s3 bucket ... Not what I was searching for.

Upvotes: 2

Views: 3061

Answers (1)

Andrew Gaul
Andrew Gaul

Reputation: 2402

Unfortunately s3fs 1.91 has poor readdir performance. There are a few open issues and pull requests that track future improvements:

You can toggle #2-4 via command-line flags today but #5 is still in-progress. #1 is the big win that would give a 100x speedup but trades off less POSIX compatibility, e.g., no UID/GID, no permissions. One alternative that you can try today is goofys which implements #1.

Upvotes: 1

Related Questions