CODEWITHSUNDEEP
linuxcachingdiskprefetchnas
Jacko
Jacko

Reputation: 13195

Linux: need to design pre-fetcher to cache files from NAS into system memory

I am designing a server for the following scenario:

  1. a series of single images are stored on a NAS, lets say 100 of them
  2. a client connects to the server over TCP socket and requests image39
  3. server reads image39 from NAS and sends back to client over socket
  4. it is likely that the client will also request other images from the series, so:
  5. I would like to launch a thread that iterates through the images, reads them, and does a cat image39 > /dev/null to force cache into memory on server
  6. thread will fetch images as follows: image38, image40, image37, image41, etc.
  7. already fetched images are ignored
  8. if client now requests image77, I want to reset the fetch thread to fetch: image76, image78, etc.

This has to scale to many series and clients. Probably on the order of 1000 concurrent prefetches. I understand that threads can cause performance hit if there are too many. Would it be better to fork a new process instead? Is there a more efficient way than threads or processes ?

Thanks!!!

Upvotes: 0

Views: 302

Answers (1)

John Zwinck
John Zwinck

Reputation: 249434

This is premature optimization. Try implementing your system without tricks to "force" the cache, and see how it works. I bet it'll be fine--and you won't then need to worry about nasty surprises if it turns out your tricks don't play nice with other things on the system.

Upvotes: 1

Related Questions