CODEWITHSUNDEEP
filesystemsbigdatalarge-datalarge-data-volumeslustre
hrs
hrs

Reputation: 507

Lustre: Sending different write requests to different OSTs

I have a typical scenario where there can write requests in parallel, and each file is a few hundred GBs in size.

My test system, a Lustre file system has 4 OSTs (3TB each) and 1 MDS.

What I practically observed is that with striping disabled, Lustre writes all the files onto a single OST unless it is full.

Is it possible to configure Lustre such that when parallel or overlapping write requests arise, the MDS automatically chooses the target that is not currently busy?

I am intrigued as to why Lustre is not doing this by default, or maybe its because I have missed out on something?

Upvotes: 1

Views: 547

Answers (2)

LustreOne
LustreOne

Reputation: 349

If you typically have large files on your filesystem (e.g. > 32MB) you should set the default stripe count for the filesystem appropriately. You can specify the default striping on a per-directory, per-filesystem basis (depending on your permissions and what your usage is), or when an individual file is created.

Use lfs setstripe -c N /path/to/directory to set the default stripe count to N on directory, which will affect all new files in that directory only. If you only have a limited number of OSTs (4 in your case), or all of your files are huge (in the GB range in your case) then you can use -c -1, which means "stripe over all OSTs", rather than an explicit stripe count.

Use lfs setstripe -c N /mnt/lustre (or whatever the mountpoint of your filesystem is) to set the default stripe count for new files in the whole filesystem (unless otherwise specified by a directory-level default layout, or explicitly at file creation).

With Lustre 2.10 and later, you can use composite file layouts so that the stripe count increases as files get larger, like:

lfs setstripe -E64M -c 1 -E4G -c 4 -E-1 -S 4M -c -1

which means that for the first 64MB of a file use only 1 stripe, for the 64MB-4GB part of the file use 4 stripes, and beyond 4GB use a stripe size of 4MB and stripe over all OSTs (which in your case is also 4, but on some filesystems it might be hundreds of OSTs).

Upvotes: 2

rread
rread

Reputation: 159

Yes, if you enable striping, then Lustre will distribute writes to an individual file across as many OSTs as you want (depending on how many stripes you set). Since you have disable striping, then all IO for a file will be limited to a single OST.

If I misunderstood and instead what you are seeing is all IO for all files sent to the same OST, then you have probably inadvertently set the "index" option to a value other than -1. This option will override the default setting which will allow the MDS to balance file creation across all OSTs.

Upvotes: 1

Related Questions