CODEWITHSUNDEEP
linuxio
Utoah
Utoah

Reputation: 1072

Why reading disk more frequently makes each read more fast on Linux? QPS1 vs. 50

I was benchmarking the synchronous read performance on a linux box with SATA disk. With a single thread reading, the weird thing is that a higher QPS(50) gave a average read time of 12ms after reading 300 entries, while a lower QPS(1) gave 63ms after reading the same 300 entries. Is there some explanation?

Code and data follows:

struct Request{
    unsigned long long len;
    unsigned long long offset;
    int                fd;
};

int read_request(Request* request){
    char* buf = (char*)malloc(request->len);
    off_t of = lseek(request->fd,request->offset,SEEK_SET);
    assert(of == request->offset);

    int len = read(request->fd,buf,request->len);
    assert(len == request->len);
    free(buf);
    return 0;
}




 int read_with_qps(Request* request,int request_num,Files* f,int mode,int qps){

    int interval = 1000 / qps;
    struct timeval start,end;
    for(int i  = 0 ; i < request_num ; i++){
        gettimeofday(&start,NULL);
        int ret = read_request(&request[i]);
        gettimeofday(&end,NULL);
        int time_used = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_usec - start.tv_usec)/1000;
        fprintf(stderr,"%lld,offset=%lld,len=%lld, read time:%d,ret=%d,mode=%d\n",
                end.tv_sec,request[i].offset,request[i].len,time_used,ret,mode);
        if(time_used < interval){
            usleep((interval - time_used) * 1000);
        }
    }
    return 0;
}

With QPS=50, a sample of the output looks like(ignore time < 4ms which is considered to hit page cache when calculating avg time):

1332233329,offset=1052299215,len=13186, read time:13,ret=0,mode=1
1332233329,offset=2319646140,len=1612, read time:10,ret=0,mode=1
1332233330,offset=1319250005,len=5654, read time:12,ret=0,mode=1
1332233330,offset=2520376009,len=2676, read time:12,ret=0,mode=1
1332233330,offset=2197548522,len=17236, read time:10,ret=0,mode=1
1332233330,offset=1363242083,len=13734, read time:11,ret=0,mode=1
1332233330,offset=4242210521,len=2003, read time:17,ret=0,mode=1
1332233330,offset=1666337117,len=2207, read time:10,ret=0,mode=1
1332233330,offset=797722662,len=5480, read time:18,ret=0,mode=1
1332233330,offset=1129310678,len=2265, read time:10,ret=0,mode=1

QPS=1, the same extract of smaple:

1332300410,offset=1052299215,len=13186, read time:19,ret=0,mode=1
1332300411,offset=2319646140,len=1612, read time:40,ret=0,mode=1
1332300412,offset=1319250005,len=5654, read time:141,ret=0,mode=1
1332300413,offset=2520376009,len=2676, read time:15,ret=0,mode=1
1332300414,offset=2197548522,len=17236, read time:21,ret=0,mode=1
1332300415,offset=1363242083,len=13734, read time:13,ret=0,mode=1
1332300416,offset=4242210521,len=2003, read time:43,ret=0,mode=1
1332300417,offset=1666337117,len=2207, read time:18,ret=0,mode=1
1332300418,offset=797722662,len=5480, read time:67,ret=0,mode=1
1332300419,offset=1129310678,len=2265, read time:12,ret=0,mode=1

kernel version is: 2.6.18-194.el5 SMP x86_64

$ cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

Thanks for reply

Upvotes: 1

Views: 156

Answers (2)

ephemient
ephemient

Reputation: 204926

blktrace can tell you for sure, but it's possible that this is due to plugging. In short, IO requests may be slightly delayed before being sent to disks, which is beneficial when many requests are coming in and can be merged, but not so much otherwise.

Upvotes: 1

Jim Garrison
Jim Garrison

Reputation: 86774

When you issue a bunch of queries the drive firmware can queue them and execute them in an optimized sequence based on rotational position and head position ("elevator seeking"), so it doesn't have to wait a full seek time or disk rotation time for every i/o request.

If you issue the same queries slowly there's no such advantage.

Upvotes: 1

Related Questions