What is the throughput of YCSB benchmark result?

Question

I run YCSB for MongoDB.

This is my result:

./bin/ycsb run mongodb -s -P workloads/workloada > my_run.dat
Loading workload...
Starting test.
 0 sec: 0 operations; 
 10 sec: 16690 operations; 1657.07 current ops/sec; [READ AverageLatency(us)=490.92] [UPDATE AverageLatency(us)=654.23] 
 20 sec: 34651 operations; 1784.68 current ops/sec; [READ AverageLatency(us)=489.92] [UPDATE AverageLatency(us)=612.62] 
 30 sec: 53152 operations; 1848.99 current ops/sec; [READ AverageLatency(us)=490.31] [UPDATE AverageLatency(us)=577.33] 
 40 sec: 69624 operations; 1645.55 current ops/sec; [READ AverageLatency(us)=538.23] [UPDATE AverageLatency(us)=661.69] 
 50 sec: 87998 operations; 1836.67 current ops/sec; [READ AverageLatency(us)=486.72] [UPDATE AverageLatency(us)=590.1] 
 60 sec: 105686 operations; 1768.62 current ops/sec; [READ AverageLatency(us)=507.04] [UPDATE AverageLatency(us)=611.8] 
 70 sec: 125513 operations; 1982.7 current ops/sec; [READ AverageLatency(us)=451.16] [UPDATE AverageLatency(us)=546.62] 
 80 sec: 144607 operations; 1909.21 current ops/sec; [READ AverageLatency(us)=471.99] [UPDATE AverageLatency(us)=565.08] 
 90 sec: 162982 operations; 1836.95 current ops/sec; [READ AverageLatency(us)=479.93] [UPDATE AverageLatency(us)=598.32] 
 100 sec: 182047 operations; 1906.31 current ops/sec; [READ AverageLatency(us)=464.83] [UPDATE AverageLatency(us)=573.04] 
 110 sec: 200449 operations; 1837.81 current ops/sec; [READ AverageLatency(us)=482.29] [UPDATE AverageLatency(us)=593.68] 
 120 sec: 219938 operations; 1948.71 current ops/sec; [READ AverageLatency(us)=450.92] [UPDATE AverageLatency(us)=565.71] 
 130 sec: 238601 operations; 1866.11 current ops/sec; [READ AverageLatency(us)=470.68] [UPDATE AverageLatency(us)=591.11] 
 140 sec: 259370 operations; 2076.69 current ops/sec; [READ AverageLatency(us)=423.18] [UPDATE AverageLatency(us)=529.47] 
 150 sec: 280439 operations; 2106.69 current ops/sec; [READ AverageLatency(us)=427.67] [UPDATE AverageLatency(us)=511.45] 
 160 sec: 300454 operations; 2001.3 current ops/sec; [READ AverageLatency(us)=443.84] [UPDATE AverageLatency(us)=536.78] 
 170 sec: 320651 operations; 2019.5 current ops/sec; [READ AverageLatency(us)=441.01] [UPDATE AverageLatency(us)=550.97] 
 180 sec: 339846 operations; 1919.31 current ops/sec; [READ AverageLatency(us)=467.42] [UPDATE AverageLatency(us)=564.23] 
 190 sec: 358263 operations; 1839.13 current ops/sec; [READ AverageLatency(us)=479.94] [UPDATE AverageLatency(us)=595.92] 
 200 sec: 377686 operations; 1941.72 current ops/sec; [READ AverageLatency(us)=457.56] [UPDATE AverageLatency(us)=561.6] 
 210 sec: 398299 operations; 2060.89 current ops/sec; [READ AverageLatency(us)=433.26] [UPDATE AverageLatency(us)=526.97] 
 220 sec: 416290 operations; 1798.92 current ops/sec; [READ AverageLatency(us)=495.91] [UPDATE AverageLatency(us)=604.73] 
 230 sec: 436193 operations; 1989.11 current ops/sec; [READ AverageLatency(us)=451.31] [UPDATE AverageLatency(us)=543.84] 
 240 sec: 457690 operations; 2149.49 current ops/sec; [READ AverageLatency(us)=411.25] [UPDATE AverageLatency(us)=508.66] 
 250 sec: 476801 operations; 1910.91 current ops/sec; [READ AverageLatency(us)=452.22] [UPDATE AverageLatency(us)=583.65] 
 260 sec: 494604 operations; 1780.12 current ops/sec; [READ AverageLatency(us)=497.51] [UPDATE AverageLatency(us)=612.22] 
 270 sec: 515721 operations; 2111.49 current ops/sec; [READ AverageLatency(us)=419.5] [UPDATE AverageLatency(us)=518.25] 
 280 sec: 533880 operations; 1815.54 current ops/sec; [READ AverageLatency(us)=486.85] [UPDATE AverageLatency(us)=604.71] 
 290 sec: 553822 operations; 1994 current ops/sec; [READ AverageLatency(us)=444.74] [UPDATE AverageLatency(us)=548.16] 
 300 sec: 576101 operations; 2227.01 current ops/sec; [READ AverageLatency(us)=398.8] [UPDATE AverageLatency(us)=489.87] 
 310 sec: 594411 operations; 1826.43 current ops/sec; [READ AverageLatency(us)=472.85] [UPDATE AverageLatency(us)=610.28] 
 313 sec: 600000 operations; 1792.5 current ops/sec; [READ AverageLatency(us)=496.76] [UPDATE AverageLatency(us)=602.38] 

Then the output file:

YCSB Client 0.1
Command line: -db com.yahoo.ycsb.db.MongoDbClient -s -P workloads/workloada -t
new database url = 172.17.0.13:27017/test
mongo connection created with 172.17.0.13:27017/test
[OVERALL], RunTime(ms), 313358.0
[OVERALL], Throughput(ops/sec), 1914.7428819433364
[READ], Operations, 299981
[READ], AverageLatency(us), 462.1898386897837
[READ], MinLatency(us), 88
[READ], MaxLatency(us), 35020
[READ], 95thPercentileLatency(ms), 1
[READ], 99thPercentileLatency(ms), 2
[READ], Return=0, 299981
[READ], 0, 278042
[READ], 1, 18615
[READ], 2, 2170
[READ], 3, 493
[READ], 4, 264
[READ], 5, 120
[READ], 6, 81
[READ], 7, 37
[READ], 8, 27
[READ], 9, 14
[READ], 10, 14
[READ], 11, 13
[READ], 12, 14
[READ], 13, 8
[READ], 14, 15
[READ], 15, 14
[READ], 16, 15
[READ], 17, 9
[READ], 18, 4
[READ], 19, 1
[READ], 20, 4
[READ], 21, 1
[READ], 22, 2
[READ], 23, 1
[READ], 24, 0
[READ], 25, 0
[READ], 26, 0
[READ], 27, 0
[READ], 28, 0
[READ], 29, 0
[READ], 30, 0
[READ], 31, 1
[READ], 32, 1
[READ], 33, 0
[READ], 34, 0
[READ], 35, 1
[READ], 36, 0
[READ], 37, 0
[READ], 38, 0
[READ], 39, 0
[READ], 40, 0
[READ], 41, 0
[READ], 42, 0
[READ], 43, 0
[READ], 44, 0
[READ], 45, 0
[READ], 46, 0
[READ], 47, 0

I want to plot the like this https://i.sstatic.net/xZkSe.jpg

However, I am confused about the output of the result. Not sure what the "throughput" as the figure means in the result. Besides, for the

10 sec: 16690 operations; 1657.07 current ops/sec;

Doesn't this 1657.07*10(sec)=16570.7 operations?

Answer 1

Read this section of YCSB wiki to understand how to interpret the results.

More specifically, your first snippet is basically the instantaneous output turned on by the '-s' switch. I am not sure about YCSB but usually in performance tests such instantaneous results are mere indicators and shouldn't be accepted to be extremely accurate.

Now, coming to the plot you want to draw (Throughput vs Latency for a given Workload across multiple database providers) you will need to use the '-target' switch available in YCSB. Varying the throughput using that switch, capturing the final log output and then mapping 95 or 99% percentile latency figures against your target throughputs should give you the plot you need.

Hope this helps.

Aside: Performance runs should have long enough runtimes. Results of a perf test that runs for only 5 mins can be disqualified for umpteen reasons. Load enough data (recordcount property of YCSB) and then set ops (operationcount property) to a large enough number so that your test runs for at least 30 mins.

Answer 2

I think the first number is the number of operations done in the past ten seconds. The second is the number of operations in the past second. The number of operations varies from second to second, so 10 * current ops/s will not necessarily be the number of operations over the past 10 seconds. The average number of operations per second has been, on average, higher in the past ten seconds than in this particular second, since 16690 / 10 > 1657.07.