When does driver datastax driver paging yields fewer pages than requested?

Question

I am trying to use datastax-driver paging using fetch-size. However datastax documentation says following

Note that setting a fetch size doesn’t mean that Cassandra will always return the exact number of rows, it is possible that it returns slightly more or less results

Don't really know the internal details of paging implementation, but can someone please clarify in what situation we get more or less results from the server? For example, if I set the fetch-size to 10, based on the above statement it's possible to get 8 or 12 rows as a result. But I am trying to understand in what situation we will receive 8 (or 12) rows?

Answer 1

Andy's answer is fairly complete, but I want to add a few more insights on why returning pages not exactly the desired size may be useful - in current or future implementations:

One reason why Cassandra may want to return short pages is filtering. Imagine that the request has ALLOW FILTERING, and needs to read a lot of data from disk just to produce a few rows that end up passing the filter and being returned to the client. The client, not aware of this, has asked for a page of 1000 rows - but in our example maybe actually generating 1000 rows passing the filter would take 10 seconds, and the client would time out if Cassandra waits 10 seconds before producing any results. So in this case, Cassandra should just return whatever rows it managed to collect before timing out - even if these are just 17 rows and not 1000 rows. The client would receive these 17 rows, and resume to the next page normally.

In the extreme case, there may be so much filtering work with so little output, that we can have a long time with not even a single row output. In this case, before timing out Cassandra may return a page with zero results, which has the has_more bit on, meaning the client should continue paging (the number of results being less than requested - or even zero - is not the sign of when to stop paging!). I'm not sure that Cassandra actually returns zero-row pages today, but Scylla (a faster Cassandra clone) definitely does, and drivers should remember to use the has_more bit as the only sign of when to stop paging.

The other question is why would paging return more rows than desired. As Andy said in his reply, I don't think this actually happens in Cassandra, nor in Scylla. But I can understand why some future implementation may want it to allow it to happen: Imagine that a coordinator needs 1000 rows for a page. So it reads up to 1000 rows from each replica, but there's inconsistent data, and one replica has an extra row, and the result is that the coordinator now has 1001 rows to return. It can (and today, does), return only the first 1000 rows but the downside is that now some of the replicas are in the wrong place in the data and will need to refind their place when asked to read the next page. Had we returned all 1001 rows we found, all of the replicas will be able to resume their reads efficiently from exactly where they left off.

Answer 2

Note that setting a fetch size doesn’t mean that Cassandra will always return the exact number of rows, it is possible that it returns slightly more or less results

I'm not confident this statement is entirely true. You can expect that its possible for a page to contain less than the desired page size. For example, if your page size is 10, and there are only 8 rows that match your query criteria, of course you will only get 8 rows back.

However, I'm not familiar of a case where the server will send back more rows than the page size in a single page result. The native protocol specification even specifies that the message returned will contain at most the page size:

If a positive value is provided for result_page_size, the result set of the RESULT message returned for the query will contain at most the result_page_size first rows of the query result.

Further, the protocol spec also states:

While the current implementation always respects the exact value of result_page_size, we reserve the right to return slightly smaller or bigger pages in the future for performance reasons.

I don't think that has been exercised, but might explain why the driver docs are phrased in this way.