How can I correctly use custom PagingSource with PagingDataAdapter, on local data?

Question

The Problem

I have locally-generated data that I need to display in a RecyclerView. I tried to use a custom PagingSource with PagingDataAdapter to reduce the amount of data in memory, but I get visual effects when I invalidate the data, for example if I insert or delete one item:

• when the visible items belong to 2 different "pages", many unchanged items flash as if they had been modified
• if all the visible items belong to the same page, everything is fine. Only the inserted / deleted item shows an animation.

I took an example application referenced by Google's doc (PagingSample) to test the concept. The original with Room does not show artefacts, but my modified version with custom PagingSource does.

The code generated and used by Room is too complicated to see any difference that would explain the issue.

My data must be locally-generated, I can't use Room as a work-around to display them.

My question

How can I properly define a PagingSource for my local data, and use it with PagingDataAdapter without visual glitches?

Optionally, how can I know when data is discarded (so I can discard my local data as well)?

Code excerpts and details

The full example project is hosted here: https://github.com/blueglyph/PagingSampleModified

Here is the data:

    private val _data = ArrayMap<Int, Cheese>()
    val data = MutableLiveData <Map<Int, Cheese>>(_data)
    val sortedData = data.map { data -> data.values.sortedBy { it.name.lowercase() } }

and the PagingSource. I'm using key = item position. I have tried with key = page number, each page containing 30 items (10 are visible), but it does not change anything.

    private class CheeseDataSource(val dao: CheeseDaoLocal, val pageSize: Int): PagingSource<Int, Cheese>() {
        fun max(a: Int, b: Int): Int = if (a > b) a else b

        override fun getRefreshKey(state: PagingState<Int, Cheese>): Int? {
            val lastPos = dao.count() - 1
            val key = state.anchorPosition?.let { anchorPosition ->
                val anchorPage = state.closestPageToPosition(anchorPosition)
                anchorPage?.prevKey?.plus(pageSize)?.coerceAtMost(lastPos) ?: anchorPage?.nextKey?.minus(pageSize)?.coerceAtLeast(0)
            }
            return key
        }

        override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Cheese> {
            val pageNumber = params.key ?: 0
            val count = dao.count()
            val data = dao.allCheesesOrdName().drop(pageNumber).take(pageSize)
            return LoadResult.Page(
                data = data,
                prevKey = if (pageNumber > 0) max(0, pageNumber - pageSize) else null,
                nextKey = if (pageNumber + pageSize < count) pageNumber + pageSize else null
            )
        }
    }

The Flow on PagingData is created in the view model:

    val pageSize = 30
    var dataSource: PagingSource<Int, Cheese>? = null
    val allCheeses: Flow<PagingData<CheeseListItem>> = Pager(
        config = PagingConfig(
            pageSize = pageSize,
            enablePlaceholders = false,
            maxSize = 90
        )
    ) {
        dataSource = dao.getDataSource(pageSize)
        dataSource!!
    }.flow
        .map { pagingData -> pagingData.map { cheese -> CheeseListItem.Item(cheese) } }

with dao.getDataSource(pageSize) returning the CheeseDataSource shown above.

and in the activity, data pages are collected and submitted:

        lifecycleScope.launch {
            viewModel.allCheeses.collectLatest { adapter.submitData(it) }
        }

When the data is modified, an observer triggers an invalidation:

        dao.sortedData.observeForever {
            dataSource?.invalidate()
        }

The scrolling and loading of pages is fine, the only problems come when invalidate is used and when items from 2 pages are displayed simultaneously.

The adapter is classic:

class CheeseAdapter : PagingDataAdapter<CheeseListItem, CheeseViewHolder>(diffCallback) {
...
    companion object {
        val diffCallback = object : DiffUtil.ItemCallback<CheeseListItem>() {
            override fun areItemsTheSame(oldItem: CheeseListItem, newItem: CheeseListItem): Boolean {
                return if (oldItem is CheeseListItem.Item && newItem is CheeseListItem.Item) {
                    oldItem.cheese.id == newItem.cheese.id
                } else if (oldItem is CheeseListItem.Separator && newItem is CheeseListItem.Separator) {
                    oldItem.name == newItem.name
                } else {
                    oldItem == newItem
                }
            }
            override fun areContentsTheSame(oldItem: CheeseListItem, newItem: CheeseListItem): Boolean {
                return oldItem == newItem
            }
        }
...

What I have tried (among many other things)

• use LiveData instead of Flow
• using / removing the cache on the Flow
• removing the observers and invalidating directly in insert/delete functions to make the code more direct
• instead of key = position, using key = page number (0, 1, 2, ...) with each page containing pageSize=30 items

At this point, I'm not sure anymore that paging-3 is meant to be used for custom data. I'm observing so many operations for a simple insert/delete, such as 2000-4000 compare ops in the adapter, reloading 3 pages of data, ... that using ListAdapter directly on my data and doing the load/unload manually seems a better option.

Answer 1

UPDATE July 2022

There are some problems with paging-3. The most noticeable ones are

• failure to properly load the dataset when using an initial position with initialKey (see initial_key branch)
• failure to properly jump to another position with scrollToPositionWithOffset or similar function (see test_scroll branch)

getRefreshKey function is called with weird (sometimes negative) values of anchorPosition, and load is called multiple times to load unnecessary data - sometimes loading iteratively every page from the end or from the beginning.

I believe the same problem can be observed when the PagingSource is generated by Room.

=> In short, it can only be used if you need to display the top items first, and if you don't need to jump to a particular position (seeing other items is only done when the user manually scrolls through the list). A fix is on the way but it could take time; it's still only alpha and compiles for a specific set of library versions / SDK.

You'll find another, simpler algorithm for getRefreshKey and load in the mentioned branches, in CheeseDao.kt.

End of update

---

I finally found out a possible solution, though I'm not sure it will work if the Paging-3 library is updated. I don't know yet if the behaviour explained above is due to bugs / limitations in the Paging-3 components or if that's just a bad explanation in the reference documentation, or something I missed entirely.

I. Work-around for the glitch issue.

1. In PagingConfig, we must have enablePlaceholders = true or it simply won't work correctly. With false I'm observing the scrollbar jumping on each load operation when scrolling up/down, and inserting items at the end will make all items glitch on the display, then the list will jump all the way to the top.

2. The logic in getRefreshKey and load, as shown in Google's guide and reference documentation, is naive and will not work with custom data. I had to modify them as follows (modifications have been pushed in the github example):

        override fun getRefreshKey(state: PagingState<Int, Cheese>): Int? = state.anchorPosition

        override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Cheese> {
            data class Args(
                var start: Int = 0, var size: Int = 0, var prevKey: Int? = null, var nextKey: Int? = null,
                var itemsBefore: Int = UNDEF, var itemsAfter: Int = UNDEF
            )
            val pos = params.key ?: 0
            val args = Args()
            when (params) {
                is LoadParams.Append -> {
                    args.start = pos
                    args.prevKey = params.key
                    //args.nextKey = if (args.start < count) min(args.start + params.loadSize, count) else null
                    args.nextKey = if (args.start + params.loadSize < count) args.start + params.loadSize else null
                }
                is LoadParams.Prepend -> {
                    args.start = max(pos - pageSize, 0)
                    args.prevKey = if (args.start > 0) args.start else null
                    args.nextKey = params.key
                }
                is LoadParams.Refresh -> {
                    args.start = max((pos - params.loadSize/2)/pageSize*pageSize, 0)
                    args.prevKey = if (args.start > 0) args.start else null
                    args.nextKey = if (args.start + params.loadSize < count) min(args.start + params.loadSize, count - 1) else null
                }
            }
            args.size = min(params.loadSize, count - args.start)
            if (params is LoadParams.Refresh) {
                args.itemsBefore = args.start
                args.itemsAfter = count - args.size - args.start
            }
            val source = dao.allCheesesOrdName()
            val data = source.drop(args.start).take(args.size)
            if (params.key == null && data.count() == 0) {
                return LoadResult.Error(Exception("Empty"))
            }
            val result = LoadResult.Page(
                data = data,
                prevKey = args.prevKey,
                nextKey = args.nextKey,
                itemsBefore = args.itemsBefore,
                itemsAfter = args.itemsAfter
            )
            return result
        }

I had to deduce that from the behaviour of Room-generated PagingSource, by inserting a wrapper between the DAO code and the view model to observe the params and LoadResult values.

Notes

• The commented line in LoadParam.Append mimics the Room-generated code behaviour, which is slightly incorrect when loading the last page. It makes the Pager loads an empty page at the end, which is not a serious issue but triggers unnecessary operations in the whole chain.
• I'm not sure about the Refresh case, it was hard to induce any logic from Room's code behaviour. Here I'm placing the params.key position in the middle of the loaded range (params.loadSize items). At worst it will append data in a 2nd load operation.

II. Predicting the discarded data

This should be possible from the params given to the load function. In a simplified heuristic (the ranges must be min/max'ed to get actual indices):

LoadParams.Append  -> loads [key .. key+loadSize[, so discard [key-maxSize..key-maxSize+loadSize[
LoadParams.Prepend -> loads [key-loadSize .. key[, do discard [key-loadSize+maxSize..key+maxSize[
LoadParams.Refresh -> discard what is not reloaded

The Args.start .. Args.start + Args.size in the code above can be used as the range which is kept, from there it is easy to deduce what is discarded.