Paging library storing only first fetched values in room database

Question

I have an application where i implemented paging library 3 to fetch data from api and paginate it , it works fine fetching data , the next implementation was to store the fetched data in room database , i have created the remotemediator class and wrote the code to store data , but the issue is that it stores only values of first page ( for example in my case im using the movie db api , each page fetched has 20 movies , and there are many pages ) , in my case it only saves the first 20 movies , even when i scroll , it is not storing more data , i have implemented the same exact code but seems to be the case , i faced it in an older project and now this one , i need some help , thank you in advance.

• Movies Dao

 @Dao
interface MoviesDao {

    @Query("SELECT * FROM movieTable ORDER BY id")
     fun getMovies() : PagingSource<Int,Result>


    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertMovies(result: List<Result>)


    @Query("DELETE FROM movieTable")
    suspend fun clearMovies()


}

• RemoteKeys Dao

@Dao
interface RemoteKeysDao {

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertAll(remoteKey: List<RemoteKeys>)

    @Query("SELECT * FROM remote_keys WHERE movieId = :movieId")
    suspend fun remoteKeysRepoId(movieId : Long): RemoteKeys?

    @Query("DELETE FROM remote_keys")
    suspend fun clearRemoteKeys()


}

• RemoteMediator Class

private  var MOVIES_API_STARTING_PAGE_INDEX = 1
@ExperimentalPagingApi
class MoviesMediator(
    private var authResponse: AuthResponse,
    private  var movieDatabase: MovieDatabase
) : RemoteMediator<Int,Result>() {

    override suspend fun load(loadType: LoadType, state: PagingState<Int, Result>): MediatorResult {
        val page = when (loadType) {
            LoadType.REFRESH -> {
                val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
                remoteKeys?.nextKey?.minus(1) ?: MOVIES_API_STARTING_PAGE_INDEX
            }
            LoadType.PREPEND -> {
                val remoteKeys = getRemoteKeyForFirstItem(state)
                val prevKey = remoteKeys?.prevKey
                if (prevKey == null) {
                    return MediatorResult.Success(endOfPaginationReached = remoteKeys != null)
                }
                prevKey
            }
            LoadType.APPEND -> {
                val remoteKeys = getRemoteKeyForLastItem(state)
                val nextKey = remoteKeys?.nextKey
                if (nextKey == null) {
                    return MediatorResult.Success(endOfPaginationReached = remoteKeys != null)
                }
                nextKey
            }
        }
        try {
            val response = authResponse.getMovies(Constants.API_KEY, Constants.LANGUAGE, page).results


            val endOfPagination = response.isEmpty()
            movieDatabase.withTransaction {
                // clear all tables in the database
                if (loadType == LoadType.REFRESH) {
                    movieDatabase.remoteKeysDao().clearRemoteKeys()
                    movieDatabase.MovieDao().clearMovies()
                }
                val prevKey = if (page == MOVIES_API_STARTING_PAGE_INDEX) null else page - 1
                val nextKey = if (endOfPagination) null else page + 1

                val keys = response.map {
                    RemoteKeys(movieId = it.movieID, prevKey = prevKey, nextKey = nextKey)
                }
                movieDatabase.remoteKeysDao().insertAll(keys)
                movieDatabase.MovieDao().insertMovies(response)
            }
            return MediatorResult.Success(endOfPaginationReached = endOfPagination)
        } catch (ex: Exception) {
            return MediatorResult.Error(ex)
        }
    }

    private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, Result>): RemoteKeys? {
        // Get the last page that was retrieved, that contained items.
        // From that last page, get the last item
        return state.pages.firstOrNull() { it.data.isNotEmpty() }?.data?.firstOrNull()
            ?.let { movieId ->
                // Get the remote keys of the last item retrieved
                movieDatabase.remoteKeysDao().remoteKeysRepoId(movieId.movieID)
            }
    }
    private suspend fun getRemoteKeyClosestToCurrentPosition(state: PagingState<Int, Result>): RemoteKeys? {
        // The paging library is trying to load data after the anchor position
        // Get the item closest to the anchor position
        return state.anchorPosition?.let { position ->
            state.closestItemToPosition(position)?.movieID?.let { movieId ->
                movieDatabase.remoteKeysDao().remoteKeysRepoId(movieId = movieId)
            }
        }
    }

    private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Result>): RemoteKeys? {
        // Get the last page that was retrieved, that contained items.
        // From that last page, get the last item
        return state.pages.lastOrNull() { it.data.isNotEmpty() }?.data?.lastOrNull()
                ?.let { repo ->
                    // Get the remote keys of the last item retrieved
                    movieDatabase.remoteKeysDao().remoteKeysRepoId(movieId = repo.movieID)
                }
    }

}

• Passing RemoteMediator to paging data

    val dataFlow : kotlinx.coroutines.flow.Flow<PagingData<Result>> =
        Pager(getPagingConfig(),
        remoteMediator = MoviesMediator(authResponse,movieDatabase)){
            MoviePagingSource(authResponse)
        }.flow
            .cachedIn(viewModelScope)

• Showing data in MainActivity

 @ExperimentalPagingApi
    private fun setUpAdapterOnline(){
        moviesAdapter = MoviesAdapter()
        lifecycleScope.launchWhenStarted {
            moviesModel.dataFlow.collectLatest {
                moviesAdapter.submitData(it)
            }
        }

        binding.recycler.adapter = moviesAdapter
        binding.recycler.adapter =  moviesAdapter.withLoadStateHeaderAndFooter(
            header = LoadingStateAdapter { moviesAdapter.retry() },
            footer = LoadingStateAdapter { moviesAdapter.retry() }
        )
    }

Answer 1

UPDATE

I tried using this NewsApi instead of this Unsplash Api. Even though the NewsApi provides the per_page as a query parameter, still I faced similar problem where only first 2 pages load and after that on debugging it shows that the APPEND block inside the Load method is getting called over and over and the value assigned to page is always 2. Hence the next pages aren't getting fetched at all.

ORIGINAL ANSWER

@dlam is right, you need to follow the Single source of truth principle here, and only request data from the Paging Source inside MoviesDao. I did not find the exact solution for your problem, but I do have some findings to share.

I would say that I debugged the Remote Mediator in the sample codelab here and I found that it actually increments the page as you scroll. However, when debugging your code from the github link you sent me, the value of page never increments and the APPEND block inside MoviesMediator is executed over and over, even when not scrolling. I tried to find the reason and the only thing I can come up with is the Api that you are using. I tried the same feature, saving from network to database but using the Unsplash Api and I was able to view all the pages when offline too (not just one page unlike your case).

I'm not sure if it is the per_page parameter that the codelab also passed as an argument inside the Network/Api Service Interface. In my case, the Unsplash Api also allowed to pass per_page as an argument.

Here I am attaching a link to my github repository. Try comparing the results and feel free to ask any more doubts.