How to get all TeamDrives, Folders, and Files with Google Drive API (using pagination)

Question

I have to go through all team drives, folders, and files and the code I have works perfectly fine:

 /// <summary>
        /// Get All files inside a folder
        /// </summary>
        /// <param name="service"></param>
        /// <param name="folderId"></param>
        /// <param name="td"></param>
        /// <returns></returns>
        private static async Task<Google.Apis.Drive.v3.Data.FileList> GetAllFilesInsideFolder(DriveService service, string folderId, TeamDrive td)
        {
            string FolderId = folderId;
            // Define parameters of request.
            FilesResource.ListRequest listRequest = service.Files.List();
            listRequest.Corpora = "drive";
            listRequest.SupportsAllDrives = true;
            listRequest.DriveId = td.Id;
            listRequest.PageSize = 100;
            listRequest.IncludeItemsFromAllDrives = true;
            listRequest.Q = "'" + FolderId + "' in parents and trashed=false";
            listRequest.Fields = "nextPageToken, files(*)";

            // List files.
            Google.Apis.Drive.v3.Data.FileList files = await listRequest.ExecuteAsync();

            return files;
        }

        /// <summary>
        /// Gets all folders from an specific team drive
        /// </summary>
        /// <param name="service"></param>
        /// <param name="td"></param>
        /// <returns></returns>
        private static async Task<Google.Apis.Drive.v3.Data.FileList> GetAllFoldersFromTeamDriveAsync(DriveService service, TeamDrive td)
        {
            try
            {
                var request = service.Files.List();
                request.Corpora = "drive";
                request.SupportsAllDrives = true;
                request.DriveId = td.Id;
                request.PageSize = 100;
                request.IncludeItemsFromAllDrives = true;
                request.Q = "mimeType = 'application/vnd.google-apps.folder'";
                var response = await request.ExecuteAsync();
                return response;
            }
            catch (Exception ex )
            {
                Console.WriteLine("GetAllFoldersFromTeamDriveAsync Error: " + ex.Message);
                throw;
            }
        }


        /// <summary>
        /// Gets all teamdrives
        /// </summary>
        /// <param name="service"></param>
        /// <returns></returns>
        private static async Task<Google.Apis.Drive.v3.Data.TeamDriveList> GetAllTeamDrivesAsync(DriveService service)
        {
            try
            {
                var request = service.Teamdrives.List();
                request.PageSize = 100;
                var response = await request.ExecuteAsync();
                return response;
            }
            catch (Exception ex)
            {
                Console.WriteLine("GetAllTeamDrivesAsync Error" + ex.Message);
                throw;
            }
        }

Main method:

public async Task  RunFix()
        {
            UserCredential credential;
            Directory.CreateDirectory(StorageLocation);
            var DataStore = new FileDataStore(StorageLocation);
            using var stream = new FileStream(CredentialsString, FileMode.Open, FileAccess.Read);

            // Please give consent to the Auth/drive scope
            credential = GoogleWebAuthorizationBroker.AuthorizeAsync(
                GoogleClientSecrets.Load(stream).Secrets,
                Scopes,
                Username,
                CancellationToken.None,
                DataStore).Result;
            try
            {
                var service = new DriveService(new BaseClientService.Initializer()
                {
                    HttpClientInitializer = credential,
                    ApplicationName = ApplicationName,
                });

                var clientbillCycleGoogleDriveFilesDEV = new TableClient("...");

                // Get All team drives
                var allTeamDrives = await GetAllTeamDrivesAsync(service);

                // We will use a CSV to copy the unmapped files
                var csvUnmappedBillCycleFiles = new StringBuilder();

                //Iterate over all Team Drives I have access to:
                foreach (TeamDrive td in allTeamDrives.TeamDrives)
                {
                    if (td.Name == "XXX")
                    {
                        Console.WriteLine("Team Drive: " + td.Name);
                        Stopwatch sAllFilesInsideFolder = Stopwatch.StartNew();

                        // Get All Folders inside this team drive
                        var allFolders = await GetAllFoldersFromTeamDriveAsync(service, td);
                        foreach (Google.Apis.Drive.v3.Data.File file in allFolders.Files)
                        {
                            Console.WriteLine("--Folder-- " + file.Name);
              
                            if (file.Name.StartsWith("xyz"))
                            {
                                 //some business logic
                              
                                foreach (Google.Apis.Drive.v3.Data.File googlefile in allFilesInsideFolder.Files)
                                {
                                   //some business logic
                                }
                       
                            }
                        }

  
                    }
                }

                System.IO.File.WriteAllText("csvUnmappedBillCycleFiles.csv", csvUnmappedBillCycleFiles.ToString());
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error" + ex.Message);
            }
        }

The problem is that I cant set PageSize to something above 100.

How can I change this code to actually to go through everything?

Answer 1

Assuming that you are using drives list and files list. Which is just a guess since you haven't included the code GetAllTeamDrivesAsync? GetAllFoldersFromTeamDriveAsync.

You can set the page size to max 1000 on your file.list method.

var request = service.Files.List();   
request.PageSize = 1000;

But thats you are still going to need to paginate over the page. I recommend using the pageStreamer. It will loop over each of the rows for you.

 var request = service.Files.List();
 request.PageSize = 1000;
            
 var pageStreamer =
                new Google.Apis.Requests.PageStreamer<Google.Apis.Drive.v3.Data.File, FilesResource.ListRequest,
                    Google.Apis.Drive.v3.Data.FileList, string>(
                    (req, token) => request.PageToken = token,
                    response => response.NextPageToken,
                    response => response.Files);
            
 // data storage
 var all = new Google.Apis.Drive.v3.Data.FileList();
 all.Files = new List<Google.Apis.Drive.v3.Data.File>();
            
 // fetching data and adding it to storage
 foreach (var result in await pageStreamer.FetchAllAsync(request, CancellationToken.None))
            {
                all.Files.Add(result);
            }

I have a video up that explains how pagestreamer works How to use the nextpagetoken from the files list method in the Google Drive api v3.

You should be able to use pagestreamer with drives list as well.