Google Script - exceeded maximum execution time

Question

I have built a script that pulls from multiple spreadsheets and calculates total values. Basically what happens is multiple employees will input hours performed on a particular task for a particular customer. I then want to calculate how much work was done for a particular customer on a master spreadsheet and pull some analytics.

So on the master spreadsheet, there is a tab for each customer. This function takes the sheet names from the master spreadsheet and creates an array of all of the customer names. The reason why I do this is so that if a new tab is created, that customer name will automatically be included in the customer array.

function getCurrentCustomers() {
  var sheets = servicesSpreadsheet.getSheets();
  for (var i=0 ; i < sheets.length ; i++) {
    if (sheets[i].getName() != "Services" && sheets[i].getName() != "Employee Files") {
      currentCustomers.push(sheets[i].getName());
    };
  };
};

The next function takes a look at all of the files in a particular Google Drive folder and returns the IDs in an array. This allows me to create a copy of an employee spreadsheet stored in this particular folder, and that spreadsheet's values will automatically be calculated as they change.

function listFilesInFolder() {
  var folder = DriveApp.getFolderById("0B7zrWIHovJrKVXlsaGx0d2NFT2c");
  var contents = folder.getFiles();

  var cnt = 0;
  var file;

  while (contents.hasNext()) {
    //Finds the file in the specified folder
    var file = contents.next();
    //Increases the count
    cnt++;
    //Gets the Id of the file
    data = [
      file.getName(),
      file.getId()
      ];
    //Appends it to the employeeId list
    employeeIds.push(data);
  };
  return employeeIds;
};

The last function is the one that is slowing it down a great deal.

First I create an array for all of the possible services. Unfortunately there are 137 individual services.

Then I loop through all of the customers. For each customer, I loop through every service to see if it appears on any of the employees spreadsheets. I am thinking there is a more efficient way to do this. The Google Script times out before I even get through one full customer. Also, I haven't even included the spreadsheets for all of the employees yet. I am just testing using dummy data.

function calculateNumbers(){
  var allServices = servicesSpreadsheet.getSheetByName("Services").getRange("Z2:Z137").getValues();
  Logger.log(allServices);
  Logger.log(allServices[0][0]);
  employeeList = listFilesInFolder();

  //Gets services spreadsheet range


  /*Loops through all of the current customers (currentCustomers comes from function getCurrentCustomers)*/
  for (var c = 0; c < currentCustomers.length; c++) {

    var currentCustomer = currentCustomers[c];
    var lastColumn = servicesSpreadsheet.getSheetByName(currentCustomer).getLastColumn();
    var servicesRange = SpreadsheetApp.openById("1X3RRR3UVeot-DYCyXOsfVo0DoKjHezltwBPwUm8ZYig").getSheetByName(currentCustomer).getRange("A4:BC227").getValues();  

    //Loops through all of the services
    var serviceTotal = 0;    
    for (var service = 0; service < allServices.length; service++){

      //Loops through employee spreadsheet Ids
      for (var i = 0; i < employeeList.length; i++) {
        //Get employee spreadsheet ID
        var spreadsheetId = employeeList[i][1];

        //Open the employee spreadsheet by ID
        var employeeSpreadsheet = SpreadsheetApp.openById(spreadsheetId);

        //Get the sheets from the particular employee spreadsheet
        var sheets = employeeSpreadsheet.getSheets();

        //Gets the name of each sheet in the employee spreadsheet
        var sheetsName = [];
        for (var j = 0; j < sheets.length; j++) {
          sheetsName.push(sheets[j].getName());
        };

        //Loops through all of the sheets in an employee spreadsheet ignoring the Services spreadsheet
        for (var q = 0; q < sheetsName.length; q++) {
          if (sheetsName[q] != "Services") {

            //Gets the range of the spreadsheet
            var range = employeeSpreadsheet.getSheetByName(sheetsName[q]).getRange("A5:E1000").getValues();

            //Loops through the range to see if range matches service and customer
            for (var r = 0; r < range.length; r++) {
              if (range[r][3] == allServices[service][0] && range[r][1] == currentCustomer) {
                serviceTotal += range[r][4];
              }; 
            };
          };
        };         
      };

      //Adds the service total to the correct customer's spreadsheet
      for (var serviceServices = 4; serviceServices <= servicesRange.length; serviceServices++){
        var end = 0;
        if (end > 0) {break}
        else if (allServices[service][0] == servicesSpreadsheet.getSheetByName(currentCustomer).getRange(serviceServices,1).getValues()) {          
          servicesSpreadsheet.getSheetByName(currentCustomer).getRange(serviceServices,6).setValue(serviceTotal);
          end += 1;
        };
      };
    };

  }; 
};

The first image shows what an employee spreadsheet looks like. The second shows what an individual customer sheet looks like. There are many customer sheets on the master spreadsheet.

One of the things you will notice, is that each service has a category. I was thinking maybe to check the category and then check the particular service. There are 23 categories.

I was also hoping there was a way to only look at services that have actually had work done on them. If no one has ever done a Campaign Setup, maybe that could be ignored.

Any help would be greatly appreciated. I apologize for the lengthy post!

Answer 1

You are calling services inside of loops multiple times. These can take up to a couple seconds each, and are generally considered to be very slow. This is against Apps Script best practice as it greatly increases your execution time.

TL;DR: You are calling Apps Script services potentially thousands of times. Each call can take up to a couple seconds to execute. You need to cache your service calls outside of your loops, otherwise your performance is going to suffer horribly.

Examples:

1:

    if (sheets[i].getName() != "Services" && sheets[i].getName() != "Employee Files")

Create and set a variable with the sheet name once, and check that instead of calling the getName() method twice. This isn't a huge deal, but will increase execution time.

2:

This is a biggie, as it's one level deep in your loop in calculateNumbers

var lastColumn = servicesSpreadsheet.getSheetByName(currentCustomer).getLastColumn();
var servicesRange = SpreadsheetApp.openById("1X3RRR3UVeot-DYCyXOsfVo0DoKjHezltwBPwUm8ZYig").getSheetByName(currentCustomer).getRange("A4:BC227").getValues();

2a: You are opening a new spreadsheet, opening a new worksheet, and then getting a range for the same sheet, and getting the values of that range once per loop for your servicesRange. These service calls will stack up quick, and bloat your execution time.

2b: I see you are getting the lastColumn, but I don't see it used anywhere? Maybe I missed something, but it going unused while making a service call for each loop will add even more to your execution time.

3:

This is massive, you are potentially calling Apps Script services thousands or tens of thousands of times here. This snippet is already two loop levels deep.

//Loops through all of the sheets in an employee spreadsheet ignoring the Services spreadsheet
        for (var q = 0; q < sheetsName.length; q++) {
          if (sheetsName[q] != "Services") {

            //Gets the range of the spreadsheet
            var range = employeeSpreadsheet.getSheetByName(sheetsName[q]).getRange("A5:E1000").getValues();

            //Loops through the range to see if range matches service and customer
            for (var r = 0; r < range.length; r++) {
              if (range[r][3] == allServices[service][0] && range[r][1] == currentCustomer) {
                serviceTotal += range[r][4];
              }; 
            };
          };
        };         
      };

      //Adds the service total to the correct customer's spreadsheet
      for (var serviceServices = 4; serviceServices <= servicesRange.length; serviceServices++){
        var end = 0;
        if (end > 0) {break}
        else if (allServices[service][0] == servicesSpreadsheet.getSheetByName(currentCustomer).getRange(serviceServices,1).getValues()) {          
          servicesSpreadsheet.getSheetByName(currentCustomer).getRange(serviceServices,6).setValue(serviceTotal);
          end += 1;
        };
      };
    };

You have multi-level nested loops in nested loops calling the same services. If you had a thrice nested loop with each level being iterated 10 times, and the bottom level calling a service once per loop. You would be calling an Apps Script service 1,000 times. Being conservative, at let's say 0.5 seconds per service call, you would already have 8.3 minutes of execution time.

Cache your service calls, and perform bulk operations.