How to get multipage JSON results from Google and Zendesk APIs

This post is about two ways to handle pagination (getting multipage results) in response to queries on Google and Zendesk APIs

While most Web API provides multi page results to a call, most API documentation will only talk about how to get the first page of those results.

It is almost impossible to find guides to pagination!

Zendesk gets a B

Here is an example from Zendesk:

If at all they have instructions on pagination, it will be hidden away somewhere in the API documentation

Here’s the documentation on pagination on Zendesk. This one was hard to find but not impossible.

Google gets an F

Sometimes the instructions may be hidden way deeper than necessary. See below for a search on “pagination” on http://developers.google.com.

I am assuming the documentation people leave the details out because for most professional developers this is trivial.

Not so for a newbie. If you are a code n00b, you are have struggled with the stubborn OAuth, pored over API docs to get the perfect call that will get you the data for your app, and finally late in the evening, you are getting the data back! The last thing you want is to cross yet another hurdle.

So here’s what I have: There are two common ways API designers tell you that there are more pages of results.

One is via a next page field in the result, which will contain the url for the next page. If there are no next pages the field will not be present in the result. Zendesk uses this approach.

The other way is to return the token for the next page, which you append to your next call. Most Google APIs uses this approach. Twitter uses a similar next cursor key.

In either case the solution boils down to the following

1. Have an array to store all the results

2. Make a call and get the result and push them to the array

3. Enter a loop if the next page field or page token is present

4. Make the new call either using the next page URL or the token

5. Get the result

6. Optionally — give the server a few milliseconds break. Be nice.

7. Here’s what I did for Zendesk. I used a do-while loop, but while-do works just fine of course.

function zd_get (cmd) {
  var url = ZD_BASE+cmd;
  var results =[];
  var service = getService();

  if (service.hasAccess()) {
    do {
      try {
        var response = UrlFetchApp.fetch(url, {headers: {'Authorization': 'Bearer ' + service.getAccessToken(),}});
      }
      catch (e) {
        return e;

      }
      var result = JSON.parse(response.getContentText());
      results.push(result);
      url = result['next_page'];
      Utilities.sleep(250); // give the server a break

    }  while (result['next_page'] !=null);
    return results;
  }
  else return null;

}

1. Here’s what I did for a Google API call

function zd_get (cmd) {
  var url = ZD_BASE+cmd;
  var results =[];
  var service = getService();

  if (service.hasAccess()) {
    do {
      try {
        var response = UrlFetchApp.fetch(url, {headers: {'Authorization': 'Bearer ' + service.getAccessToken(),}});
      }
      catch (e) {
        return e;

      }
      var result = JSON.parse(response.getContentText());
      results.push(result);
      url = result['next_page'];
      Utilities.sleep(250); // give the server a break

    }  while (result['next_page'] !=null);
    return results;
  }
  else return null;

}