API error when fetching repository PRs


#1

error code: E7F6:7B68:F1C332:22523CC:5B239416

The repository is pretty big so having seen a similar thread below I reduced the pagination from the default 100 to 50 then 20 then 10, the error still occurs (on the 6th page as of a pagesize of 10).

Not sure if that’s of interest but the failing query takes ~10s to reply while the successful ones took 3s or so to fetch and process.

The query worked fine when I last used it a few months (?) back.

query($owner: String!, $name: String!, $cursor: String, $pagesize: Int!) {
  rateLimit { remaining }
  repository(owner: $owner, name: $name) {
    pullRequests(last: $pagesize, before: $cursor) {
      pageInfo { startCursor hasPreviousPage }
      nodes {
        author { # optional
          login
        }
        number
        title
        body
        state
        repository { nameWithOwner }
        baseRefName
        headRefOid
        headRepositoryOwner { # optional
          login
        }
        headRefName
        headRef { # optional
          target {
            ... on Commit {
              status {
                contexts {
                  context
                  state
                }
              }
            }
          }
        }
        commits { totalCount }
      }
    }
  }
}

#2

Are you able to narrow it down to a single repository that is problematic? In other words, if you were to get 1 per page, are there some pages that are fast and others that are slow?


#3

@kytrinyx well odoo/odoo is definitely problematic in the general case, there doesn’t seem to be any specific PRs and testing/iterating likely caches various information on your side so the pages which can be reached changes as time goes on and new tests are performed (especially the small-page ones which I guess manage to trickle-fill various caches which followup large-page runs can then leverage?). For instance last run the early limit was ~4000 but yesterday evening (CET) it was around 2200 and yesterday morning things would start failing between 300 and 400 PRs.

A pagination of 1 (and sometimes but not always 2) seems to succeed in getting everything, but it basically takes days to fetch everything. And a pagination of 5 or below triggers some sort of spam protection in my tests so I’ve had to slow it down by adding a 1s delay between every page.

Here’s attempts of running with paginations of 100, 20, 5 and 2, with a graph of average response time per PR (1s delay excluded), and the error codes for those which eventually failed (note: the discrepancy between the first and second line of each section is that the first is generated by tqdm and does not strip out the 1s spam protection delay between each page, the second excludes the delay, so does the graph):

Fetching 100 prs/page: 4100pr [02:05, 37.99pr/s]

Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include FF5D:7DF9:DBA286:1A525B5:5B2925C1 when reporting this issue.
4100 prs fetched in 74.0s

Fetching 20 prs/page: 4120pr [05:41, 12.38pr/s]

Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include FF5D:7DF9:DC7804:1A6D6D6:5B292717 when reporting this issue.
4120 prs fetched in 125.0s

Fetching 5 prs/page: 4350pr [26:33, 1.87s/pr]

Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include FF5D:7DF9:E05FCA:1AE841A:5B292D51 when reporting this issue.
4350 prs fetched in 712.0s

Fetching 2 prs/page: 15081pr [7:59:27, 2.37s/pr]

15081 prs fetched in 21218.0s

Fetching 1 prs/page: 7712pr [5:55:18, 2.94s/pr]

2.718281828459045 (unclear what that is)
7712 prs fetched in 13591.0s

graph

Sadly the order of operation is … not great and I didn’t think to save the raw data, but we can just see the “Pagination 2” sequence starting with a low lower-bound, then shooting up (disappearing behind the big purple blob) a bit after 4000 prs fetched, slightly further than the 100/20/5 paginations get killed entirely.