Should REST API quotas be above 5k for larger projects?


#1

Happy holidays! Our organization has a private Jenkins setup in an intranet, so webhooks are not an option and we have to use polling to discover commits and PRs. Jobs are generally created via Jenkins Pipelines and Github Organization Folder technologies, so we told the Jenkins server once which GHO to scan (and then to check out code over SSH), and then it is all automatic. But as we have more and more components and branches and PRs to them, a limit arises…

While we’re trying to balance the frequency of polls vs. latency of CI, we sometimes hit the REST API quota of apparently 5000 points (the Jenkins plugins actually seem to implement a lower watermark internally as some “budget” that the scanning job exceeds while there are some 1000-2000GH points remaining, and I don’t know if they use caching requests, but that’s not a question for you I guess).

The question is whether our setup is eligible for a higher quota? According to https://developer.github.com/apps/building-github-apps/rate-limits-for-github-apps/ our reading was that having many collaborators and repo components in our organization (42ity) should have added 50 requests per hour for each to the quotas - but I don’t see it in practice. Does that note only apply to organizations with paid accounts?

Also, the documentation is not quite clear - does it add 50 pts for each of the total head/repo counts, or just for those above the base 20 items?


#2

To use the webhooks we used a reverse proxy called Ngrok . Setup was easy and we had no issues triggering builds in our CI system.


#3

The problem with hooks is that a public IP address is needed, and the higher blessing from corporate to use one, to essentially publish a service to the creepy scary wild internet. This can take years. Literally. Overheads of big company.


#4

I went through the same pain but Reverse proxy ngrok will give you a public IP immediately and made my life much easier. You don’t need to worry about purchasing one.


#5

Ok, I misread the reference I guess and assumed it was a common proxy like nginx. Looking at it closer, seems intriguing, thanks for the idea!


#6

Have you looked into conditional requests? If there is no change and the server returns a 304, it does not count against a quota


#7

I’ve read into that on Github docs, yes.
I did not yet get definite info from Jenkins IRC (and had no time to deep-dive into code so far) whether Jenkins plugins use that. I think at some point the SCM API had support for caching (HTTP requests-responses as a whole), then it caused issues and was dropped…

I think for this use-case, it would suffice for their GitHub plugin to ask its questions with If-Modified-Since timestamp of last run of this job – no real caching needed, right?


#8

Yes, exactly. You would send the timestamp since the last time you pulled in new data and the GitHub API will respond with 304 if there are no new changes, which will not count against your quota


#9

By the way, do you know if there are plans for some SSE or similar support in Github? So a hook-less client like Jenkins behind firewalls would open a persistent HTTP connection and get essentially webhook messages through that channel?


#10

I’m not aware of anything official, but there is some experimentation in that direction from the folks working on Probot. Check out https://smee.io/