Repo search access control and rate limiting


#1

Hi! We’ve been trying to port over our GitHub API usage from the OAuth App model to a GitHub App. We’ve run into a few sticking points, one in particular being around repo search:

Our product has a search box component that allows users to search their GitHub repos and jump to them for quick access. With the current (OAuth app) API, we use the /search/repositories API to search over all repositories, and since we request the private repo scope, the user’s private repositories show up in the search results. Each user is granted 30 requests per minute per the GitHub search strict rate-limit restrictions.

The issues we’re having with switching over to GitHub App-based search:

  1. Currently the /search/repositories API is only accessible using an installation token, not an app user token. This means that search results are not tailored to a specific user. With the private repositories permission enabled, this also means that private repositories that the org for that installation has access to, but not necessarily any given user of that installation are returned in the results. So the issue is there is no way to search for repositories on an installation while respecting user-specific repo access controls.
    A user-to-server endpoint for repo search that is access-controlled would be ideal.

  2. Since the only way to search for repositories is via an installation, all users of that installation are subject to a single, global strict rate limit. After testing this I found that a X-Ratelimit-Limit [30] header is returned for search requests, regardless of how many users exist in the org associated with the installation that made the search request. 30 requests per minute is way too small of a number to support an installation with many users. Since with the OAuth model grants each user token 30 requests, I don’t see why each user of an installation can’t be given their own user-specific rate limit as well, or have the global rate limit for an installation scale with each additional user like it does for the other endpoints.

Having these issues resolved would help us get towards the point where we can smoothly transition over to supporting GitHub apps without negatively impacting our users.


#2

Thanks so much for the detailed and thoughtful feedback, @renfredxh!

I’ll ask the team to consider enabling the search endpoints for user identity tokens so that all the right permission-based filtering can kick in, and also ask them to consider whether we can offer higher rate limits for search (e.g. based on the number of users). I can’t promise an ETA, but we’ll followup as soon as there’s any news.


#3

@renfredxh It’s possible to use the OAuth token of a user with this endpoint. This would solve the rate-limit issue since you are using a different token per user. Note: you need to use the OAuth credentials associated to the Github App to get a valid token (they can be found on the settings page of the app, below the private key).

There is one problem though: to get the list of all repositories for a given user, you’ll need to iterate over all the installations returned by this endpoint. If the user belongs to many organizations, or is an external contributor on many public repositories, that’s a long list of API calls.

@github-staff +1 for the following endpoint (I have a scenario similar to @renfredxh)

# Authorization: JWT XXX (a JWT token from the App)
GET /user/app/repositories?access_token=...

That would return all the repositories that the user can access via the Github App.


Filter repositories by name
#4

@bdelbasso Thanks, I’m aware of this endpoint. That’s a repository list API, not a repo search endpoint which is what I’m looking for. List doesn’t accept the same query parameters and doesn’t perform the same filtering/ranking etc. that happens on GitHub’s end.