Business


Migrating to GitHub Apps — Code Climate shares their story

We first started talking about GitHub Apps in 2017 as the recommended way for developers to build integrations on GitHub. With GitHub Apps, developers can use either GitHub’s REST or GraphQL APIs to interact with the GitHub ecosystem and provide a more flexible security model for users. Since announcing, we have released new documentation and guides to help simplify the process of building new integrations or migrating existing OAuth Apps and talking to integrators about their experiences. For those considering building or migrating to GitHub Apps, we thought it might be interesting to share one of those stories from Code Climate.

Code Climate has built on GitHub from the very beginning. Now they’re using GitHub Apps to build bots, manage server-to-server integrations, limit permission scopes, and provide better support to the engineers using their products. Here’s how they build with GitHub Apps, and what they learned in the process, in the words of Kathryn Chen, VP of Product.

Code Climate Switches to GitHub Apps

Code Climate switched to GitHub Apps

Why are we so excited about GitHub Apps?

Though GitHub OAuth provides great capabilities for managing user-initiated actions, it presented us with a few limitations for building server-to-server integrations.

Using GitHub Apps to build bots

In our Quality product, we wanted to provide users with an Automated Code Review feature. Before GitHub Apps, we used individual user credentials to retrieve and post data into GitHub. We couldn’t post review comments on our customers’ pull requests as a Code Climate bot. GitHub Apps solve this problem.

A Bot found code similarities and suggests refactoring

Using GitHub Apps to manage server-to-server integrations

Some of our customers have immediate security policies that don’t allow us to use SSH to clone repository data from GitHub. Prior to GitHub Apps, we implemented HTTPS repository data cloning by rotating through various users’ OAuth access tokens. This means that these HTTPS calls are associated with specific GitHub users, even though they are not initiated by them. With GitHub Apps, we can authenticate to pull this type of data as a service.

Using GitHub Apps to limit permission scope

In the world of GitHub OAuth, read and write access to repository data are bundled together. GitHub Apps provides much more granularity in permissions. We are now able to ask for only the access level we need (read-only) on the specific resources we need (e.g. pull requests). This aligns the requested permission with the services we provide, making our users more comfortable.

Permission request to install Code Climate Velocity testing

How did we go about implementing GitHub Apps?

We found that switching to GitHub Apps did not require our existing system to be substantially restructured. The API endpoints and queries that we implemented via OAuth remained the same for GitHub Apps, and only our method of authentication changed.

For querying, we use GitHub’s graphql-client gem, with our client defined as:

class HTTP < ::GraphQL::Client::HTTP
  def initialize(vcs, access_token)
    super(vcs.graphql_api_url)
    @access_token = access_token
  end
  
  def headers(context)
    {}.tap do |h|
      if (token = context.fetch(:access_token, @access_token))
        h.merge!("Authorization" => "Bearer #{token}")
      end
    end
  end
end

With OAuth, for access_token, we provided the authorized OAuth token of a random user in the organization. Randomization helped us to balance usage across the organization, but was not ideal. We wanted to make the API request on behalf of the organization itself, which is where GitHub Apps came in.

With GitHub Apps, access is provided via short-lived, renewable tokens belonging to the installation, and the installation is granted the appropriate permissions. We introduced a component in our code that would generate a token for the organization’s GitHub App installation using the octokit gem and the credentials of our GitHub App:

  def create_token
    create_token_client.create_app_installation_access_token(
      external_database_id,
      accept: Octokit::Preview::PREVIEW_TYPES[:integrations],
    )
  end
  
  def create_token_client
    @create_token_client ||= Octokit::Client.new(
      bearer_token: generate_jwt_token,
      api_endpoint: vcs.api_url,
      web_endpoint: vcs.web_url,
      connection_options: {
        url: vcs.api_url,
      },
    )
  end
  
  def generate_jwt_token
    now = Time.now.to_i
    key = OpenSSL::PKey::RSA.new(vcs_app.private_key)
    opts = {
      iat: now,
      exp: now + 60,
      iss: vcs_app.external_database_id.to_i,
    }
    
    JWT.encode(opts, key, "RS256")
  end

We then pass this token to the API client in exactly the same way as before, keeping the overall change small and isolated.

Additionally, each generated token comes with an associated expires_at timestamp. To improve performance, we store an encrypted copy of the temporary token along with this timestamp in our database, allowing us to refresh the token only when necessary.

What did we learn in the process?

API Rate Limit Calculation

During this migration, we learned about the different rate-limiting mechanisms between GitHub OAuth and GitHub Apps.

With OAuth, we were able to cycle through user tokens to make API requests. This provided a very high capacity, as each token allowed for 5,000 GraphQL points per hour. When switching to GitHub Apps, however, the rate limit became organization-wide, meaning we had to think more critically about how often we were hitting the API and how expensive our queries were.

To understand our current GraphQL usage, we used the rateLimit object received from our GraphQL queries, and began tracking statistics using StatsD around how expensive each query was, as well as how often the tokens we used approached their rate limits:

  def process_RateLimit(object)
    object = Connectors::GitHub::Graphql::Fragments::RateLimitFragment.new(object)

    prefix = ["graphql", definition.name.demodulize, "limit"].join(".")
    $statsd.gauge("#{prefix}.cost", object.cost)
    $statsd.gauge("#{prefix}.remaining", object.remaining)
  end

We found that many of our GraphQL queries were fairly inexpensive, but we identified one particularly complex query to be expensive (15 GraphQL points) and frequent (every 10 minutes for every repository in an organization). We thought about how this query would perform for a large organization in our system (for example, 100 repositories) when using GitHub Apps:

100 repositories * 15 GraphQL points * 6 queries per hour = 9,000 GraphQL points / hour

This was initially concerning, as the number was significantly more than the 5,000 GraphQL point rate limit provided for an installation. However, another advantage of GitHub Apps is that its rate limit scales with the size of the organization. For installations with more than 20 repositories, an additional 50 requests (or 50 GraphQL points) is provided for each repository per hour.

With this increase, we identified our modified rate limit would be:

5,000 GraphQL points base + 50 additional points * 100 repositories = 10,000 GraphQL points

This provided us with enough capacity to maintain our current system. To help us stay comfortably within our limit going forward, we worked with GitHub to identify additional strategies for keeping our API usage within its rate limit:

  • Ingest real-time data via inbound webhooks
  • Catch-up with historical data using the API
  • Throttle your API requests to stay within your rate limits
  • Use conditional requests, where possible (currently only available via v3 REST API)

Impact on User Onboarding

Another lesson we learned during this migration is that GitHub Apps requires a different onboarding approach. Unlike OAuth, where users typically identify themselves and grant access in one step, with GitHub Apps, users are required to leave our site and install the application on GitHub before data can be processed. To smooth out this flow, we use the GitHub App installation endpoints to track the user’s onboarding progress and display in-app calls to action linking out to the GitHub App setup URL (when appropriate).

Wrap up

We are very excited to continue to build on GitHub Apps and to take advantage of all the data exposed through the GraphQL API. We’re also looking forward to partnering with GitHub and pioneering more beta features in the future.

If you’re interested in taking a more data-driven approach to engineering management (and want to see our GitHub Apps workflow in action) check us out!

Introducing the new Premium Support

Introducing the new Premium Support

Whether you use GitHub independently or with a team, we are proud to offer unparalleled support to everyone in our community. Over time we’ve learned that some teams require around-the-clock care and peace of mind that extend beyond our standard support offering. For these customers, we introduced Premium Support at last year’s Universe. Today we’re building on our initial launch and announcing an improved Premium Support program featuring two new plans: Premium and Premium Plus.

Both plans include:

  • 24/7 web and phone support
  • Priority ticket handling
  • Guaranteed initial response times (30 minutes for urgent, four hours for high priority)
  • Access to Premium Content
  • Scheduled health checks

Premium Plus also includes:

  • Named Technical Support Account Manager
  • Monthly administration support hours
  • One virtual training class on Git and GitHub best practices

View the new Premium Support program, or contact us to learn more.

Introducing GitHub Enterprise 2.14

For the first time, your team can connect to the power of the open source community—and find lots of ways to get more work done—with our latest Enterprise release.

Ready to start using these new tools? Upgrade your instance:

Download Enterprise 2.14

New developer tools

Now, if you have GitHub Business Cloud in addition to your Enterprise instance, you can tap into everything you love about open source from behind your firewall. Find public content and collaborate with the entire GitHub community without sacrificing security.

Learn more about unified search

Checks API (public beta)

The Checks API helps integrators build sophisticated tools for continuous integration, linting, and acceptance testing on GitHub. Previously, integrators could report success or failure of a build and include a link to more information using our Statuses API. With the new Checks API, they can specify more status information during builds and collect richer data, providing a more integrated experience for developers.

Learn more about the Checks API

Multiple issue templates

Project maintainers can now organize contributions and reduce noise for projects that have lots of collaborators with multiple issue templates and an improved setup process.

Learn more about issue templates

Ignore white space

When reviewing code, a diff with a number of white space changes can distract from the changes that matter. Now it’s easier to understand code updates from white space changes with an improved diff view. Change your view to automatically filter it out by clicking Diff settings.

white-space

Learn more about ignoring white space

Multiple required reviewers

As projects grow, you may want additional reviews for your team’s code changes. With the new multiple reviewer requirement, you can set how many reviewers are required for every pull request on a protected branch—so your organization’s important projects are protected from unwanted changes.

required-reviewers

Learn more about multiple required reviewers

Easier administration

Automated support tickets

Now you can create a support ticket directly from Enterprise and send diagnostic information to get help faster. We’ve also created a new command—ghe-support-upload—to directly upload your support tickets with the output of commands.

Learn more about automated support tickets

Dormancy threshold configuration

Enterprise 2.14 adjusts the default threshold for developers who are considered dormant from 30 to 90 days, so developers can retain access longer—and admins can now configure the threshold to fit their team.

Learn more about dormancy thresholds

Anonymous git access

You can now opt in to configure anonymous git access to public repositories when your Enterprise environment is in private mode. This helps continuous integration tools and build machines seamlessly access code for testing or deploy pipelines—and makes it easier for them to distribute Go or Swift projects that use git for submodules.

Learn more about anonymous git access

Additional updates

  • Improved project permissions: With more granular permissions for your team’s projects and the ability to create public projects, it’s now easier to update who has access to projects in your organization.
  • User hovercards: Developers will now be able to see more information about project collaborators when they hover over their avatars—or most places you see their username—including information that’s specific to their interaction with them.
  • Jump to: Now developers can quickly navigate to team pages, repositories, and projects they recently visited when they click the GitHub search bar (or hit the “/” key) with our new “jump to” feature.
  • Personal and organization dashboard improvements: Developers will now see more activity posts and better organization of content with improvements to personal and organization dashboards.

To see the full list of updates, check out our release notes.

Download Enterprise 2.14

Introducing GitHub Enterprise 2.13

Our latest GitHub Enterprise release has arrived with new ways for teams to collaborate, manage permissions, build custom tools, and more. Here’s an overview of what you’ll get when you upgrade to 2.13.

Download GitHub Enterprise 2.13

More collaboration

Team discussions

Team discussions give your team a dedicated home for conversations that aren’t related to code changes—so they’re easy to find when you need them without cluttering issues and pull requests.

Learn more about team discussions

team-discussions

Commit co-authors

With commit co-authors, your team can see who has contributed to every commit, regardless of how many contributors there are and make sure every author gets attribution in the pull request and in their contribution graph.

Learn more about commit co-authors

co-authors

Seamless admin

Built-in authentication with external providers

It’s easier than ever to make sure the right developers, contractors, and machine users have access to your instance with built-in authentication providers. And having more than one authentication process means that you always have a fallback when an external adaptor goes down or is undergoing maintenance.

Extended pre-receive hooks

We’ve extended pre-receive hooks so policies can be configured when code is pushed to your instance. With Git push options (--push-options), developers can transmit strings to the server for your pre-receive hooks.

Hotpatching for clustering

With hotpatching updates, you’ll now be able to roll out hotpatches for clustering without downtime.

Grafana in your Monitor Dashboard

Your Monitor Dashboard just got an upgrade of its own with the help of Grafana! Now you’ll be able to access more detailed performance graphs for your instance and have more control over how you analyze and share data in your organization.

grafana

Custom tools

GitHub Apps

GitHub Apps are fresh out of preview and fully available for teams using GitHub Enterprise 2.13. Now you can customize your workflow in just about any way you’d like with the help of GitHub APIs.

Learn more about GitHub Apps

Download GitHub Enterprise 2.13

To learn more, check out our release notes or RSVP for one of our upcoming webcasts:

Introducing GitHub Enterprise 2.12

GitHub Enterprise 2.12 has arrived with new organization tools to help your team stay focused and do their best work. Get project board enhancements, global webhooks, repository archiving, and more by upgrading to our latest release.

Ready to upgrade? Download GitHub Enterprise 2.12

More accessible project management

Project boards help you keep track of what’s happening with a group of tasks. Now, with automation events, you can let your project boards update the status of your tasks for you. Whether you or a teammate add a new card to a project, close an issue, or merge a pull request, your cards will automatically move between columns in real time.

Learn more about project board automation

If you rely on assistive technology or prefer using the keyboard to manage project boards, you can now use our new keyboard shortcuts to quickly navigate between cards and columns. Simple keystrokes allow you to move cards and columns anywhere on the board, commit changes, or review the keyboard shortcuts in more detail.

Learn more about project navigation

Project boards are a great way to organize tasks, but often, those tasks have external dependencies. With our new Enterprise release, you can use notes to add links to those dependencies. When you reference an issue or pull request in a note, you’ll now see a preview of the cross-referenced link in a summary card, giving you more context.

Learn more about project note summary cards

Global webhooks that keep you in the know

Now you can configure webhooks for an entire enterprise instance instead of one repository or organization at a time. Admins will be notified about new users and organizations, deleted users and organizations, and membership changes.

Learn more about global webhooks

Better repository organization tools

You don’t have to delete repositories when they’re inactive. Now you can archive older projects to make them read-only to owners and contributors. While no one will be able to add new issues, pull requests, or comments, you’ll be clutter-free and still have access to the repositories if you decide to iterate on them later.

Learn more about archiving repositories

With Enterprise 2.12, you can also search for repositories by their license using license family names or queries like “license:mit” and “license:gpl-3.0”. License searching gives you another way to find projects that fit your licensing requirements or repos you’d like to work on. We’ve even added a license filter on the “Advanced Search” page to help you craft the exact query you need.

Learn more about searching repositories by license

Automatic HTTPS deployment

GitHub Enterprise now supports Let’s Encrypt, a public Certificate Authority (CA), which automatically obtains and renews TLS and SSL certificates. Admins can get a secure “green HTTPS padlock” by running a simple CLI command or requesting a TLS certificate in the Management Console, with no fees or manual steps for certificate renewal.

Learn more about HTTPS deployment

Additional Updates

  • Use hotpatching to schedule patch upgrades during a designated upgrade window
  • Get geo-replication improvements like updated commands, archive generation, and support for GitHub Pages
  • View edit history for comments, use Markdown for topic discussions, and support new contributors with a contribution guidelines sidebar
  • Configure permissions around a repository fork
  • User-level requests for moves from child teams to parent teams

Upgrade today

Download GitHub Enterprise 2.12 to start using these tools. You can also check out the release notes to see what else is new or enable update checks to automatically check for the latest GitHub Enterprise release.

Not using GitHub Enterprise yet? Request a 45-day free trial

Newer

Changelog

Subscribe

GitHub Universe logo

GitHub Universe

October 16-17 in San Francisco
Get tickets today

Discover new ways to build better

Try Marketplace apps free for 14 days

Learn more