# Standards for API Integrations in the Nitro Ecosystem

# Summary

Here we define a streamlined and standardized way to integrate with external APIs using net-http with a thin wrapper to ensure proper error handling and reporting. This will provide developers with a clear foundation for API integrations, catalyze smoother collaboration across teams, reduce friction during onboarding, improve maintainability, and fortify the overall resilience of the Nitro ecosystem.

# Motivation

[In August 2023, we faced a production incident arising from an external API call.](https://github.com/powerhome/incidents/commit/651f573fef67c116b85bcdb5ca431d639a74c925?short_path=35f03c9#diff-35f03c98fc83b3a2e1ad4ef130bc7d8d00a4157e3fb4b26c1dbb62a23eff5398) We discovered that we were making many concurrent requests that took 60 seconds to time out, placing large load on web servers. Additionally, long running transactions / metrics on the API were not getting reported to New Relic, which made the incident take longer to pinpoint and debug.

The results of this outage indicated that we missed some fundamental considerations and edge cases for this endpoint, but they could easily have been missed for others.

In the dynamic landscape of our development environment, different teams have organically adopted distinct tools, often driven by the preexistence API clients in their respective domains. While this decentralized approach initially addressed immediate needs, it has inadvertently led to a proliferation of varied methods to accomplish essentially the same tasks. It also propagated some unwanted behaviors; if one integration missed proper consideration of timeouts or reporting, that behavior could get passed down to subsequent integrations that modeled their usage after it.

The divergence and lack of clear guidelines has already played a role in production outages. It has also resulted in an inconsistent codebase, presenting challenges in terms of maintainability, collaboration, and strong standards. The absence of a unified approach has made it difficult for developers to seamlessly transition between different components, hampered efficiency, and contributed to a higher cognitive load.

## Current Scenario

In a recent nitro-web survey, we counted at least 48 instances of external API calls with varying clients:

* 20 use Net::HTTP
* 15 use HTTParty
* 9 use RestClient
* 2 use Excon
* 2 use gems specific to their integration

Notably, the majority operate under basic use cases, embodying straightforward integrations. There are two complicated or more restricted circumstances:

1. [Greensky] this issue is due to our integration partner's failure to adhere to [HTTP guidelines around case sensitivity for headers](https://www.rfc-editor.org/rfc/rfc7540#section-8.1.2). In this case, the team decided to go with Excon because that gem does not perform string-casting on headers. In this specific case, Excon was a win, but this is actually a drawback to Excon for most use cases.
2. [Run Payments] This API requires the use of a short-expiring token (one with an expiration date of 1 hour.) The team has outlined their plan [here](https://hackmd.io/1vFP4RgKS9-ri7K8dRjDEw). In this case, they opted to use Net::HTTP and database the token as they routinely update it.

Given this context, our aim is straightforward: establish a consistent approach and guidelines for Nitro's external API integrations. This standardization encompasses fundamental aspects like timeouts, elegant failure handling, and issue notifications, aiming to streamline our ecosystem and simplify the API integration process.

# Solution

### _net-http_

Moving forward, we propose a consolidated approach for handling APIs within the Nitro ecosystem. Our solution involves adopting net-http as the default for API interactions.

Using this gem is the most straightforward solution with the least path of resistance for conversion in nitro-web. net-http is already included in ruby and is supported as part of that language. In the event that ruby maintainers choose an alternative to net-http, it means that other common APIs that wrap net-http (including HTTParty and RestClient) would also need to find alternatives. At best, their solutions would mean more upgrades for us; at worst, they may introduce breaking changes, awkward upgrade paths, and assertions that no longer align with our own. Since net-http comes bundled with ruby and is the default gem for requests, adopting this as the standard means we do not have to concern ourselves with additional maintenance costs.

Additionally, we have evidence of other successful enterprise-level companies (Shopify, for example) that establish their standards around net-http; going through another package may just be introducing an unnecessary layer of abstraction.

### _api_chai_

To enhance usability and resilience, we will implement a straightforward layer between API calls and net-http, which we will call [api_chai](https://rubygems.org/gems/api_chai). Once this layer is equipped, developers will be expected to use net-http through api_chai for their external API integrations.

The intermediary layer serves a triple purpose. First, it establishes a default timeout for API requests, ensuring we aren't waiting too long with failed connections or non-responses. Second, it will integrate with New Relic and Sentry to ensure prompt issue reporting, providing real-time insights into failure points. Finally, it codifies graceful error handling, preventing disruptions and allowing a smooth flow of information back to the caller.

By standardizing on net-http and implementing the api_chai lightweight layer, we aim to simplify API interactions across the board, making our system more robust, reliable, and developer-friendly, and avoiding incidents where failures cause production outages or hamper debugging tools.

# Implementation Plan

This structured plan provides a phased and collaborative approach, leveraging HFH support, iterative testing, and detailed transition guides to ensure a smooth adoption of the net-http standard and associated tools.

1. Establish Standard:

Clearly define and communicate the net-http standard for all teams during the transition, ensuring a unified approach.

2. Develop an api_chai component:

This will be created in nitro-web and used as a versatile stopgap for easy testing and implementation throughout the transition period.

3. Implement api_chai for net-http calls:

Initiate the transition by implementing api_chai for existing net-http calls, starting with nitro_search in components/nitro_search/lib/tasks/nitro_search.rake.

HFH will lead initial implementations and extend them to other instances inside of nitro-web.

Facilitate testing by involving one or two teams who would implement changes in their domains, gathering valuable feedback.

HFH will publish a comprehensive transition guide, aiding teams in adopting api_chai.

4. Migrate HTTParty Use Cases:

Begin migrating HTTParty use cases, starting with core_models/suppliers in components/core_models/lib/core_models/suppliers/clients/distributor.rb.

Once they have migrated a few components as examples, HFH will spearhead initial conversions and provide detailed transition guides for wider team adoption.

5. Migrate RestClient Use Cases:

Address RestClient use cases, beginning with components like components/mdm/lib/mdm/jamf/jamf_api.rb.

HFH to handle initial migrations and guide teams through the transition, again publishing transition guides that other teams can rely on.

6. Migrate Excon Use Cases:

Migrate the existing Excon use cases, with HFH taking the lead in the transition.

7. External Client Support:

Ensure adaptability by allowing external clients to be supported, accommodating unique use cases.

The integration will be something like this:

ApiChai.with_external_client do
    GemSpecificToIntegration.call(my_args_here)
end

This would allow us to continue to use gems specific to integrations, while still giving us all the benefits of monitoring, etc that the ApiChai wrapper provides.

8. Extract api_chai to power-tools:

Elevate api_chai by extracting it into power-tools, extending its accessibility and utility across the entire Nitro ecosystem.

# Drawbacks / Alternatives

_net-http is unwieldy_

As one developer put it:

> nobody likes writing net-http code. And it’s easy to see why, just look at this cheatsheet: its API is convoluted, verbose, needlessly OO-heavy (why does one need an exception for every HTTP error status code…), it just does not enact joy.

The good news is that we can address the most frustrating parts of this integration via the thin wrapper that we already plan to use to help us manage our other concerns.

_Build vs Buy / Other API Clients_

A similar argument is that instead of directly referencing net-http, there is always the option to use an existing, well-documented and supported library that already implements the standards we require. For example, HTTParty, which has long been a staple, much-loved and utilized gem due to ease of use. However, HTTParty and RestClient, our second and third most prolific clients, are just a user-friendly wrapper for net-http, so we're not really getting the intended benefit if we (1) have mostly straightforward implementations already and (2) have a thin layer that was already written to be user-friendly and helpful. These other clients also avoid being opinionated by way of enforcing reporting to any specific third-party (NewRelic) or enforcing shorter default timeouts, so they don't really provide the desired behavior we're looking for. Using net-http encourages us to continue the path of straightforward, simplistic code and behavior.

### How We Teach This

1. API Standards in BT Handbook & api_chai

We'll integrate API standards into our BT Handbook for easy access and reference. These standards will emphasize practical implementation by pointing to api_chai-specific documentation, providing examples to ensure consistency across projects.

2. Migration Guides for nitro-web

To make transitions smoother, we will develop straightforward migration guides. These guides will address common challenges, offering step-by-step instructions. Interactive elements like code snippets and troubleshooting tips will be included, ensuring developers have practical support during the migration process.

3. Lunch & Learn Session with Summary Publication

We will offer Lunch & Learn sessions designed to keep the team in the loop with the latest in API practices and chat about it real-time. These sessions aim to create an open space for questions, collaboration, and continuous learning, fostering a more connected and informed development community. For those unable to attend, we'll publish a concise summary capturing key insights, takeaways, and any discussions.