# Change management in ConnectorAPI

The document discusses a tool (API versioning) that can be used for faster change management in Connector API.

## Problem

Connector API uses deprecation approach to manage changes in the API. The approach considers only backward-compatible changes. Sometimes, doing a change in the API using backward-compatibility is either super complicated or not possible at all. We have one year deprecation policy period, however the policy is mostly ignored, due to the lack of monitoring of the deprecated changes, so we still keep the deprecated code.

**The main problem is our inability to evolve API changes as fast as we'd like to.**

There are numerous reasons why we want to evolve API:
1. Problematic adaptions of domain changes while keeping the backward compatibility. Some endpoints are overly complicated just to be backward compatible.
2. Slow performance of some API endpoints. Partners selecting and serializing 10k+ entities. We currently don't have a solution to return a larger amount of data. Operation ends with 408 -- request too heavy.
3. Inability to leverage and enforce new design/infrastructure concepts. E.g Switching to read-only replica can't be done in a non-breaking way.

## Solution

Each API to evolve needs to have a way to manage changes. We need to improve/introduce an approach that would allow us to discontinue deprecated implementation and have better control over it.
Industry-standard is to leverage some kind of API versioning in order to handle breaking changes.

*Versioning would allow us to create basically a new endpoint for the same resource--starting from scratch. We could simplify the design of the endpoint, don't care for supporting numerous older implementations, and introduce + enforce some new practices like paginations, shorter timeouts, read-only replicas, etc. Not only that, but at the same time we could easily monitor the usage of the deprecated endpoints.*

We managed to live 7 years without versioning in the ConnectorAPI. We should be really cautious when introducing a new version--bulk multiple deprecated versions together, keeping in mind forward compatibility...

Using API versioning wouldn't solve the whole problem. It's just a tool that can allow for a faster-paced development and address the limitation in the backward-compatible approach. In the meantime we created a [survey](https://mews.typeform.com/to/JCpBXN9h) to collect thoughts on what is the preferred way of API versioning for partners. From what we've heard so far, endpoint versioning using URI seems like the most likable approach. So far we've seen one internal partner (Kiosk team) preferring API versioning whilst the survey shows that the majority of our external partners advocate for endpoint versioning. Process of discontinunation is not part of RFC and could be done in future, aligned with the partners preferences. List to our current deprecations
[deprecation documentation](https://github.com/MewsSystems/gitbook-connector-api/tree/develop/deprecations)

### What to version

We can consider multiple versionable units in our case. Endpoint, entity, or the whole API.

| **Scope** | **Explanation** | **Pros** | **Cons** |
|-----------|---------------------------------------------------------------------------|---------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
| Endpoint | Individual endpoint. `v2/reservations/getAll` | + More granular + Client can update only the desired endpoint | - Each endpoint evolving on its own -- variety in latest version |
| Entity | All operations on entity. `v2/reservations/{getAll\|add\|update\|...}` | + Unified version per entity | - Client must update all operation for the entity - In between from endpoint and API versioning. |
| API | Entire API. `v2/{reservations\|companies\|rates\|...}/{getAll\|add\|...}` | + Unified latest version across the whole API | - Client must incorporate all previously changed endpoints - Single change in an endpoint requires bumping the whole API version |

There is not that big of a difference for us from the implementation point of view. When it comes to our internal consumers, currently there are some preferring endpoint and some API versioning.

API versioning unifies the version across the whole API and ensures that all changes are consumed when upgrading to the latest version. Endpoint versioning does it more granularly. There is no need to incorporate changes implemented from other endpoints.

### How to version

Once we settle up on what to version, next important question is to discuss how to implement the API versioning itself.

| **Versioning approach** | **Explanation** | **Pros** | **Cons** |
|-------------------------|------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|
| URI | Increase the version in the URL. `**v2**/reservations/getAll` | + Simple and unambiguous, easy to test + We already have the version in the API + Simplest implementation + Easier caching for clients | - Depends on the versioning scope. Might be confusing to have multiple versions for different endpoints/resources. |
| Custom header | Keep the path, but include the custom header with the version. It could be integer, date, whatever. | + URI is not cluttered by the version | - Custom header is required - Caching becomes more complicated |
| Request body | Keep the path, and header. Version is passed as a field in the request body. | + Everything is handled in a single method | - More complicated documentation (swagger) - More complicated endpoint logic - Deserialization of the whole request is required |
| Query parameters | `v1/reservations/getAll?v=1` | + Unified root version | - 2 versions at the same URL might creates confusion |

As the version is already part of our URI end it's not tieing us more to the REST, URI versioning seems like our best bet.

### Detailed design of endpoint versioning

The idea is pretty simple really. All of our API paths typically follow: `/api/{controller}/{version}/{entity}/{operation}` e.g. `/api/connector/v1/reservations/getAll`. By having a version embedded in the URL, this shouldn't be that big of a hassle from an API's uniformity and usability point of view. Once there is a need to create a new version of a particular endpoint, we simply introduce a new version of that endpoint whilst immediately deprecating the current one. Ensuring we have at most *2* versions of the same endpoint available is key not to lose control over numerous different behaviors in different versions. It can be achieved by code analyzers. At the same time, it would keep us cautious with introducing a new version each time there is a change in the API. We should rather pragmatically introduce a new version only when there is a need for that -- performance, too much backward compatibility, breaking change.
Leveraging API versioning and having problematic endpoints rewritten in an optimized and more scalable way would allow us to simply ask the affected partner to adopt the newer version.

*All of the problems mentioned in the problem section are addressable by adopting the versioning. Once a new version of an endpoint is available, we need to make a strict deadline for deprecation and notify the partners about it. All new integrations should use the latest version. We should monitor the traffic to older endpoints and could build some New Relic dashboards with the ratio between old/new versions. This should give us insight into the usage of the deprecated endpoint as we currently can't tell whether the deprecated fields are still used by some partner*

It seems like URI versioning is the right path for us. It is simple, unambiguous, intuitive, easily understandable + testable, when the logic changes, the URL changes. Even more when we already have the version embedded in the URL. There is probably no need to introduce the minor versions (v1.1) for connector API (could be beneficial in our internal APIs, though). When the survey will show us that API versioning is indeed something partners are used to and we could go for it, not much is going to change. We would simply bump the version of the whole API, introduce the newly versioned endpoint and route the rest of the endpoints to the old implementation.
New versions of the endpoint must be included in the swagger next to the old version, whilst the old version is marked as deprecated.

We could benefit from implementing preview endpoint versions. Such versions of endpoints are not included in the swagger/documentation and can be used by a pilot partner to test, and tune the functionality. We would be able to do breaking changes presuming those will be well communicated. This should prevent us from unnecessary versioning of an endpoint, as the new version should be more mature.

The simplest possible implementation is to move the version from the controller level to each individual endpoint. Creating a new endpoint and bumping up the API version.

    [Route("api/general")]
    public partial class GeneralApiController : JsonApiController<Dto.Parameters>
    ...

All endpoints then need to be versioned explicitly.

    [HttpPost]
    [Route("v1/enterprises/getAll")]
    [RequestBody(typeof(Dto.MultipleEnterpriseParameters))]
    [SuccessResponse(typeof(Dto.MultipleEnterpriseResult))]
    public Task<ActionResult> GetAllEnterprisesAsync()
    ...

Afterward, adding a new version of an endpoint should be as simple as adding a new endpoint with bumped-up version.

    [HttpPost]
    [Route("v2/enterprises/getAll")]
    [RequestBody(typeof(Dto.MultipleEnterpriseParameters))]
    [SuccessResponse(typeof(Dto.MultipleEnterpriseResult))]
    public Task<ActionResult> GetAllEnterprisesV2Async()
    ...

We could introduce something like VersionedRouteAttribute, that prepends the version `v{version}/` into the route.

POC can be found [here](https://github.com/MewsSystems/gitbook-connector-api/). Run locally, then just open http://localhost:61785/Swagger/general/swagger.yaml to check the generated swagger.yaml.
See: 

Alternatively, we could also use the package `Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer`. That would allow versioning via query parameters, headers, etc. It could benefit from parametrized version (no custom attribute needed). But for our purposes, the approach specified above should be enough.

### Version lifecycle

*The creation of a new version should be considered as a last resort. All changes that could be done in a backward-compatible way, should be done that way.*

[Endpoint versioning process](../../images/api-versioning.PNG)

*Simple idea of endpoint versioning is displayed above.*
*There is v1 version. Add v2 version, while marking the v1 as deprecated (e.g. in 6 months). Once the deprecation period is reached, notify partners still using soon to be discontinued version (by client token). Give some more time (2 months) and discontinue v1. When requested it can return 301 -- permanently moved to with the link to documentation/new version.
Alternatively, we could do discontinuation once v3 is going to be added.*

### Drawbacks

- Supporting multiple versions of basically the same thing -- possibly violating DRY

### Alternatives

-Keep managing the API changes using deprecation policy, but follow strict deprecation policy:
Issue with this approach is that although it might be more convenient for the clients, endpoints would be cluttered by numerous execution paths to support different "versions". Thus the implementation would be more error prone. At some point (6-12 months) we would need to discontinue the deprecated endpoints/fields so the not-migrated partners will be broken. There is no simple way to granularly monitor usage of the deprecated endpoints, so we can't pinpoint clients still using the deprecated fields and warn/notify just those.

-One option of change management is to create a new resource each time we need to "version" an endpoint. `reservations/getAll` -> `serviceOrders/getAll` while this applies to reservations endpoint, it wouldn't be scalable, and we could be cluttered by numerous deprecated/obsolete resources.

-Another way is to do something like `v1/reservations/getAllV2` to get the granularity of the endpoint versioning while keeping the API version unified. But this, similarly to query parameters, just seems overly confusing and not user-friendly.

### Next steps

- Analyze the results of the survey sent to the partners and agree on the scope of the versioning. The current proposal assumes endpoint versioning. If we will find out that API versioning is more feasible, the RFC is going to be updated.
- Setup the discontinuation approach. Agree on how to communicate changes, and deprecation timeline (again from the survey). This should not have a big impact on the implementation.

## Unresolved questions

We are creating questionnaire for partners about which versioning scope suits them most (endpoint/API). Technology (URI/header/...). Deprecation duration. How they'd like to be notified about the changes.