# KPI's for EmbedAPI and sphinx-hoverxref

_(draft of work for this task https://trello.com/c/R1XrXxwm/868-create-kpis-for-user-base-of-embed-api-api-calls-hoverxref-dependents)_

* How many projects are using the Embed API?
* We are _not_ logging good responses from `embed/v3/views.py` (eg. `log.info("Success. project= url= referer= user-agent=")`)
* We are not sending NGINX logs to util01, so we don't have history --only the current day per instance
* APIv3 `grep "/api/v3/embed/" /var/log/nginx/readthedocs-api.log | cut -d " " -f 8,12 | less` gives us the EmbedAPI URL + the referer
* APIv3 `grep "/api/v3/embed/" /var/log/nginx/readthedocs-api.log | cut -d " " -f 12 | sort | uniq | wc -l` gives us how many URLs are using EmbedAPI
* APIv2 `grep "/api/v2/embed/" /var/log/nginx/readthedocs-api.log | cut -d " " -f 8 | cut -d "&" -f 1 | sort | uniq | less` show projects using the EmbedAPI

> some of these logs are saved into readthedocs@util01(org):~/embedapilogs

* How many projects are using Hoverxref?
* Currently, we don't have an exact way to know this. There was an attempt to collect this type of data already in https://github.com/readthedocs/readthedocs.org/pull/8124 (telemetry)
* Dependents analysis:
* 147 projects are using it from https://github.com/readthedocs/sphinx-hoverxref/network/dependents
* Those 147 projects include several forks
* 3 dependent packages and 1 dependent repository according to https://libraries.io/pypi/sphinx-hoverxref
* Note there is no way to differentiate a request comming from a random page that one coming from sphinx-hoverxref. _We need to add some specific User-Agent or similar in our client_.
* What is the rate of change of adoption of these projects?

----

# EmbedAPI's and sphinx-hoverxref's data

Instead of using `log.info` to acummulate data in logs, we could use the same implementation's idea from "Build's data" by triggering Celery tasks and strore structured data using a different database.

## Structure

embed:
  api:
  date:
  url:
    full:
    query:
      doctool:
      doctoolversion:
  referer:
  user-agent:
  http-status-code:
  project:
    id:
    slug:
  organization:
    id:
    slug:
  version:
    id:
    slug:
  language:
  user:
    id:
    username:

## Questions

* What projects are hitting rate limit?
* What projects use EmbedAPI?
* How many projects use EmbedAPI via `sphinx-hoverxref`?
* How many projects are using APIv3/APIv2?
* How many projects use `doctoolversion==1.8.x`?

----

# Build's data

_(based on https://github.com/readthedocs/readthedocs.org/pull/8124/)_

Having a complete structure of build's data will allow us to answer several questions that will help us to make bussiness decisions.

## Structure

build:
 id:
 date:
 length:
 success:
 project:
   id:
   slug:
 organization:
   id:
   slug:
 version:
   id:
   slug:
 config:
   user:
   final:
 packages:
   user:
     pip:
     conda:
     nodejs:
     apt:
   final:
     pip:
     conda:
 os:
 python:

## Questions

* How much build time a particular project is using in a date range?
* How much build time a particular organization is using in a date range?
* How many projects are installing `sphinx-hoverxref`, and which of them are using a specific version?
* What projects are using `sphinx<=1.8.x`?
* How many builds using `sphinx<=1.8.x` have failed in the last 2 weeks?
* How many projects are using the new `build.tools` config?
* What projects would require `USE_LATEST_SPHINX` feature flag due that their builds are failing because of `docutils==0.18` release?
* How many projects have tunned configuration for search (eg. `search.ranking`)?
* How many projects have enabled PDF output?
* How many projects use Conda compared to pip?

## Implementation

Once the build is finished, we can trigger a Celery task to be executed in a particular queue `telemetry` that gather(*) all of this information from the `Build` object (using the default database) and dumps it into a new database used specifically for this purpose (with _real_ JSON fields).

_(*) we could call the Celery task passing all the data or we can parse the build's output when the task is executed._

## Querying data

Once we have this data stored in a PostgreSQL database, we can just use django's shell for quick queries or we can integrate both databases with Metabase to have nice real-time graph. Besides, we could also just do an SQL dump (or directly connect to production database) and analyze it using a Jupyter Notebook for a more extensive work.

# Plan to action

1. ~~add User-Agent to sphinx-hoverxref and make a new release~~
2. ~~add `log.info` to EmbedAPI Python view~~

> at this point we will start collecting at least some small data about EmbedAPI that we can parse/view from Read the Docs' application logs

3. ~~Migrate to Django 3.2 (https://www.djangoproject.com/weblog/2021/apr/06/django-32-released/)~~
4. ~~(optional) migrate current JSONFields for PostgreSQL built-in~~
5. create new db for `telemetry`
6. dump data into new db using Celery tasks triggered by EmbedAPI Python view
7. (optional) implement build's telemetry

## Concerns about this plan

- Builders do not have access to the db. So, they will need to trigger a task once the build have finished passing all the data to be executed on webs that have access to the db, and store the data
- Metabase does not natively support PostgreSQL JSONb type (see https://github.com/metabase/metabase/issues/708)
- there is a comment that says what we need (I think) it's supported, tho. https://github.com/metabase/metabase/issues/708#issuecomment-375042214
- creating a view is a current workaround, since Metabase see it as a regular table; it doesn't make too much sense for data that's too nested, tho
- maybe consider using MongoDB which is natively supported by Metabase? (https://www.metabase.com/docs/latest/administration-guide/databases/mongodb.html)