# Manual Gaze Mapping / Manual Mapping

---

Design: https://www.figma.com/file/rVdulet73OQ65GJAfYdXCL/PC-%3A-Redesign-P2-Development?type=design&node-id=743-108397&mode=design&t=4XkzNvtVjyMTj4qt-4

---

---

## Database Tables

### Postgresql table `manually_mapped_fixations` to store the mapped fixation coordinates and its details

Table `manually_mapped_fixations`

- use `mapping_id` as primary key
- unique index using `BRIN(enrichment_id, recording_id, start_time)`
- `enrichment_id` having foreign key reference on `project_enrichments (id)`
- `recording_id` having foreign key reference on `recordings (id)`

- Generation of `mapping_id`:

- maximum `fixation_id` in `production` is `41369` and start time is `10583 seconds` or `2.94 hours`, around `4 fixations/sec`
- maximum `fixation_id` in `staging` is `23725` and start time is `12481 seconds` or `3.4 hours`, around `2 fixations/sec`
- to encode `fixation_id` to hexadecimal notation, maximum `fixation_id` should be `65535` to get the hexadecimal notation within 4 digits, which means `65535/4 secs` or `16383 secs` or `4.5 hours` of recording considering `4 fixations/sec`, neon supports up to 4 hours of recording time

  # generate mapping id from enrichment_id, fixation_id and recording_id
  mapping_id = <group_of_8_digits_from_enrichment_id>-<group_of_4_digits_from_enrichment_id>-<group_of_8_digits_from_hex_notation_of_fixation_id>-<group_of_12_digits_from_recording_id>

  enrichment_id = 5465a465-d245-4fc9-b141-4d5d9dc98c1f
  recording_id = 13d8ab9e-491e-48d7-986a-6ee86bd4fbc7
  fixation_id = 1048575 (0x000fffff)

  mapping_id = 5465a465-d245-000f-ffff-6ee86bd4fbc7

- Retrieve `enrichment_id`, `recording_id`, `fixation_id` from `mapping_id`

- Since mapping_id is `<group_of_8_digits_from_enrichment_id>-<group_of_4_digits_from_enrichment_id>-<group_of_8_digits_from_hex_notation_of_fixation_id>-<group_of_12_digits_from_recording_id>`
- enrichment_id's prefix is <group_of_8_digits_from_enrichment_id>-<group_of_4_digits_from_enrichment_id> of mapping_id
- fixation_id in hex notation form is <group_of_8_digits_from_hex_notation_of_fixation_id>
- recording_id's suffix is <group_of_12_digits_from_recording_id>

  mapping_id = 5465a465-d245-000f-ffff-6ee86bd4fbc7

  enrichment_id = 5465a465-d245-*
  recording_id = *-6ee86bd4fbc7
  fixation_id = 000fffff

| column_name | column_description | column_type |
| ------------- | ------------------------------------------------------------------------------------------------------ | ----------- |
| mapping_id | unique uuid generated using enrichment_id, fixation_id and recording_id | uuid |
| enrichment_id | enrichment_id having foreign key reference on project_enrichments(id) | uuid |
| recording_id | recording_id having foreign key reference on recordings(id) | uuid |
| fixation_id | fixation id | integer |
| x | fixation x coordinate, scaled to the reference image size | smallint |
| y | fixation y coordinate, scaled to the reference image size | smallint |
| start_time | fixation start time in nanoseconds, matching with recording's `fixations.csv`'s `start timestamp [ns]` | bigint |
| end_time | fixation end time recording offset in nanoseconds) ie. 15s into recording = 15000000000 | bigint |

| data_type | storage_size | range |
| --------- | ------------ | -------------------------------------------- |
| uuid | 16 bytes | |
| integer | 4 bytes | -2147483648 to [phone] |
| smallint | 2 bytes | -32768 to +32767 |
| bigint | 8 bytes | -9223372036854775808 to +9223372036854775807 |

CREATE TABLE manually_mapped_fixations (
  mapping_id uuid NOT NULL,
  enrichment_id uuid NOT NULL,
  recording_id uuid NOT NULL,
  fixation_id integer NOT NULL,
  x smallint NOT NULL,
  y smallint NOT NULL,
  start_time bigint NOT NULL,
  end_time bigint NOT NULL,
  CONSTRAINT pk_manually_mapped_fixations PRIMARY KEY (mapping_id),
  CONSTRAINT fk_manually_mapped_fixations_enrichment_id FOREIGN KEY (enrichment_id) REFERENCES public.project_enrichments(id) ON DELETE RESTRICT ON UPDATE CASCADE,
  CONSTRAINT fk_manually_mapped_fixations_recording_id FOREIGN KEY (recording_id) REFERENCES public.recordings(id) ON DELETE RESTRICT ON UPDATE CASCADE
);

CREATE INDEX ix_mapped_fixations_enrichment_id_recording_id_start_time_end_time
ON manually_mapped_fixations
USING BRIN (enrichment_id, recording_id, start_time, end_time);

---

### Postgresql table `static_image_mapper` to store `static_image_id` that inherits from table `project_enrichments`

Table `static_image_mapper`

| column_name | column_description | column_type |
| -------------------------------- | ------------------ | ----------- |
| columns from project_enrichments | | |
| . | | |
| . | | |
| . | | |
| static_image_id | | uuid |

ALTER TYPE public."enrichment_kind" ADD VALUE 'static-image-mapper';


CREATE TABLE static_image_mapper (
static_image_id uuid references files(id),
kind public."enrichment_kind" CHECK (kind = 'static-image-mapper')
) INHERITS (project_enrichments);

---

---

## Endpoints

### Endpoint to create/update/get/delete static-image-mapper enrichment

`/v2/workspaces/{workspace_id}/projects/{project_id}/mappers/static/`

- Return `static_image_id` in `args` param of response to be in sync with the current frontend development

Supported methods:

- `POST` on `/mappers/static/` **KARTIKEY**

Payload

  {
    "name": "Untitled",
    "static_image_id": "23d2f16d-564c-274d-234f-00d7c49add32"
    "from_event_name": "recording.begin",
    "to_event_name": "recording.end"
  }

Response

  {
    "code": 201,
    "message": "201 CREATED",
    "request_id": "9a7ccbec5c8a4f45b0bb8690defd8197",
    "result": 
      "static_image_id": "23d2f16d-564c-274d-234f-00d7c49add32",
      "created_at": "2024-01-01T01:15:29.395379Z",
      "created_by_user_id": "b911f2b3-e9aa-4be5-a3bf-30251f94735d",
      "deleted_by_user_id": null,
      "from_event_name": "recording.begin",
      "id": "c4390f5c-1a3c-47f5-ab87-316996067598",
      "kind": "static-image-mapper",
      "name": "Untitled",
      "project_id": "4b6604f8-6caf-474c-b1be-49ad2cb11903",
      "slices": [],
      "status": {
        "COMPUTING": 0.0,
        "ERROR": 0.0,
        "READY": 1.0,
        "SUCCESS": 0.0
      },
      "to_event_name": "recording.end",
      "updated_at": "2024-01-01T01:15:29.395379Z"
    },
    "status": "success"
  }

- Use to create a new `static-image-mapper`
- Initially, only `name`, `kind`, `from_event_name`, `to_event_name` suffice to create a new enrichment

- `GET on /projects/{project_id}/enrichments/{enrichment_id}`
- **TODO: return same api response format as current one, ie. with {"args": {"static_image_id": "23d2f16d-564c-274d-234f-00d7c49add32"}**

- `GET`on `/mappers/static/{static_enrichment_id}` **STAVROS**

Response

  {
    "code": 200,
    "message": "200 OK",
    "request_id": "a10c179e44f94ef8b7805bf8ed689387",
    "result": {
      "static_image_id": "23d2f16d-564c-274d-234f-00d7c49add32",
      "created_at": "2024-01-01T01:15:29.395379Z",
      "created_by_user_id": "b911f2b3-e9aa-4be5-a3bf-30251f94735d",
      "deleted_by_user_id": null,
      "from_event_name": "recording.begin",
      "id": "c4390f5c-1a3c-47f5-ab87-316996067598",
      "kind": "static-image-mapper",
      "name": "Untitled",
      "project_id": "4b6604f8-6caf-474c-b1be-49ad2cb11903",
      "slices": [],
      "status": {
        "COMPUTING": 0.0,
        "ERROR": 0.0,
        "READY": 1.0,
        "SUCCESS": 0.0
      },
      "to_event_name": "recording.end",
      "updated_at": "2024-01-01T01:16:49.987532Z"
    },
    "status": "success"
  }

- `PATCH`on `/mappers/static/{static_enrichment_id}`
**STAVROS**

Payload

  {
  "static_image_id": "23d2f16d-564c-274d-234f-00d7c49add32",
  "name": "new",
  "from_event_name": "recording.begin",
  "to_event_name": "recording.end"
  }

Response

  {
    "code": 200,
    "message": "200 OK",
    "request_id": "ffa1358280b942bbbe137a29baa5c053",
    "result": {
      "static_image_id": "23d2f16d-564c-274d-234f-00d7c49add32",
      "created_at": "2024-01-01T01:15:29.395379Z",
      "created_by_user_id": "b911f2b3-e9aa-4be5-a3bf-30251f94735d",
      "deleted_by_user_id": null,
      "from_event_name": "recording.begin",
      "id": "c4390f5c-1a3c-47f5-ab87-316996067598",
      "kind": "static-image-mapper",
      "name": "new",
      "project_id": "4b6604f8-6caf-474c-b1be-49ad2cb11903",
      "slices": [],
      "status": {
        "COMPUTING": 0.0,
        "ERROR": 0.0,
        "READY": 1.0,
        "SUCCESS": 0.0
      },
      "to_event_name": "recording.end",
      "updated_at": "2024-01-01T01:28:00.842571Z"
    },
    "status": "success"
  }

- Use to update `static_image_id`, `name`, `from_event_name`, etc.

- `DELETE`on `/v2/workspaces/{workspace_id}/projects/{project_id}/enrichments/{static_enrichment_id}` (already implemented) **STAVROS (test it)**

Response

  {
    "code": 200,
    "message": "200 OK",
    "request_id": "b59da74421364f4c89788d6c95635b5b",
    "result": null,
    "status": "success"
  }

- **No need** to implement `DELETE` on `/mappers/static/{static_enrichment_id}`

### Endpoint to add/update/get/delete mapped fixations for static-image-mapper enrichment

`/v2/workspaces/{workspace_id}/projects/{project_id}/mappers/static/{static_enrichment_id}/recordings/{recording_id}/fixations/{fixation_id}`

Supported methods:

- `GET` on `/mappers/static/{static_enrichment_id}/recordings/{recording_id}/fixations` **KARTIKEY**

Response

  {
    "code": 200,
    "message": "200 OK",
    "request_id": "281d44c5e27c46248ecd79c8f8f92b9c",
    "result": {
      "fixations": [
        {
          "end_timestamp": 1.984,
          "fixation_id": 1,
          "start_timestamp": 1.979,
          "x": 793,
          "y": 559
        },
        {
          "end_timestamp": 1.989,
          "fixation_id": 2,
          "start_timestamp": 1.984,
          "x": null,
          "y": null
        },
        {
          "end_timestamp": 1.994,
          "fixation_id": 3,
          "start_timestamp": 1.989,
          "x": null,
          "y": null
        }
      ],
      "mapped_fixations": 1,
      "total_fixations": 3
    },
    "status": "success"
  }

- `GET` response will provide all the `fixations`, of the particular recording, between `from_event_name` and `to_event_name` of the enrichment, irrespective of whether it's mapped or not.
- Fixations that are not mapped will be set to `null` by default which mean that there is no entry of these fixations in table `manually_mapped_fixations` and they are returned as null by backend
-

flowchart LR
    subgraph s3
        fixations.json
    end
    subgraph db
        subgraph manually_mapped_fixations
            exists
            not_exists
        end
    end
    fixations.json
    -->|"id=1,x=null,y=null"|exists
    -->|"id=1,x=4,y=6"|response
    fixations.json
    -->|"id=1,x=null,y=null"|not_exists
    -->|"id=1,x=null,y=null"|response

- Reads `fixation_id`, `start_timestamp`, `end_timestamp` from `recording_id`'s `fixations.json`'s columns `fixation id`, `start timestamp [ns]`, `end timestamp [ns]`
- Reads `x` and `y` from table `manually_mapped_fixations` otherwise sets to `null`
- Fetches data from `fixations.json` (s3 bucket)
- Return data from `fixations.json` with x, y values replaced with the values from db and if db does not return value, replace it with null
- `null` values in the fixation coordinates can be used by frontend to jump to the fixations that is not mapped yet
- `mapped_fixations` is the number of mapped fixations, i.e., those have an entry in table `manually_mapped_fixations`
- `total_fixations` is the total numbers of fixations present in that particular recording
- `mapped_fixations` and `total_fixations` can be used by frontend to show the progress of the mapped fixations

- `PUT` on `/mappers/static/{static_enrichment_id}/recordings/{recording_id}/fixations/{fixation_id}` **ANYONE**

Payload

  {
    "x": 573,
    "y": 359
  }

Response

  {
    "end_timestamp": 1994320000,
    "fixation_id": 3,
    "start_timestamp": 1989320000,
    "x": 573,
    "y": 359
  }

- No `POST` and `PATCH` is not needed, update/create a new entry in table `manually_mapped_fixations` using same
- GET endpoint, by default, fetches start and end timestamp, in ns, from the `fixations.json`
- Use `enrichment_id`, `fixation_id` and `recording_id` to build `mapping_id`
- Use the `mapping_id` to create/update the row in table `manually_mapped_fixations`
- We may not need to add `start_timestamp` and `end_timestamp` in table `manually_mapped_fixations` as we already have it in `fixations.json`!

- `DELETE` on `/mappers/static/{static_enrichment_id}/recordings/{recording_id}/fixations/{fixation_id}`
**ANYONE**

Response

  {
    "code": 200,
    "message": "200 OK",
    "request_id": "a69da74421364f4c89788d6c95635b5b",
    "result": null,
    "status": "success"
  }

- Use `enrichment_id`, `fixation_id` and `recording_id` to build `mapping_id`
- Delete entry for that `mapping_id`

- `POST` on `/mappers/static/{static_enrichment_id}/recordings/{recording_id}/fixations:clear` Only implement if needed by ui to clear all fixations

## Enrichment download

**STAVROS**

- Implement endpoint `/mappers/static/{static_enrichment_id}/export`
- `GET` on `/mappers/static/{static_enrichment_id}/export`
- Provides zip stream containing csv file
- CSV contains columns `section id`, `recording id`, `fixation id`, `start timestamp [ns]`, `end timestamp [ns]`, `duration [ms]`, `fixation detected in reference image`, `fixation x [px]`, `fixation y [px]`
- Downloaded csv will provide all the `fixations` of all the recordings of the enrichment irrespective of whether it's mapped or not.
- Fixations that are not mapped will be set to `""` by default which mean that there is no entry of these fixations in table `recording_gaze_on_aoi` and they are sent as empty string by backend
- `"fixation detected in reference image"` column in `fixations.csv` could be renamed to something else as now have three status based on `gaze_on_aoi` value in `recording_gaze_on_aoi` table in clickhouse.
- if gaze_on_aoi == None, then fixation not mapped
- if gaze_on_aoi == 0 , then fixation not in reference image
- if gaze_on_aoi == 1, then fixation in reference image
- Export can be done similar to how RIM enrichment does.
- Files to be returned: `enrichment_info.txt`, `fixations.csv`, `static_image.jpeg`, `sections.csv`, and aoi related files similar to enrichment like RIM.

## Enrichment Progress

- `GET` on `/mappers/static/{static_enrichment_id}/recordings/{recording_id}/fixations` provides `checked_fixations` count and `total_fixations` count that can be used to show progress of the mapping

---