# Recording Offset Corrections
---
<details><summary>References</summary>
<p>
### Discussion
Offset Correction:
https://www.notion.so/pupillabs/2024-Roadmap-aba2fb2987074af28ed40da7452e1cf9?p=770140ff8eee428ebc5ec043393e43b7&pm=c
- Recording hash is computed based on columns in recordings table
- Recording hash
- logic for recording hash
- store recording hash in db
- All new recordings will have recording hash
- gaze mapper function should apply bias
- get_gaze function will apply bias
- Affected tables: recording_gazepoints, recording_fixations_scanpath, recording_gaze_on_aoi
- do research in code and make changes
---
### Related code
- `get_gaze_data(` being used in
- Markerless post processing
- Apriltag post processing
- Raw Data Export
- Face Mapper Export
- /<uuid:recording_id>/gaze.<format>
- Table `recording_gazepoints` being used
- `def raw_densified_files`
Other Details
- Patch on `/<uuid:recording_id>` already supports update of gaze_bias_x, gaze_bias_y
---
- Websocket events for all the changed recordings
- Specific event to reload the project when there is change in recording offset
---
</p>
</details>
___
<details><summary>Iteration 1</summary>
<p>
## Potential candidates for discussion and changes
- Gaze offset in info.json (from app) is added to the `distorted` gaze which is later used to get the `undistorted` gaze, this is done in gaze pipeline (`process_recording`)[https://github.com/pupil-labs/pupil-cloud-pipeline/blob/b11da1cca33fae3a3d32aec587994f2a6ee9ec40/process_recording.py#L400-L405]
- `/gaze.json` endpoint which is used to show gaze on recording in UI - (uses distorted coordinates)[https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/apis/v2/namespaces/recordings.py#L1068-1069]
- When exporting recording's `Timeseries Data`
- Is fixations, blinks, imu, saccades also affected?
- imu is not affected, the rest are, but they are read from recording_gazepoints table
- Gaze Mapper - `get_gaze_data(` - `__get_gaze_data_from_clickhouse(` adds bias to undistorted coordinates, then distort coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/recording_gazepoints.py#L141-153
- Enrichments display and export
- Marker Mapper - Uses distorted coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/processing/surfaces.py#L171
- Slam Mapper - Uses distorted coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/processing/enrichments/markerless_enrichment.py#L88
- Face Mapper Export - Uses distorted coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/project_enrichment_wrapper/face_mapper_wrapper.py#L399-425
- Render
- Manual Mapper
- Raw Data Export - Uses distorted coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/project_enrichment_wrapper/raw_data_export_wrapper.py#L506-514
- How fixations data will be affected when gaze bias is changed?
- Add gaze bias to fixations as well?
- In enrichments like marker mapper and slam mapper, we already have mapped data, with gaze_bias already added in gaze mapper, do we need to consider the addition of gaze bias in mapped data as well?
---
- Cloud export csv with two new columns?
- Need to write new 200hz?
- Bias need to be applied as normalized or as pixel?
- offset applied in distorted, bias applied in undistorted, undistorted in -1.2 to + 1.2
---
</p>
</details>
___
# Discussion Points
> We should go with constant distorted offset similar to how android does it rather than apply the offset in undistorted space, since it's easier all around for people to understand.
>
> We should rename the gaze_bias_x/gaze_bias_y to gaze_offset_x/gaze_offset_y, and the migration step should populate the value from info.json into there as a default value, ie. if a recording was made with offset `[10, 20]` then the db should have `[10, 20]` for that recording.
>
> This means that the offset will show up as `[10, 20]` in cloud, and then the user can change that to eg. `[0, 50]`
>
> We can keep the db as undistorted values, but when applying gaze offset we will need to apply it *after* the distortion step, and also subtract the original android gaze offset, ie. in the previous case:
>
> Assuming distorted gaze from db = [600, 700]
>
> If the gaze offset in db is [10,20] we return 600,700
> If the gaze offset is [0, 50] we return 590, 730
>
> This means we will need to:
>
> 1. read the undistorted gaze from db
> 2. distort the gaze
> 3. subtract the original android gaze offset (from info.json)
> 4. add the cloud gaze offset
Unlike having two different offset (from android and from cloud), we now have only one offset that is added during gaze pipeline and can be changed in cloud.
___
### CSV Files that are exported
**AOI Common CSVs**
- aoi_metrics.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- aoi_fixations.csv
- using `recording_gazepoints`, no offset is being applied
**Face Mapper Export**
- face_detections.csv
- Uses `face_detections`
- gaze_on_face.csv
- Uses `face_detections`
- Uses `recording_gazepoints`, which uses `__get_gaze_data_from_clickhouse(` where offset is being applied in online fashion
- fixations_on_face.csv
- Uses `face_detections`
- Uses `recording_gazepoints`, no offset is being applied
**Raw Data Export**
- gaze.csv
- Uses `recording_gazepoints`, which uses `__get_gaze_data_from_clickhouse(` where offset is being applied in online fashion
- events.csv
- N/A
- imu.csv
- N/A
- fixations.csv
- Uses `recording_gazepoints`, no offset is being applied
- saccades.csv
- Uses `fixations.json`
- blinks.csv
- N/A
- Uses `recording_gazepoints` but don't use the coordinates
- world_timestamps.csv
- N/A
- 3d_eye_states.csv
- N/A
- Uses `recording_gazepoints` but don't use the coordinates
- sections.csv
- N/A
**Render Export**
- section.csv
- N/A
**Static Image Mapper Export**
- fixations.csv
- Uses `recording_gaze_on_aoi`, but no need to update it anywhere OR do we want to apply the gaze offset when mapping gaze on UI?
**Marker Mapper Export**
- gaze.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- fixations.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- surface_positions.csv
- N/A
- Uses `apriltag_detections`
**Common CSVs**
- sections.csv
- N/A
**Slam Mapper**
- gaze.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- fixations.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
___
### Apart from the suggested changes quoted above, few things won't require any other changes:
- Getting `/gaze.json`: https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/apis/v2/namespaces/recordings.py#L1068-1069
- `gaze.csv` in face mapper export: https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/project_enrichment_wrapper/face_mapper_wrapper.py#L399-425
- `gaze.csv` in raw data export: https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/project_enrichment_wrapper/raw_data_export_wrapper.py#L506-514
### Few implementations that require additional changes:
- **Marker Mapper post processing** - Uses distorted coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/processing/surfaces.py#L171
- **Slam Mapper post processing** - Uses distorted coordinates - https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/processing/enrichments/markerless_enrichment.py#L88
- Should we automatically run the postprocessing for that enrichment recording whenever a gaze offset is changed? (show in UI as processing with note that this is because of changed offset)
- Or should we make the enrichment computable and (show in UI as processing with note that this is because of changed offset), and on pressing the run button, we launch the postprocessing for that enrichment recording? (this to me looks a valid approach since we won't need to run postprocessing for all the enrichments but only those user is interested in)
- Or should we completely make the computed enrichment invalid?
- Or can we somehow apply the gaze offset once the gaze is mapped to surface or image when retrieving
- gaze data (https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/recording_gaze_on_aoi.py#L359)
- and fixation data (https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/recording_gaze_on_aoi.py#L481)
***Decided to allow to run the enrichment again which will launch post-processing for those slices in marker-mapper or slam-mapper enrichment***
- Generate recording hash using `recording_id`, `gaze_offset_x` and `gaze_offset_y`
- `recording_hash = json.dumps({"recording_id":"8c560ba3-fb82-4ba8-8ad5-b7cbdb1f2bf2","gaze_offset_x":0,"gaze_offset_y":0}).encode("UTF-8")`
- Create a new column `recording_hash` in `slices` table
- When creating a slice, update the value of `recording_hash`
- For existing `slices`, ingest `recoding_hash` column in migration using its value generated from `recording_id`, `gaze_offset_x` and `gaze_offset_y`
- When retreving slices, check if the `recording_hash` value retrieved from db matches the current `recording_hash` value
- Create a new slice status `STALE` in `slices` table for running the postprocessing step
- If the value of `recording_hash` differs, then update the slice as `STALE` and add queue for post processing
- **Render enrichment compute**: I think we need to make change in pikit's
- `GazeCircleBurner` (https://gitlab.com/pupil-labs/pupil-cloud/pupil-invisible-toolkit/-/blob/3e3701d59ab7918558893859d1d58d143d83cf93/src/pikit/lib/tools/video/frame_processors.py#L34) or at `get_gaze_data` (https://gitlab.com/pupil-labs/pupil-cloud/pupil-invisible-toolkit/-/blob/3e3701d59ab7918558893859d1d58d143d83cf93/src/pikit/lib/clickhouse_db.py#L57)
- and `FixationScanpathBurner` - which uses scanpath.json, so we also need to apply the gaze offset to the scanpath data as well when reading? (https://gitlab.com/pupil-labs/pupil-cloud/pupil-invisible-toolkit/-/blob/3e3701d59ab7918558893859d1d58d143d83cf93/src/pikit/lib/tools/video/frame_processors.py#L169)
to add support of gaze offset when rendering gaze and/or fixations (https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/enrichment_containers/render/render_and_upload.py#L212), as there is seemingly no such option in pikit!
- **Gaze pipeline result files** that are affected as they the derived from `filtered_offset_gaze_distorted` (after applying device's offset in the distorted gaze):
- fixations.json
- scanpath.json
- **DB tables** that can get affected:
- `recording_gazepoints`: we can apply the changes, you mentioned earlier, at https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/recording_gazepoints.py#L141-153 to always get the gaze data with applied gaze offset
- `recording_fixations_scanpath`: this can get tricky as `recording_fixations_scanpath` is ingested from `scanpath.json` (https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/enrichment_containers/gaze-pipeline/gazepipeline_runner.py#L248)
- so we would have to update it everytime?
- Or can we somehow apply gaze offset in online fashion when retrieving data from table like we are doing with `recording_gazepoints`?
- `recording_gaze_on_aoi`: This is being used in `marker-mapper` (data is ingested during post processing), `slam-mapper` (data is ingested during post processing), `manual-mapper` (only time and fixation data is used)
- `manual-mapper` does not need any changes
- `marker-mapper` and `slam-mapper` can allow user to launch post processing and ingest with new gaze offset applied
- `recordings`: it has column `gaze_offset_x` and `gaze_offset_y`
- Allow `gaze_offset_x` and `gaze_offset_y` value set when a recording is uploaded and using `PATCH` request - ref: `https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/merge_requests/373/`
- Migration or script to ingest `gaze_offset_x` and `gaze_offset_y` from `info.json` of all the recordings
- **CSV files from export** that can get affected:
- **AOI Common CSVs**
- aoi_metrics.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- aoi_fixations.csv
- using `recording_gazepoints`, no offset is being applied
- **Face Mapper Export**
- fixations_on_face.csv
- Uses `face_detections`
- Uses `recording_gazepoints`, no offset is being applied
- **Raw Data Export**
- fixations.csv
- Uses `recording_gazepoints`, no offset is being applied
- saccades.csv
- Uses `fixations.json`
- **Static Image Mapper Export**
- fixations.csv
- Uses `recording_gaze_on_aoi`, but no need to update it anywhere OR do we want to apply the gaze offset when mapping gaze on UI?
- **Marker Mapper Export**
- gaze.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- fixations.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- **Slam Mapper**
- gaze.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- fixations.csv
- using `recording_gaze_on_aoi`, which has data ingested after applying offset in gaze mapper during post processing
- **Other things** that get affected:
- `/scanpath.json` (https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/apis/v2/namespaces/recordings.py#L1125) uses table `recording_fixations_scanpath` in `get_scanpath_data(` (https://gitlab.com/pupil-labs/pupil-cloud/pupil-api/-/blob/05a73bac73f9d845ea66a89b944ad0b444449e74/pupilcloud/models/recording_gazepoints.py#L635)
### Implementation that does not require any changes
- Manual Mapper: as we don't use fixation coordinates at all OR do we want to apply the gaze offset when mapping gaze on UI?
- Gaze pipeline result files that are not affected:
- gaze_offline.npy
- headset_on_off_predictions.npy
- gaze_offline_timestamps.npy
- blinks.json
- eye_states.npy
___
What can get tricky?
- Apply gaze offset on `scanpath.json` which is being used in pikit to render fixations on the video
- Apply gaze offset on the `fixations.json`
- Apply gaze offset on the `recording_fixations_scanpath` in online fashion without ingesting it with new data
- We insert into `recording_gazepoints`, `recording_fixations_scanpath`, update `fixations.json`, `scanpath.json`, and insert into `recording_gaze_on_aoi` (ondemand) everytime gaze offset is updated
- OR apply gaze offset every where `recording_gazepoints` is being used (e.g., `get_gaze_data(`) in online fashion and apply gaze offset every where `recording_fixations_scanpath` being used in online fashion and update `fixations.json`, `scanpath.json`
___
## Subtasks for gaze offset correction implementation
- Add support at recording's level
- [x] Add columns `gaze_offset_x` and `gaze_offset_y`
- [x] Update existing `gaze_offset_x` and `gaze_offset_y` with the data from `info.json`
- [x] Update `gaze_offset_x` and `gaze_offset_y` when a recording is uploaded
- [x] Update `PATCH` endpoint to update `gaze_offst_x` and `gaze_offst_y`
- [x] Add support at data export level when using gaze data from `recording_gazepoints`
- read the undistorted gaze from db
- distort the gaze
- subtract the original android gaze offset (from info.json)
- add the cloud gaze offset
- Recording hash implementation
- [x] Add new column `recording_hash` in `slices` table
- [x] Add new enum `STALE` in `status` in `slices` table
- [x] Migration to update existing rows value in `recording_hash` in `slices` with the hash or default hash or None
- [x] Update `recording_hash` when an enrichment slice is created
- [x] `slam-mapper` - Add support to check the `recording_hash` on every `GET` and show slice as `STALE`, and update it as `STALE` in db, if not updated
- [x] `marker-mapper` - Add support to check the `recording_hash` on every `GET` and show slice as `STALE`, and update it as `STALE` in db, if not updated
- [x] On enrichment compute, if a slice's `status` is `STALE`, schedule postprocessing job and set `status` to `PROCESSING`
- [ ] Behaviour of enrichment like `slam-mapper` and `marker-mapper` when a recording's `gaze_offset_x` and `gaze_offset_y` is changed
- When recording's gaze offset is changed, it will send `Recording` model ws event, can UI make use of it to reload the enrichment if that recording is part of the enrichment and display those recording slice as stale and ask for recompute, which will add slice in postprocessing queue and set status to `Processing` in backend?
- Render enrichment fix
- [x] Add support in pikit to read gaze data with offset applied
- [x] Add support in pikit to read fixations data with offset applied
- [x] Ensure that render is done with gaze and fixations with latest offset applied
- [x] Make command changes in local compute repo to support gaze offset
- [ ] Make command changes in `pupil-cloud-terraform` repo to support gaze offset
- [ ] Add, test and verify gaze offset correction support for every exported CSV when downloading enrichment
- Other changes:
- [ ] Ensure that `fixations.json` is always read at all places with latest gaze offset applied
- [ ] Ensure that `scanpath.json` is always read at all places with latest gaze offset applied
- [x] Change in pikit is done
- [x] Ensure that retrieve on table `recording_fixations_scanpath` should always apply the latest gaze offset
- [x] Fix endpoint `/scanpath.json` to have latest gaze offset applied
- [x] Fix retrieval of fixations when retrieving fixations in aois, face mapper export and raw data export
- [ ] Ensure that `manual-mapper` also has latest gaze offset applied ?
- [ ] Add gaze offset support in implementation of `get_fixation_seek_timestamps` ?
- [ ] Behaviour when a slice is in post-processing and during that gaze offset is changed ?
- [x] Fix upload script to update info.json with the correct recording id or use the recording id from info.json when uploading a recording, that was leading to failure in accessing gaze data from db when computing render enrichment of the recordings uploaded using the script as `info.json` had different recording id than the newly generated recording
---
UI Changes:
- PATCH request on `/recording` to update gaze, new gaze offset should consider gaze without the current gaze offset
- After patch, enrichments will have slices with `STALE` status
- Enrichments can be run once we have `STALE` slices for postprocessing
---
Google Drive
Gist
Clipboard
Sign in with Wallet
