---
# System prepended metadata

title: SEAGULL-Egocentric Perception

---

# SEAGULL-Egocentric Perception
[TOC]
## Panoptic Segmentation
### Architecture
[Mask2Former-Swin-Base-IMG21K](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask2former)

Input:
- RGB image with size 900x900

Output:
- Semantic segmentation
- Object detection result, each with bbox/mask/confidence score

### Results


## State Estimator
### Architecture

Preprocess:
- Ignore Img frames < 1000 pixels
- Enlarge BBox by 30% for each sideto enclude more context

Input:
- Object centric image 
- Class number for the object 

Model:
- Object cat => Cat Embedding 8 dim
- Img Feat => Clip ViT-B/32 => Proj => Img Embbeding, 128 dim
- Two Embeddings (128 + 8) dim => MLP => 128 dom => 3
- For each attribute, sigmoid
- loss is only calculated on attributes that actually have a GT

Stats:
- Metric:
    - Acc
    - Recall
    - F1
- Cat:
    - Per attributes

python Ultimate_Evaluator.py --model_path /scratch/chaijy_root/chaijy2/jiayipan/outputs/model-mlm32-all-weights-open/AOA_Checkpoints/checkpoint_step270000.pth --log_path /scratch/chaijy_root/chaijy2/jiayipan/outputs/model-mlm32-all-weights-open/AOA_Results/checkpoint_step270000.pth
### Results
#### Val-seen
```
====isDirty====
TP: 16667, TN: 19273, FP: 1122, FN: 439
Precision: 0.937
Recall: 0.974
Accuracy: 0.958
Balanced Accuracy: 0.960
F1: 0.955
====isFilledWithWater====
TP: 15431, TN: 13763, FP: 4133, FN: 1972
Precision: 0.789
Recall: 0.887
Accuracy: 0.827
Balanced Accuracy: 0.828
F1: 0.835
====isToggled====
TP: 9175, TN: 17014, FP: 610, FN: 7205
Precision: 0.938
Recall: 0.560
Accuracy: 0.770
Balanced Accuracy: 0.763
F1: 0.701
```

#### Val-unseen
```
====isDirty====
TP: 48546, TN: 63073, FP: 2400, FN: 1657
Precision: 0.953
Recall: 0.967
Accuracy: 0.965
Balanced Accuracy: 0.965
F1: 0.960
====isFilledWithWater====
TP: 50297, TN: 43598, FP: 8792, FN: 6210
Precision: 0.851
Recall: 0.890
Accuracy: 0.862
Balanced Accuracy: 0.861
F1: 0.870
====isToggled====
TP: 30965, TN: 60583, FP: 3093, FN: 27988
Precision: 0.909
Recall: 0.525
Accuracy: 0.747
Balanced Accuracy: 0.738
F1: 0.666
```
## Depth Estimator
from [HLSM](https://github.com/valtsblukis/hlsm)

