# Introduction to safety models

## MMI model

MMI model is based on I3D https://arxiv.org/pdf/1705.07750.pdf,
but our model has no opticalflow stream

Codebase: safety-model-i3d-known-hazards
It's trained from pretrained model https://github.com/deepmind/kinetics-i3d

### Dataset preparation:

You can download labelled data from SI Portal or from safetydatadev blob

Then use ffmpeg to transfrom videos to frames
ffmpeg -i VIDEO_PATH.mp4 FRAME_PATH/i/%05d.jpg

### How to train and test:

You can refer to the README in the safety-model-i3d-known-hazards codebase

Run (Train and test)

./experiments/scripts/run.sh $config_file $gpu_num $experiment_name

Train

./experiments/scripts/train.sh ./experiments/config/BridgeStone/base.yaml 0 bridgestone

Test

./experiments/scripts/test.sh ./experiments/config/BridgeStone/base-test.yaml 0 bridgestone

In base.yaml or base-test.yaml:

PRETRAIN: /root/I3D_checkpoint => pretrain_model_path
SAVE_CKPT_INTERVAL: 100

TRAIN:
MAX_LOOP: 1000 => total batches
DATA:
- DOMAIN_NAME: '136' => device name
NUM_CLASSES: 4 => number classes on this device
DATALIST:
- list: '/list/path' => a list to store file paths of all frames
batch_size: 1 => batch size of this list

## Feedback loop

### Add a new camera and copy weights to it
1. Download the latest I3D-release model
https://ml.azure.com/model/list?wsid=/subscriptions/bd26395a-031e-498b-acfe-4066c3b64edf/resourcegroups/safetymodelprod/workspaces/safety-feedbacktraining-prod&tid=301f59c4-c269-4a66-8a8c-f5daab211fa3

2. Create a folder architecture mentioned in configs/model.yaml
bstw/
├── compiled_dev_dataset
│   └── dev_test.list
└── lists
├── train_133_hazards.list
├── train_133_safe.list
├── train_134_A-7.list
├── train_134_hazards.list
├── train_134_safe.list
├── train_136_A-7.list
├── train_136_hazards.list
...

3. Revise file paths defined in configs/model.yaml
list: /root/compiled_dev_dataset/dev_test.list => list: bstw/compiled_dev_dataset/dev_test.list
list: /root/lists/train_133_hazards.list => list: bstw/lists/train_133_hazards.list
PRETRAIN: /root/onlinelearning_97471f99-3bf2-4831-b1f6-5a539eabec15/ckpt/I3D_checkpoint => PRETRAIN: I3D-release file path
NEAREST_DOMAIN_NAME => modify what you want
[camera set NEAREST_DOMAIN_NAME means that the camera will not train at all, representing the camera will copy the NEARES_DOMAIN_NAME weights]

4. Freeze a new model
python tools/ckpt_to_pb.py --cfg [I3D-release]/configs/model.yaml --output model.pb

5. Modify other config files
modify config.ini, custom_device_events_mapping.json, event_label_map.json to match the new model

6. Pack the new model and upload it to AzureML

## Anomaly detection

Anomaly detection model is based on autoencoder

Codebase: safety-model-anomaly-detection

### Dataset preparation:

You can download labelled data from SI Portal

Then use ffmpeg to transfrom videos to frames
ffmpeg -i VIDEO_PATH.mp4 FRAME_PATH/i/%05d.jpg

### How to train and test:

You can refer to the README in the safety-model-anomaly-detection codebase

Finally, we use one-encoder-multiple-decoders model to support 135 and 136 cameras (w/ PLC signals) simulatenously.

Preparation
In Codes/multiple/utils_multi.py
DECODER_NAMES = ['h142nd', 'h41st', 'h42nd', 'h141st']
It defines the list of output nodes, and for your training and testing set, their names need to include these names, like h142nd_1, h41st_1, h141st_3, h42nd_5, and so on.
Another, need to define ROIs for each camera and set them in Codes/training_hyper_params/hyper_params_bridgestone_multi.ini

Train model:
python Codes/multiple/train_multi.py --dataset bridgestone --train_folder TRAIN_SET_PATH --test_folder TEST_SET_PATH --gpu 0 --height 512 --width 512 --layers 8 --features_root 8 --share_layers 8 --config Codes/training_hyper_params/hyper_params_bridgestone_multi.ini --snapshot_dir MODEL_DESCRIPTION

Evaluate model:
The same as before, need to revise Codes/multiple/evaluate.py BRIDGESTONE_LABEL_PATH to your test set label path
python Codes/multiple/inference_multi.py --dataset bridgestone --test_folder TEST_SET_PATH --gpu 0 --height 512 --width 512 --layers 8 --features_root 8 --share_layers 8 --config Codes/training_hyper_params/hyper_params_bridgestone_multi.ini --snapshot_dir MODEL_DESCRIPTION --psnr PSNR_DESCRIPTION

Calculate PSNR:
Need to define FPS(--fps), camera index (--index), and ROIs (--roi_top, --roi_bottom, --roi_left, --roi_right)
The following script is for 136 camera
python Codes/multiple/cal_psnr_multi.py TEST_SET_PATH CKPT_PATH --gpu 0 --height 512 --width 512 --layers 8 --features_root 8 --share_layers 8 --fps 10 --index 0 --roi_top 96 --roi_bottom 480 --roi_left 136 --roi_right 520

Freeze graph:
CUDA_VISIBLE_DEVICES=0 python Codes/multiple/freeze_multi.py CKPT_PATH OUTPUT_PB_PATH --height 512 --width 512 --layers 8 --features_root 8 --share_layers 8

Generate output and loss map images: [for debug]
python Codes/multiple/gen_output_multi.py --dataset bridgestone --test_folder TEST_SET_PATH --gpu 0 --height 512 --width 512 --layers 8 --features_root 8 --config Codes/training_hyper_params/hyper_params_bridgestone_multi.ini --snapshot_dir MODEL_DESCRIPTION

## Pointer confirmation

Pointer confirmation is divided into three categories:
1. Not-do detection
Detect if people do actions when crossing intersections
2. Not-stop detection
Detect if people or vehicles stop at "STOP" area
3. Not-align detection
Detect if people hand/head not align

Codebase: safety-skill-pointer-confirmation

### Dataset preparation

You can download labelled data from SI Portal or from safetydatadev blob

### Not-Stop

It's fully rule-based to decide if people or vehicles stop at "STOP" area, but we train a classifier to check if he/she is a person walking or driving on a vehicle. The classifier is based on OpenPose keypoints.
code: src/region_of_interest/*.py
src/people_vehicle_classifier.py

### Not-do

Detecting not-do behaviors divides into three stages:
rule-based, image-based, keypoint video-based
code: src/not_do_*.py

### Not-align

Detecting not-align behaviors uses keypoint video-based classifier
code: src/not_align*.py