# Video Alignment Module
## Source Code:
https://github.com/b10815061/VideoAlignment
or
> new boxing dataset path : /home/weihsin/datasets/BoxingDatasetPkl/newBoxingDataset
```bash
/home/c1l1mo/projects/VideoAlignment
```
## Paper
[CARL](https://arxiv.org/pdf/2203.14957)
[PR-VIPE](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123500052.pdf)
## Project Structure
```
Root -
| - dataset <dataloader functions>
| - evaluation <eval functions>
| - example config <an example config for preparing experiment>
| - loss <loss functions>
| - model <model functions>
| - notebooks
| - result <store experiments>
| - experiment name
| - checkpoints
| - NC_align <Nancy's algorithm's visual>
| - visualization <DTW algorithm's visual>
| - config.yaml
| - (If specify --align_standard) output_test.pkl
| - (If specify --align_standard) output_train.pkl
| - (If specify --align_standard) output_val.pkl
| - (If specify --align_standard) standard.pkl
| - utils
| - train.py
| - evaluate.py
```
## Data Preparation
For Cropping videos, refer to [Data Preparation](/YoJgPksiQpWHi3prdzyLaA)
Current data path can be found in the config files.
(We train the alignment mode on mere jumping, but test it on a whole clip, so the testing name is started with long)
### Prepare experiment configuration
Create a folder and a ```config.yaml``` in result folder. Check out example_config.
## Prepare
### FS dataset
#### one jump
This dataset contains 'Axel' and 'Axel_com'.
The skater in the video performs a single jump.
```bash
/home/c1l1mo/projects/VideoAlignment/result/Axel_merged_one_jump_128/config.yaml
```
#### two jump
This dataset contains 'Axel' and 'Axel_com'.
The skater in the video performs a double jump.
```bash
/home/c1l1mo/projects/VideoAlignment/result/Axel_merged_two_jump_128/config.yaml
```
#### loop
```bash
/home/c1l1mo/projects/VideoAlignment/result/Loop_two_jumps_128/config.yaml
```
#### Lutz
```bash
/home/c1l1mo/projects/VideoAlignment/result/Lutz_two_jumps_128/config.yaml
```
## Training
```bash
$ python -m torch.distributed.launch --nproc_per_node <gpu_count> train.py --cfg_file <cfg_file_path>
```
pytorch update:2025-08-27
```bash
torchrun --nproc_per_node=<gpu_cnt> train.py --cfg_file <cfg_file_path>
```
### FS dataset
sample number frame = average number of frame / 2
In the config file, the LOGDIR must be equal to the parent dir of the config file. Otherwise assertion error raise.
## Testing
```
python -m torch.distributed.launch --nproc_per_node <gpu_count> evaluate.py --cfg_file <cfg_file_path> (args)
```
pytorch update:2025-08-27
```
torchrun --nproc_per_node=<gpu_count> evaluate.py \
--cfg_file <config_path> \
--demo_or_inference <eval_pkl_path>
```
### Dump_nearest_frame
```
python -m torch.distributed.launch --nproc_per_node 1 evaluate.py --cfg_file /home/c1l1mo/projects/VideoAlignment/result/NACL_TCC_1 --generate --no_compute_metrics --nc --align_standard --demo_or_inference long_test_with_assumed_standard_label.pkl
```
This will generate videos in NC_align folder and ```*.pkl```s for the error localization module.
Args Table:
| Name | Usage | Type |
| -----| ----- | ---- |
| demo_or_inference | The name of the test pickle | String |
| record | Whether to record on W&B | Bool |
| ckpt | Specify a checkpoint (absolute path) to test (Default is last ckpt)| String |
|generate | Whether to visualize the output videos| Bool |
|no_compute_metrics | Whether to evaluate the metric (e.g.:Retreival, Kendalls Tau) | Bool |
| nc | Whether to use Dynamic Time Wrapping or Nancy's alignment argument | Bool |
| align_standard | Whether to also align ```long_train``` and ```long_val``` with the standard video | Bool |
| query/candidate| The q/c index in the split, useful if we want to check 2 certain videos' alignment| String |
| overwrite | Whether to overwrite the output of the ```dump_nn_frames``` module | Bool
[Alignment Progress Report](/D3ptVVt3SNedf25seupoaQ)
## label augment
/home/c1l1mo/datasets/scripts/skating_pipeline/Alignment_scripts/label_augmentation.py
## Yolo
> find /home/weihsin/datasets/BoxingDatasetPkl/newBoxingDataset/2025-03-18_rename/ -name "*.MP4" -exec ./run.sh {} ./runs/newBX/ \;
/home/c1l1mo/testings/yolov7
> python -m torch.distributed.launch --nproc_per_node 1 train.py --cfg_file ./result/Axel_single_jump/config.yaml
> /home/weihsin/datasets/boxing_safetrim/utils/GT_back_test_w_label.pkl
```
###################################
key: video_file, type: <class 'str'>
value: /home/c1l1mo/datasets/boxing_safetrim/processed_videos/8_back_4_cam3.mp4
###################################
key: frame_label, type: <class 'torch.Tensor'>
value: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0.])
###################################
key: seq_len, type: <class 'int'>
value: 100
###################################
key: name, type: <class 'str'>
value: 8_back_4_cam3
###################################
key: id, type: <class 'int'>
value: 37
###################################
key: labels, type: <class 'list'>
value: ["The body's center of gravity is too high.", "When throwing a punch, don't stick your butt out. It's the rotation of the waist and feet that drives the hand to punch, not just the upper body moving.", 'The back foot is not lifted. The hip joint is not rotated. The hands are too stiff. There is too much forward pressure on the body when punching. The core is not exerting force.', 'Leaning too far forward caused the buttocks to stick out because the hip joint forgot to rotate.']
###################################
key: features, type: <class 'numpy.ndarray'>
value: [[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... -4.97029512e-04
-1.75009698e-01 -1.22211434e-01]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... -6.94153830e-04
-1.75029323e-01 -1.22981407e-01]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... -1.50179106e-03
-1.72031537e-01 -1.22644141e-01]
...
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... -1.41648378e-03
-1.73207730e-01 -1.30805939e-01]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... -1.06615262e-05
-1.71268240e-01 -1.28437266e-01]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... -4.64254757e-04
-1.73038319e-01 -1.27384186e-01]]
(100, 66)
----------------------------------------
```
> /home/c1l1mo/datasets/boxing_safetrim/utils/BX_check.py
> CONFIG_PATH: /home/c1l1mo/projects/VideoAlignment/result/boxing_safe_trim_front_all/config.yaml
> CKPT_PATH: /home/c1l1mo/projects/VideoAlignment/result/boxing_safe_trim_front_all/checkpoints/checkpoint_epoch_00149.pth
> /home/c1l1mo/SkateAppModels/SkatingApp/engine/BoxingAlignment/config.yaml
> /home/c1l1mo/projects/VideoAlignment/result/boxing_safe_trim_front_all/config.yaml
```
ModuleNotFoundError: No module named 'icecream'
conda activate motion2text
20469 pip install icecream
20470 python crop_video.py --dataset new_BX --alphacrop 1
ModuleNotFoundError: No module named 'rich_pixels'
20471 pip install rich-pixels
Sign in with Wallet
Connect another wallet