---
title: '4D-Humans Code'
disqus: hackmd
---
4D-Humans(Code)
===
## Step By Step
[TOC]
## Outline of demo.py
1. We load models:
* HMR 2.0 (including checkpoints)
* Detectron2
2. Setup Renderer
3. Pass the image into detectron2 to get segmented images that contains humans
4. Pass the segmented images into HMR 2.0 as batches
5. Using the predicted pose and camera render the mesh and add it to image
Detectron-2
---
* This helps us to get segmented images containing one human
* We pass the original image into detector(Detectron 2)
```python=
det_out = detector(img_cv2)
```
* ***det_out*** will return object which contain :
1. **num_instances** : Number of instances
2. **image height** : Height of image
3. **image width** : Width of image
4. **fields**
1. **pred_boxes** : Coordinates of bounding box which contain human
2. **scores** : prediction score whether that box contain human or not
3. **pred_classes** : 0 if its human in image
4. **pred_masks** : predicted segmentation mask.
* We create variable ***boxes*** that contains pred_boxes data for every particular box that contains human using ***valid_idx*** which contains indexes where pred_classes is 0 and score >0.5.
```python=
det_instances = det_out['instances']
valid_idx = (det_instances.pred_classes==0) & (det_instances.scores > 0.5)
boxes=det_instances.pred_boxes.tensor[valid_idx].cpu().numpy()
```
* Visualizing detectron-2 results:-
All detected object boxes:-
Boxes containing humans and boxes corner coordinates:-
<img src="https://hackmd.io/_uploads/rJ4dUp8ZC.png" alt="image" width="300" height="auto">
HMR 2.0
---
### Preprocessing data for HMR2.0
* We pass the predicted bounding ***boxes*** containing humans from detectron to ***VitDetDataset***
```python=
dataset = ViTDetDataset(model_cfg, img_cv2, boxes)
```
* ***dataset*** will contain :
1. center : coordinates of center point in box
2. scale : scale of bounding box
3. personid : unique value for each bounding box which is basically person
4. img_size, mean, std: default values which can be changed from config file
5. img_cv2 : image
6. cfg : configuration
7. train : which is set false while testing
``` python=
self.cfg = cfg
self.img_cv2 = img_cv2
assert train == False, "ViTDetDataset is only for inference"
self.train = train
self.img_size = cfg.MODEL.IMAGE_SIZE
self.mean = 255. * np.array(self.cfg.MODEL.IMAGE_MEAN)
self.std = 255. * np.array(self.cfg.MODEL.IMAGE_STD)
# Preprocess annotations
boxes = boxes.astype(np.float32)
self.center = (boxes[:, 2:4] + boxes[:, 0:2]) / 2.0
self.scale = (boxes[:, 2:4] - boxes[:, 0:2]) / 200.0
self.personid = np.arange(len(boxes), dtype=np.int32)
```
* ***dataloader*** is provided by PyTorch library which helps to load data in batches
``` python=
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8, shuffle=False, num_workers=0)
```
### Output of model
* We load data into HMR2.0 model in batches and result is stored in ***out***
``` python=
for batch in dataloader:
batch = recursive_to(batch, device)
with torch.no_grad():
out = model(batch)
```
* The model will return :
1. pred_cam : Camera prediction
2. pred_smpl_params : this contains smpl parameters which help us extract pred_vertices, pred_keypoints_3d etc.
3. pred_cam_t : Predicted camera translation
4. focal_length : Default focal length from cfg.
5. pred_keypoints_3d : 3D vertices of human body key points/joint points
6. pred_vertices : This contains 3D vertices of human body in image(resized image that is sent as input)
* we use these pred_vertices to generated human mesh
``` python=
import trimesh
mesh = trimesh.Trimesh(out['pred_vertices'][n].detach().cpu().numpy().copy(), model.smpl.faces.copy())
# initial
mesh.show()
```
7. pred_keypoints_2d
### Working of model
1. We use ***ViT transformer*** backbone to extract image tokens
``` python=
# 4D-Humans/hmr2/models/hmr2.py
def create_backbone(cfg):
if cfg.MODEL.BACKBONE.TYPE == 'vit':
return vit(cfg)
else:
raise NotImplementedError('Backbone type is not implemented')
self.backbone = create_backbone(cfg)
# Compute conditioning features using the backbone
# if using ViT backbone, we need to use a different aspect ratio
conditioning_feats = self.backbone(x[:,:,:,32:-32])
```
2. ***Transformer Decoder*** -
###### (refer 4D-Humans/hmr2/models/components/pose_transformer.py , 4D-Humans/hmr2/models/heads/smpl_head.py)
* We pass the image tokens (conditioning_feats) from ***ViT transformer*** as input
* We use a standard transformer with multi-head self-attention. This processes a single (zero) input token by cross-attending to the output image and ends with linear readout of Θ.
``` python=
# 4D-Humans/hmr2/models/heads/smpl_head.py
def build_smpl_head(cfg):
smpl_head_type = cfg.MODEL.SMPL_HEAD.get('TYPE', 'hmr')
if smpl_head_type == 'transformer_decoder':
return SMPLTransformerDecoderHead(cfg)
else:
raise ValueError('Unknown SMPL head type: {}'.format(smpl_head_type))
```
```python=
# 4D-Humans/hmr2/models/hmr2.py
# Create SMPL head
self.smpl_head = build_smpl_head(cfg)
# smpl_head is Cross-attention based SMPL Transformer decoder
pred_smpl_params, pred_cam, _ = self.smpl_head(conditioning_feats)
```
* From config, we get number of joints and joints representation type that we want.
* ***npose*** is basically number of pose parameters we want
(if joint_rep_type=23, joint_rep_dim=6 then npose=144 )
``` python=
# 4D-Humans/hmr2/models/heads/smpl_head.py
self.joint_rep_type = cfg.MODEL.SMPL_HEAD.get('JOINT_REP', '6d')
self.joint_rep_dim = {'6d': 6, 'aa': 3}[self.joint_rep_type]
npose = self.joint_rep_dim * (cfg.SMPL.NUM_BODY_JOINTS + 1)
```
* Initialize body_pose, betas(shape), camera
***mean_params*** refers to a set of average or typical parameters representing various aspects of human body characteristics
``` python=
# 4D-Humans/hmr2/models/heads/smpl_head.py
# SMPL.MEAN_PARAMS: data/smpl_mean_params.npz
mean_params = np.load(cfg.SMPL.MEAN_PARAMS)
init_body_pose = torch.from_numpy(mean_params['pose'].astype(np.float32)).unsqueeze(0)
init_betas = torch.from_numpy(mean_params['shape'].astype('float32')).unsqueeze(0)
init_cam = torch.from_numpy(mean_params['cam'].astype(np.float32)).unsqueeze(0)
```
* Transfomer is imported from pose_transformer.py
``` python=
# 4D-Humans/hmr2/models/heads/smpl_head.py
self.transformer = TransformerDecoder(
**transformer_args)
```
* Here we pass the input token(x) into transformer(***token***) and get the desired output
(need to understand IEF_ITERS, assuming its Iterative Error Feedback)
```python=
# 4D-Humans/hmr2/models/heads/smpl_head.py
pred_body_pose = init_body_pose
pred_betas = init_betas
pred_cam = init_cam
pred_body_pose_list = []
pred_betas_list = []
pred_cam_list = []
for i in range(self.cfg.MODEL.SMPL_HEAD.get('IEF_ITERS', 1)):
# Input token to transformer is zero token
if self.input_is_mean_shape:
token = torch.cat([pred_body_pose, pred_betas, pred_cam], dim=1)[:,None,:]
else:
token = torch.zeros(batch_size, 1, 1).to(x.device)
# Pass through transformer
token_out = self.transformer(token, context=x)
token_out = token_out.squeeze(1) # (B, C)
# Readout from token_out
pred_body_pose = self.decpose(token_out) + pred_body_pose
pred_betas = self.decshape(token_out) + pred_betas
pred_cam = self.deccam(token_out) + pred_cam
pred_body_pose_list.append(pred_body_pose)
pred_betas_list.append(pred_betas)
pred_cam_list.append(pred_cam)
# Convert self.joint_rep_type -> rotmat
joint_conversion_fn = {
'6d': rot6d_to_rotmat,
'aa': lambda x: aa_to_rotmat(x.view(-1, 3).contiguous())
}[self.joint_rep_type]
pred_smpl_params_list = {}
pred_smpl_params_list['body_pose'] = torch.cat([joint_conversion_fn(pbp).view(batch_size, -1, 3, 3)[:, 1:, :, :] for pbp in pred_body_pose_list], dim=0)
pred_smpl_params_list['betas'] = torch.cat(pred_betas_list, dim=0)
pred_smpl_params_list['cam'] = torch.cat(pred_cam_list, dim=0)
pred_body_pose = joint_conversion_fn(pred_body_pose).view(batch_size, self.cfg.SMPL.NUM_BODY_JOINTS+1, 3, 3)
pred_smpl_params = {'global_orient': pred_body_pose[:, [0]],
'body_pose': pred_body_pose[:, 1:],
'betas': pred_betas}
return pred_smpl_params, pred_cam, pred_smpl_params_list
```
* ***joint_conversion_fn*** is a function that converts a given representation of joint rotations to rotation matrices. It is determined based on the value of joint_rep_type, which can be either '6d' or 'aa' (axis-angle representation).
* If joint_rep_type is '6d', then joint_conversion_fn refers to the rot6d_to_rotmat function, which converts a batch of 6D rotation representations to corresponding rotation matrices using the method described in the paper by Zhou et al., "On the Continuity of Rotation Representations in Neural Networks", CVPR 2019.
```python=
# 4D-Humans/hmr2/utils/geometry.py
def rot6d_to_rotmat(x: torch.Tensor) -> torch.Tensor:
"""
Convert 6D rotation representation to 3x3 rotation matrix.
Based on Zhou et al., "On the Continuity of Rotation Representations in Neural Networks", CVPR 2019
Args:
x (torch.Tensor): (B,6) Batch of 6-D rotation representations.
Returns:
torch.Tensor: Batch of corresponding rotation matrices with shape (B,3,3).
"""
x = x.reshape(-1,2,3).permute(0, 2, 1).contiguous()
a1 = x[:, :, 0]
a2 = x[:, :, 1]
b1 = F.normalize(a1)
b2 = F.normalize(a2 - torch.einsum('bi,bi->b', b1, a2).unsqueeze(-1) * b1)
b3 = torch.cross(b1, b2)
return torch.stack((b1, b2, b3), dim=-1)
```
* The ***Transformer Decoder(smpl_head)*** will return :
1. ***pred_smpl_params*** : this contains global_orient, body_pose, betas
2. ***pred_cam***
3. ***pred_smpl_params_list***
3. Now using the above outputs we get :
1. pred_cam
2. pred_smpl_params : this contains smpl parameters which help us extract pred_vertices, pred_keypoints_3d etc.
3. pred_cam_t
4. focal_length
5. pred_keypoints_3d
6. pred_vertices
7. pred_keypoints_2d : Using perspective_projection we extract pred_keypoints_2d from pred_keypoints_3d and pred_cam_t
``` python=
# 4D-Humans/hmr2/models/hmr2.py
pred_smpl_params, pred_cam, _ = self.smpl_head(conditioning_feats)
# Store useful regression outputs to the output dict
output = {}
output['pred_cam'] = pred_cam
output['pred_smpl_params'] = {k: v.clone() for k,v in pred_smpl_params.items()}
# Compute camera translation
device = pred_smpl_params['body_pose'].device
dtype = pred_smpl_params['body_pose'].dtype
focal_length = self.cfg.EXTRA.FOCAL_LENGTH * torch.ones(batch_size, 2, device=device, dtype=dtype)
pred_cam_t = torch.stack([pred_cam[:, 1],
pred_cam[:, 2],
2*focal_length[:, 0]/(self.cfg.MODEL.IMAGE_SIZE * pred_cam[:, 0] +1e-9)],dim=-1)
output['pred_cam_t'] = pred_cam_t
output['focal_length'] = focal_length
# Compute model vertices, joints and the projected joints
pred_smpl_params['global_orient'] = pred_smpl_params['global_orient'].reshape(batch_size, -1, 3, 3)
pred_smpl_params['body_pose'] = pred_smpl_params['body_pose'].reshape(batch_size, -1, 3, 3)
pred_smpl_params['betas'] = pred_smpl_params['betas'].reshape(batch_size, -1)
smpl_output = self.smpl(**{k: v.float() for k,v in pred_smpl_params.items()}, pose2rot=False)
pred_keypoints_3d = smpl_output.joints
pred_vertices = smpl_output.vertices
output['pred_keypoints_3d'] = pred_keypoints_3d.reshape(batch_size, -1, 3)
output['pred_vertices'] = pred_vertices.reshape(batch_size, -1, 3)
pred_cam_t = pred_cam_t.reshape(-1, 3)
focal_length = focal_length.reshape(-1, 2)
pred_keypoints_2d = perspective_projection(pred_keypoints_3d, translation=pred_cam_t, focal_length=focal_length / self.cfg.MODEL.IMAGE_SIZE)
output['pred_keypoints_2d'] = pred_keypoints_2d.reshape(batch_size, -1, 2)
```
Renderer
---
###### (refer 4D-Humans/hmr2/utils/renderer.py)
* Resize image to size 255x255
``` python=
image = image.clone() * torch.tensor(self.cfg.MODEL.IMAGE_STD, device=image.device).reshape(3,1,1)
image = image + torch.tensor(self.cfg.MODEL.IMAGE_MEAN, device=image.device).reshape(3,1,1)
image = image.permute(1, 2, 0).cpu().numpy()
```
* Initialize renderer and material
``` python=
renderer = pyrender.OffscreenRenderer(viewport_width=image.shape[1],
viewport_height=image.shape[0],
point_size=1.0)
material = pyrender.MetallicRoughnessMaterial(
metallicFactor=0.0,
alphaMode='OPAQUE',
baseColorFactor=(*mesh_base_color, 1.0))
```
* We create mesh from pred_vertices(from model output)
```python=
mesh = trimesh.Trimesh(vertices.copy(), self.faces.copy())
```
* To get top view or side view we rotate the mesh
``` python=
if side_view:
rot = trimesh.transformations.rotation_matrix(
np.radians(rot_angle), [0, 1, 0])
mesh.apply_transform(rot)
elif top_view:
rot = trimesh.transformations.rotation_matrix(
np.radians(rot_angle), [1, 0, 0])
mesh.apply_transform(rot)
```
* We convert the trimesh to pyrender mesh and add it to scene (It contains the 3D objects (meshes, lights, cameras) that you want to render).
```python=
mesh = pyrender.Mesh.from_trimesh(mesh, material=material)
# initialize pyrender scene
scene = pyrender.Scene(bg_color=[*scene_bg_color, 0.0],
ambient_light=(0.3, 0.3, 0.3))
scene.add(mesh, 'mesh')
```
* Add camera to scene
```python=
camera_pose = np.eye(4)
camera_pose[:3, 3] = camera_translation
camera_center = [image.shape[1] / 2., image.shape[0] / 2.]
camera = pyrender.IntrinsicsCamera(fx=self.focal_length,
fy=self.focal_length,
cx=camera_center[0],
cy=camera_center[1],
zfar=1e12)
scene.add(camera, pose=camera_pose)
```
* Adding lights
``` python=
light_nodes = create_raymond_lights()
for node in light_nodes:
scene.add_node(node)
```
* Get image(color variable) from the scene using renderer.render
```python=
color, rend_depth = renderer.render(scene, flags=pyrender.RenderFlags.RGBA)
color = color.astype(np.float32) / 255.0
renderer.delete()
```
>visualize of ***color*** variable
* Add this mesh to the resized image
```python=
valid_mask = (color[:, :, -1])[:, :, np.newaxis]
if not side_view and not top_view:
output_img = (color[:, :, :3] * valid_mask + (1 - valid_mask) * image)
else:
output_img = color[:, :, :3]
output_img = output_img.astype(np.float32)
```
>
Evaluation
---
* Testing on 3DPW test dataset.
* ***MPJPE Mean Per Joint Position Error*** (in mm) It measures the average Euclidean distance from prediction to ground truth joint positions. The evaluation adjusts the translation (tx,ty,tz) of the prediction to match the ground truth.
>$\text{mpjpe} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{D} (\text{pred_joints}_{ij} - \text{gt_joints}_{ij})^2}$
>Where:
**mpjpe** represents the mean per joint position error.
**N** is the number of samples.
**D** is the number of dimensions (e.g., 2 for 2D coordinates, 3 for 3D coordinates).
**pred_joints** is the jth dimension of the predicted joint position for the iith sample.
**gt_joints** is the jth dimension of the ground truth joint position for the iith sample.
* ***RE Reconstruction Error*** Computes the mean Euclidean distance of 2 set of points S1, S2 after performing Procrustes alignment.
>$\text{re} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{D} \left(\hat{S}_{1_{ij}} - S_{2_{ij}}\right)^2}$
>Where:
**re** represents the reconstruction error.
**N** is the number of samples.
**D** is the number of dimensions (e.g., 3 for 3D coordinates).
**S1ij** is the jth dimension of the aligned predicted joint position for the ith sample.
**S2ij** is the jth dimension of the ground truth joint position for the ith sample.
```python=
# re
def reconstruction_error(S1, S2) -> np.array:
"""
Computes the mean Euclidean distance of 2 set of points S1, S2 after performing Procrustes alignment.
Args:
S1 (torch.Tensor): First set of points of shape (B, N, 3).
S2 (torch.Tensor): Second set of points of shape (B, N, 3).
Returns:
(np.array): Reconstruction error.
"""
S1_hat = compute_similarity_transform(S1, S2)
re = torch.sqrt( ((S1_hat - S2)** 2).sum(dim=-1)).mean(dim=-1)
return re
#mpjre and re
def eval_pose(pred_joints, gt_joints) -> Tuple[np.array, np.array]:
"""
Compute joint errors in mm before and after Procrustes alignment.
Args:
pred_joints (torch.Tensor): Predicted 3D joints of shape (B, N, 3).
gt_joints (torch.Tensor): Ground truth 3D joints of shape (B, N, 3).
Returns:
Tuple[np.array, np.array]: Joint errors in mm before and after alignment.
"""
# Absolute error (MPJPE)
mpjpe = torch.sqrt(((pred_joints - gt_joints) ** 2).sum(dim=-1)).mean(dim=-1).cpu().numpy()
# Reconstruction_error
r_error = reconstruction_error(pred_joints, gt_joints).cpu().numpy()
return 1000 * mpjpe, 1000 * r_error
pred_keypoints_3d = output['pred_keypoints_3d'].detach()
pred_keypoints_3d = pred_keypoints_3d[:,None,:,:]
batch_size = pred_keypoints_3d.shape[0]
num_samples = pred_keypoints_3d.shape[1]
gt_keypoints_3d = batch['keypoints_3d'][:, :, :-1].unsqueeze(1).repeat(1, num_samples, 1, 1)
# Align predictions and ground truth such that the pelvis location is at the origin
pred_keypoints_3d -= pred_keypoints_3d[:, :, [self.pelvis_ind]]
gt_keypoints_3d -= gt_keypoints_3d[:, :, [self.pelvis_ind]]
# Compute joint errors
mpjpe, re = eval_pose(pred_keypoints_3d.reshape(batch_size * num_samples, -1, 3)[:, self.keypoint_list], gt_keypoints_3d.reshape(batch_size * num_samples, -1 ,3)[:, self.keypoint_list])
```
* ***kp_l2_loss***: compute 2d keypoint error
```python=
# Compute 2d keypoint errors
pred_keypoints_2d = output['pred_keypoints_2d'].detach()
pred_keypoints_2d = pred_keypoints_2d[:,None,:,:]
gt_keypoints_2d = batch['keypoints_2d'][:,None,:,:].repeat(1, num_samples, 1, 1)
conf = gt_keypoints_2d[:, :, :, -1].clone()
kp_err = torch.nn.functional.mse_loss(
pred_keypoints_2d,
gt_keypoints_2d[:, :, :, :-1],
reduction='none'
).sum(dim=3)
kp_l2_loss = (conf * kp_err).mean(dim=2)
kp_l2_loss = kp_l2_loss.detach().cpu().numpy()
```
* After running 4D-Humans/eval.py on 3DPW test dataset, here are the results :
* **re** : 54.32551111588927
* **mpjpe** : 81.26772677982657
PHALP
---
### Tracker(Matching tracks and detections)
* Initialize:
* Metric using is **NearestNeighborDistanceMetric**
```python=
def setup_deepsort(self):
log.info("Setting up DeepSort...")
metric = nn_matching.NearestNeighborDistanceMetric(self.cfg, self.cfg.phalp.hungarian_th, self.cfg.phalp.past_lookback)
self.tracker = Tracker(self.cfg, metric, max_age=self.cfg.phalp.max_age_track, n_init=self.cfg.phalp.n_init, phalp_tracker=self, dims=[4096, 4096, 99])
```
* Tracker wll contain set of tracks. Tracks basically contains tracking data of each person.
* This is code from trackers/PHALP.py
```python=
############ tracking ##############
self.tracker.predict()
self.tracker.update(detections, t_, frame_name, self.cfg.phalp.shot)
```
* `self.tracker.predict()` does is for each track it will increase age & time_since_update by 1.
```python=
#deep_sort_/tracker.py
def predict(self):
"""Propagate track state distributions one time== step forward.
This function should be called once every time step, before `update`.==
"""
for track in self.tracks:
track.predict(self.phalp_tracker, increase_age=True)
#deep_sort_/track.py
def predict(self, phalp_tracker, increase_age=True):
if(increase_age):
self.age += 1; self.time_since_update += 1
```
* `self.tracker.update()` will perform measurement update and track management. Function inputs are detections(outputs from HMR2.0), frame_name and self.cfg.phalp.shot which tells whether there is scene change in frame or not.
* Using deep_sort_/linear_assignment.py we try to match detection with tracks. min_cost_matching() is where this matching takes place. distance_matric over here is gated_metric from tracker.py that cost matrix of shape len(targets), len(features), where element (i, j) contains the closest squared distance between `targets[i]` and `features[j]`.
* Basically we have tracking data for before frames. Now after running HMR on new frame we try to match detected people with tracking data for before frames using nearest neighbor distance metric (Euclidean) (Check out linear_assignment.py and nn_matching).
### self.tracker.update(self, detections, frame_t, image_name, shot)
* Below statistics = [cost_matrix, track_gt, detect_gt, track_idt, detect_idt]. Matches contain list of matching tracks and detections.
```python=
matches, unmatched_tracks, unmatched_detections, statistics = self._match(detections)
self.tracked_cost[frame_t] = [statistics[0], matches, unmatched_tracks, unmatched_detections, statistics[1], statistics[2], statistics[3], statistics[4]]
```
* If we get unmatched detections then we create new track
```python=
for detection_idx in unmatched_detections:
self._initiate_track(detections[detection_idx], detection_idx)
def _initiate_track(self, detection, detection_id):
new_track = Track(self.cfg, self._next_id, self.n_init, self.max_age,
detection_data=detection.detection_data,
detection_id=detection_id,
dims=[self.A_dim, self.P_dim, self.L_dim])
new_track.add_predicted()
self.tracks.append(new_track)
self._next_id += 1
```
* If track is not found/matched from detections then we wait for certain age(self._max_age) or if the track is tentative, then delete
```python=
for track_idx in unmatched_tracks:
self.tracks[track_idx].mark_missed()
self.tracks = [t for t in self.tracks if not t.is_deleted()]
def mark_missed(self):
"""Mark this track as missed (no association at the current time step).
"""
if self.state == TrackState.Tentative:
self.state = TrackState.Deleted
elif self.time_since_update > self._max_age:
self.state = TrackState.Deleted
```
* If track and detection is matching then we append the track with latest detection values
```python=
# tracker.py
for track_idx, detection_idx in matches:
self.tracks[track_idx].update(detections[detection_idx], detection_idx, shot)
self.accumulate_vectors([i[0] for i in matches], features=self.cfg.phalp.predict)
# track.py
def update(self, detection, detection_id, shot):
self.track_data["history"].append(copy.deepcopy(detection.detection_data))
```
* Accumulate vectors will save all tracks pose and location(we use these for prediction) features under p_features and l_features respectively and then predict.
```python=
def accumulate_vectors(self, track_ids, features="APL"):
a_features = []; p_features = []; l_features = []; t_features = []; l_time = []; confidence = []; is_tracks = 0; p_data = []
for track_idx in track_ids:
t_features.append([self.tracks[track_idx].track_data['history'][i]['time'] for i in range(self.cfg.phalp.track_history)])
l_time.append(self.tracks[track_idx].time_since_update)
if("L" in features): l_features.append(np.array([self.tracks[track_idx].track_data['history'][i]['loca'] for i in range(self.cfg.phalp.track_history)]))
if("P" in features): p_features.append(np.array([self.tracks[track_idx].track_data['history'][i]['pose'] for i in range(self.cfg.phalp.track_history)]))
if("P" in features): t_id = self.tracks[track_idx].track_id; p_data.append([[data['xy'][0], data['xy'][1], data['scale'], data['scale'], data['time'], t_id] for data in self.tracks[track_idx].track_data['history']])
if("L" in features): confidence.append(np.array([self.tracks[track_idx].track_data['history'][i]['conf'] for i in range(self.cfg.phalp.track_history)]))
is_tracks = 1
```
* Now we run prediction
```python=
if(is_tracks):
with torch.no_grad():
if("P" in features): p_pred = self.phalp_tracker.forward_for_tracking([p_features, p_data, t_features], "P", l_time)
if("L" in features): l_pred = self.phalp_tracker.forward_for_tracking([l_features, t_features, confidence], "L", l_time)
```
### POSE TRANSFORMER (pose_prediction)
(refer phalp/models/predictor/pose_transformer_v2.py)
* We use it in predicting pose(forward_for_tracking() from phalp/trackers/PHALP.py)
* We use Bidirectional Encoder Representations from Transformers(BERT).
* This prediction work is taken from LART paper(https://arxiv.org/pdf/2304.01199)
* If any track detection is missing then we mask it.
### Location prediction
(refer phalp/trackers/PHALP.py forward_for_tracking function)
* We try to predict location values i.e n, x & y.
* We use Ridge(Linear) regression for this prediction
Rough Points
---
* In fluid mechanics, we have Lagrangian and Eulerian specifications of the flow field.
* In Eulerian, we fix specific locations in space through which fluid flows as time passes (Concerned with the fluid properties at a specific space-time point).
* In Lagrangian we fix particular particles as it moves through space.
* LART is basicallyy focused on Lagrangian viewpoint for analysing human actions.
* Given these 3D representations of people (i.e., 3D pose and 3D location), we use them as the basic content of each token.This allows us to build a flexible system where the model, here a transformer, takes as input tokens corresponding to the different people with access to their identity, 3D pose and 3D location.
* In PHALP paper(earlier version), for prediction they have used probability but here we use BERT.
To look into (IMPORTANT)
---
* The predicted pose and location values are not at all used. They are deleted afterwards.
* 4D-humans on video is just running HMR-2.0 on each frame and generating meshes.
* https://github.com/shubham-goel/4D-Humans/issues/29
Sign in with Wallet
Connect another wallet