---
# System prepended metadata

title: Face module functionalities in details
tags: [Edsolabs]

---

---
title: Face module functionalities in details
tags: Edsolabs
---

## Foundation
**Base model export engine**.
* *Purpose*. TensorRT engine wrapper for model export from ONNX to TRT
* *Actors*.
    * *Developers*. Developers use this module to export ONNX models to TRT for development and deployment purpose
    * *System admin*. System admins use this module to export model whenever there is a model update
* *Overall functionality*.
    * Export model from ONNX to TRT given configuration file
* *Detailed functionality*.
    * *Initialization function*.
        * Load configuration
        * Load TensorRT components
    * *Iteration functions*.
        * Load calibration table (for INT8 mode)
        * Calibrate model (for INT8 mode)
        * Export model from ONNX to TRT
    * *Termination functions*.
        * Release TensorRT components
    * *Debug functions*.
        * Log debug messages

**Base model inference engine**.
* *Purpose*. Interface for perception modules in Primary analysis
* *Actors*.
    * *Developers*. Developers use this module as an interface for developing perception modules in Primary analysis, and for optimization purpose
    * *System admin*. System admins use this module to
        * Start the system, i.e. load inference engines
        * Plug-and-play modules in the system to ensure reasonable workload during system execution, i.e. by releasing (when not used) and reloading models (when needed)
        * Terminate the system, i.e. release inference engines
    * *Primary analysis modules*. Use this module as the base module
* *Overall functionality*. Contain must-have functionality of each perception module
    * Load and release TensorRT models
    * Run TensorRT model for inference
    * Reload TensorRT model
* *Detailed functionality*.
    * *Initialization function*.
        * Load configuration
        * Load model
    * *Iteration functions*.
        * Reload model
        * Run model as a whole, i.e. pre-process + run model + postprocess
        * Run model step-by-step (for optimization purpose)
    * *Termination functions*.
        * Release model
    * *Debug functions*.
        * Visualize results
        * Measure runtime
        * Log debug messages

**Base module**. A tempate parametrized by input / output types
* *Purpose*. Interface for processing modules in the whole system
* *Actors*.
    * *Developers*. Developers use this module as an interface for developing processing modules in Primary analysisand Secondary analysis
    * *System admin*. System admins use this module to
        * Start the system, i.e. load modules
        * Plug-and-play modules in the system to ensure reasonable workload during system execution, i.e. by releasing (when not used) and reloading modules (when needed)
        * Terminate the system, i.e. release modules
    * *Primary analysis modules*. Use this module as the base module
    * *Secondary analysis modules*. Use this module as the base module
* *Overall functionality*. Contain must-have functionality of each processing module
    * Load and release TensorRT models
    * Run TensorRT model for inference
    * Reload TensorRT model
    * Record module log files during system execution
* *Detailed functionality*.
    * *Initialization functions*.
        * Load configuration
        * Load module
    * *Iteration functions*.
        * Reload module
        * Run module as whole
        * Run module step-by-step (for optimization purpose)
    * *Termination functions*.
        * Release module
    * *Debug functions*.
        * Visualize results
        * Measure runtime
        * Log debug messages
        * Save log file during execution and core-dump file

**Health checker module**. A class responsible for checking the health of all modules and report to master node if there is any failure
* *Purpose*. For systems running for a long time, we need a health checker to automatically detect system failure
    * *Explain*.
        * All modules will record a log file during running, containing the most recent states of them before failure
        * The health checker periodically probes the modules to see if they are down or not
        * If there is a module failure, the health checker takes the module's log file and report to the master node
        * The master node then may decide to restart the failed module, or report the problem to system admin
    * *Log files*. Include core-dump file (in case the module runs into segment fault core-dump) and log file produced by the failed module
* *Actors*.
    * *Developers*. Developers use this module as a tool for developing and debugging processing modules in Primary analysis
    * *System admin*. System admins use this module as a tool for system monitoring, i.e. detect failures, and debugging
* *Overall functionality*.
    * Probe system modules to check for module failure
    * Report module failures to master node

## Primary analysis (Perception modules)
**Perception modules**. Carry out perception analysis on input camera streams
* *Main functionalities*.
    * Communicate with RTSP camera streams, feature modules, and communication channels (for debugging purpose) 
    * Run model
    * Visualize results
    * Measure runtime

### Foundation
**Base detection module**. Inherited from *Model inference engine* and *Base module*
* *Purpose*. Base module for detection modules in Primary analysis
* *Actors*.
    * *Developers*. Developers use this module as an interface for developing object detection modules in Primary analysis, and for optimization purpose
    * *Primary analysis object detection modules*. Use this module as the base module
    * *Secondary analysis modules*. Use this module for related processing tasks
* *Overall functionality*. Run object detection model with the following I/O
    * *Input*. RGB frame
    * *Output*. A list of detected boxes, each box is a struct of (box, class, score)
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Model inference engine* and *Base module*
    * *Iteration functions*. Inherited from *Model inference engine* and *Base module*
    * *Termination functions*. Inherited from *Model inference engine* and *Base module*
    * *Debug functions*. Inherited from *Model inference engine* and *Base module*
        * Stream out detected boxes
        * Stream out measured runtime
        * Stream out log messages

**Base tracking module**. Inherited from *Model inference engine* (if use Deep SORT) and *Base module*
* *Purpose*. Base module for object tracking modules in Primary analysis
* *Actors*.
    * *Developers*. Developers use this module as an interface for developing object tracking modules in Primary analysis, and for optimization purpose
    * *Primary analysis object tracking modules*. Use this module as the base module
    * *Secondary analysis modules*. Use this module for related processing tasks
* *Overall functionality*. Run object tracking with the following I/O
    * *Input*. RGB frame and a list detected boxes
    * *Output*. A list of tracked boxes, i.e. detected boxes with IDs
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module*
    * *Iteration functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module*
    * *Termination functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module*
    * *Debug functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module*
        * Stream out tracked boxes with IDs
        * Stream out measured runtime
        * Stream out log messages

**Base classification module**. Inherited from *Model inference engine* and *Base module*
* *Purpose*. Base module for object classification modules in Primary analysis
* *Actors*.
    * *Developers*. Developers use this module as an interface for developing object classification modules in Primary analysis, and for optimization purpose
    * *Primary analysis object classification modules*. Use this module as the base module
    * *Secondary analysis modules*. Use this module for related processing tasks
* *Overall functionality*. Run object classification with the following I/O
    * *Input*. Cropped RGB images
    * *Output*. A list of classification results, i.e. structs of format (class, score)
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Model inference engine* and *Base module*
    * *Iteration functions*. Inherited from *Model inference engine* and *Base module*
    * *Termination functions*. Inherited from *Model inference engine* and *Base module*
    * *Debug functions*. Inherited from *Model inference engine* and *Base module*
        * Stream out detected boxes with class name
        * Stream out measured time
        * Stream out log messages

### Priority 1
**Human detection**. Inherited from *Base deteciton module*
* *Purpose*. Detect people in a camera stream
* *Actors*. Inherited from *Base deteciton module*
* *Overall functionality*. Detect people in a camera stream
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base deteciton module*
    * *Iteration functions*. Inherited from *Base deteciton module*
    * *Termination functions*. Inherited from *Base deteciton module*
    * *Debug functions*. Inherited from *Base deteciton module*
* *Notes*.
    * How to handle mannequin and other human-like objects, e.g. statues, toys, models, etc.
    * How to handle the case where some one is reflected on a glass, i.e. 2 different people will be detected
    * Are babies considered "human", i.e. objects which must be detected

**Human tracking**. Inherited from *Base tracking module*
* *Purpose*. Track detected people in a camera stream
* *Actors*. Inherited from *Base tracking module*
* *Overall functionality*. Track detected people in a camera stream
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base tracking module*
    * *Iteration functions*. Inherited from *Base tracking module*
    * *Termination functions*. Inherited from *Base tracking module*
    * *Debug functions*. Inherited from *Base tracking module*
* *Notes*.
    * How to handle overlapping objects during tracking?
    * How to handle occluded objects during tracking?
    * How to handle visually similar objects during tracking? (if use Deep SORT)

### Priority 2
**Human descriptor by clothes color**. Inherited from *Base classification module*
* *Purpose*. Extract clothes color information from detected people in a camera stream
* *Actors*. Inherited from *Base classification module*
* *Overall functionality*. Extract clothes color information from detected people in a camera stream
* *Detailed functionality*. NEED MORE SURVEY
* *Note*. NEED MORE SURVEY
    * *Idea*.
        * Divide human body into top and bottom and formulate the problem as color classification
            * *Drawbacks*. Difficult to cope with clothes with multiple colors
        * Use metric learning, embed clothes colors into vectors and use clothes recognition

**Human action recognition**. Inherited from *Base classification module*
* *Purpose*. Recognize behaviors of people in a camera stream
* *Actors*. Inherited from *Base classification module*
* *Overall functionality*. Recognize behaviors of people in a camera stream
* *Detailed functionality*. NEED MORE SURVEY
* *Note*. NEED MORE SURVEY

## Secondary analysis (Feature modules)
**Feature modules**. Carry out analysis based on perception outputs, without interacting with camera streams
* *Main functionalities*.
    * Communicate with output communication channel and DB
    * Run module
    * Visualize results
    * Measure runtime

### Priority 1
**Human searching by ID**. Inherited from *Base module*
* *Purpose*. Search for people in a camera stream based on their IDs
* *Actors*.
* *Overall functionality*. 
    * Search for people according to the following I/O
        * *Input*. A list of IDs for searching
        * *Output*. A list of histories (trajectories and behaviors) of the desired people
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base module*
    * *Iteration functions*. Inherited from *Base module*
    * *Termination functions*. Inherited from *Base module*
    * *Debug functions*. Inherited from *Base module*
        * Stream out histories of the desired people to a file
        * Stream out measured time
        * Stream out log messages

**Human searching by clothes color**. Inherited from *Base module*
* *Purpose*. Search for people in a camera stream based on clothes color
* *Actors*.
* *Overall functionality*. 
    * Search for people according to the following I/O
        * *Input*. A list of clothes colors for searching
        * *Output*. A list of histories (trajectories and behaviors) of the desired people
* *Detailed functionality*. NEED MORE SURVEY

**Human searching by behavior**. Inherited from *Base module*
* *Purpose*. Search for people in a camera stream based on behavior
* *Actors*.
* *Overall functionality*. 
    * Search for people according to the following I/O
        * *Input*. A list of behaviors for searching
        * *Output*. A list of histories (trajectories and behaviors) of the desired people
* *Detailed functionality*. NEED MORE SURVEY

### Priority 2
**People counting**. Inherited from *Base module*
* *Purpose*. Count the number of people appeared in the video stream
* *Actors*.
* *Overall functionality*. 
    * Count the number of people appeared in the video stream
        * *Input*. A list of tracked boxes with IDs
        * *Output*. Number of people in framedesired people
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base module*
    * *Iteration functions*. Inherited from *Base module*
    * *Termination functions*. Inherited from *Base module*
    * *Debug functions*. Inherited from *Base module*
        * Stream out the number of people on the camera stream
        * Stream out measured time
        * Stream out log messages

**Crowd detection**. Inherited from *Base module*
* *Purpose*. Detect crowds in a camera stream
* *Actors*.
* *Overall functionality*. 
    * Detect crowds in a camera stream
        * *Input*. A list of tracked boxes with IDs
        * *Output*. A list of detected crowds
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base module*
    * *Iteration functions*. Inherited from *Base module*
    * *Termination functions*. Inherited from *Base module*
    * *Debug functions*. Inherited from *Base module*
        * Stream out the crowds information
        * Stream out measured time
        * Stream out log messages

**Violence detection**. Inherited from *Base module*
* *Explain*. PENDING

**Abandoning position detection**. Inherited from *Base module*
* *Purpose*. Detect abandoning positions in a camera stream
* *Actors*.
* *Overall functionality*. 
    * Detect abandoning positions in a camera stream
        * *Input*. A list of tracked boxes with IDs
        * *Output*. A list of abandoning positions
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base module*
    * *Iteration functions*. Inherited from *Base module*
    * *Termination functions*. Inherited from *Base module*
    * *Debug functions*. Inherited from *Base module*
        * Stream out the abandoning positions
        * Stream out measured time
        * Stream out log messages

**Human instrusion detection by virtual bonudary**. Inherited from *Base module*
* *Idea*.
    * *Option 1*. Have a virtual boundary and any box crossing the boundary is an invalid box
        * *Pros*. Easy to implement
        * *Cons*. Not very accurate
    * *Option 2*. Map the virtual boundary and the detected boxes to ground plane and any box crossing the boundary is an invalid box
        * *Pros*. Hard to implement
        * *Cons*. More accurate than option 1
* *Purpose*. Detect human instrusion in a camera stream
* *Actors*.
* *Overall functionality*. 
    * Detect human instrusion in a camera stream
        * *Input*. A list of tracked boxes with IDs
        * *Output*. A list of valid and invalid boxes, i.e. boxes which violate the virtual boundary
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base module*
    * *Iteration functions*. Inherited from *Base module*
    * *Termination functions*. Inherited from *Base module*
    * *Debug functions*. Inherited from *Base module*
        * Stream out tracked boxes with valid-or-invalid predictions
        * Stream out measured time
        * Stream out log messages

**Human instrusion detection by restricted region**. Inherited from *Base module*
* *Idea*.
    * *Option 1*. Have a virtual boundary and any box crossing the boundary is an invalid box
        * *Pros*. Easy to implement
        * *Cons*. Not very accurate
    * *Option 2*. Map the virtual boundary and the detected boxes to ground plane and any box crossing the boundary is an invalid box
        * *Pros*. Hard to implement
        * *Cons*. More accurate than option 1
* *Purpose*. Detect human instrusion in a camera stream
* *Actors*.
* *Overall functionality*. 
    * Detect human instrusion in a camera stream
        * *Input*. A list of tracked boxes with IDs
        * *Output*. A list of valid and invalid boxes, i.e. boxes which violate the restricted region
* *Detailed functionality*.
    * *Initialization functions*. Inherited from *Base module*
    * *Iteration functions*. Inherited from *Base module*
    * *Termination functions*. Inherited from *Base module*
    * *Debug functions*. Inherited from *Base module*
        * Stream out tracked boxes with valid-or-invalid predictions
        * Stream out measured time
        * Stream out log messages

### Priority 3
**Human tracing**. Inherited from *Base module*
* *Note*. PENDING