Face module functionalities in details

--- title: Face module functionalities in details tags: Edsolabs --- ## Foundation **Base model export engine**. * *Purpose*. TensorRT engine wrapper for model export from ONNX to TRT * *Actors*. * *Developers*. Developers use this module to export ONNX models to TRT for development and deployment purpose * *System admin*. System admins use this module to export model whenever there is a model update * *Overall functionality*. * Export model from ONNX to TRT given configuration file * *Detailed functionality*. * *Initialization function*. * Load configuration * Load TensorRT components * *Iteration functions*. * Load calibration table (for INT8 mode) * Calibrate model (for INT8 mode) * Export model from ONNX to TRT * *Termination functions*. * Release TensorRT components * *Debug functions*. * Log debug messages **Base model inference engine**. * *Purpose*. Interface for perception modules in Primary analysis * *Actors*. * *Developers*. Developers use this module as an interface for developing perception modules in Primary analysis, and for optimization purpose * *System admin*. System admins use this module to * Start the system, i.e. load inference engines * Plug-and-play modules in the system to ensure reasonable workload during system execution, i.e. by releasing (when not used) and reloading models (when needed) * Terminate the system, i.e. release inference engines * *Primary analysis modules*. Use this module as the base module * *Overall functionality*. Contain must-have functionality of each perception module * Load and release TensorRT models * Run TensorRT model for inference * Reload TensorRT model * *Detailed functionality*. * *Initialization function*. * Load configuration * Load model * *Iteration functions*. * Reload model * Run model as a whole, i.e. pre-process + run model + postprocess * Run model step-by-step (for optimization purpose) * *Termination functions*. * Release model * *Debug functions*. * Visualize results * Measure runtime * Log debug messages **Base module**. A tempate parametrized by input / output types * *Purpose*. Interface for processing modules in the whole system * *Actors*. * *Developers*. Developers use this module as an interface for developing processing modules in Primary analysisand Secondary analysis * *System admin*. System admins use this module to * Start the system, i.e. load modules * Plug-and-play modules in the system to ensure reasonable workload during system execution, i.e. by releasing (when not used) and reloading modules (when needed) * Terminate the system, i.e. release modules * *Primary analysis modules*. Use this module as the base module * *Secondary analysis modules*. Use this module as the base module * *Overall functionality*. Contain must-have functionality of each processing module * Load and release TensorRT models * Run TensorRT model for inference * Reload TensorRT model * Record module log files during system execution * *Detailed functionality*. * *Initialization functions*. * Load configuration * Load module * *Iteration functions*. * Reload module * Run module as whole * Run module step-by-step (for optimization purpose) * *Termination functions*. * Release module * *Debug functions*. * Visualize results * Measure runtime * Log debug messages * Save log file during execution and core-dump file **Health checker module**. A class responsible for checking the health of all modules and report to master node if there is any failure * *Purpose*. For systems running for a long time, we need a health checker to automatically detect system failure * *Explain*. * All modules will record a log file during running, containing the most recent states of them before failure * The health checker periodically probes the modules to see if they are down or not * If there is a module failure, the health checker takes the module's log file and report to the master node * The master node then may decide to restart the failed module, or report the problem to system admin * *Log files*. Include core-dump file (in case the module runs into segment fault core-dump) and log file produced by the failed module * *Actors*. * *Developers*. Developers use this module as a tool for developing and debugging processing modules in Primary analysis * *System admin*. System admins use this module as a tool for system monitoring, i.e. detect failures, and debugging * *Overall functionality*. * Probe system modules to check for module failure * Report module failures to master node ## Primary analysis (Perception modules) **Perception modules**. Carry out perception analysis on input camera streams * *Main functionalities*. * Communicate with RTSP camera streams, feature modules, and communication channels (for debugging purpose) * Run model * Visualize results * Measure runtime ### Foundation **Base detection module**. Inherited from *Model inference engine* and *Base module* * *Purpose*. Base module for detection modules in Primary analysis * *Actors*. * *Developers*. Developers use this module as an interface for developing object detection modules in Primary analysis, and for optimization purpose * *Primary analysis object detection modules*. Use this module as the base module * *Secondary analysis modules*. Use this module for related processing tasks * *Overall functionality*. Run object detection model with the following I/O * *Input*. RGB frame * *Output*. A list of detected boxes, each box is a struct of (box, class, score) * *Detailed functionality*. * *Initialization functions*. Inherited from *Model inference engine* and *Base module* * *Iteration functions*. Inherited from *Model inference engine* and *Base module* * *Termination functions*. Inherited from *Model inference engine* and *Base module* * *Debug functions*. Inherited from *Model inference engine* and *Base module* * Stream out detected boxes * Stream out measured runtime * Stream out log messages **Base tracking module**. Inherited from *Model inference engine* (if use Deep SORT) and *Base module* * *Purpose*. Base module for object tracking modules in Primary analysis * *Actors*. * *Developers*. Developers use this module as an interface for developing object tracking modules in Primary analysis, and for optimization purpose * *Primary analysis object tracking modules*. Use this module as the base module * *Secondary analysis modules*. Use this module for related processing tasks * *Overall functionality*. Run object tracking with the following I/O * *Input*. RGB frame and a list detected boxes * *Output*. A list of tracked boxes, i.e. detected boxes with IDs * *Detailed functionality*. * *Initialization functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module* * *Iteration functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module* * *Termination functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module* * *Debug functions*. Inherited from *Model inference engine* (if use Deep SORT) and *Base module* * Stream out tracked boxes with IDs * Stream out measured runtime * Stream out log messages **Base classification module**. Inherited from *Model inference engine* and *Base module* * *Purpose*. Base module for object classification modules in Primary analysis * *Actors*. * *Developers*. Developers use this module as an interface for developing object classification modules in Primary analysis, and for optimization purpose * *Primary analysis object classification modules*. Use this module as the base module * *Secondary analysis modules*. Use this module for related processing tasks * *Overall functionality*. Run object classification with the following I/O * *Input*. Cropped RGB images * *Output*. A list of classification results, i.e. structs of format (class, score) * *Detailed functionality*. * *Initialization functions*. Inherited from *Model inference engine* and *Base module* * *Iteration functions*. Inherited from *Model inference engine* and *Base module* * *Termination functions*. Inherited from *Model inference engine* and *Base module* * *Debug functions*. Inherited from *Model inference engine* and *Base module* * Stream out detected boxes with class name * Stream out measured time * Stream out log messages ### Priority 1 **Human detection**. Inherited from *Base deteciton module* * *Purpose*. Detect people in a camera stream * *Actors*. Inherited from *Base deteciton module* * *Overall functionality*. Detect people in a camera stream * *Detailed functionality*. * *Initialization functions*. Inherited from *Base deteciton module* * *Iteration functions*. Inherited from *Base deteciton module* * *Termination functions*. Inherited from *Base deteciton module* * *Debug functions*. Inherited from *Base deteciton module* * *Notes*. * How to handle mannequin and other human-like objects, e.g. statues, toys, models, etc. * How to handle the case where some one is reflected on a glass, i.e. 2 different people will be detected * Are babies considered "human", i.e. objects which must be detected **Human tracking**. Inherited from *Base tracking module* * *Purpose*. Track detected people in a camera stream * *Actors*. Inherited from *Base tracking module* * *Overall functionality*. Track detected people in a camera stream * *Detailed functionality*. * *Initialization functions*. Inherited from *Base tracking module* * *Iteration functions*. Inherited from *Base tracking module* * *Termination functions*. Inherited from *Base tracking module* * *Debug functions*. Inherited from *Base tracking module* * *Notes*. * How to handle overlapping objects during tracking? * How to handle occluded objects during tracking? * How to handle visually similar objects during tracking? (if use Deep SORT) ### Priority 2 **Human descriptor by clothes color**. Inherited from *Base classification module* * *Purpose*. Extract clothes color information from detected people in a camera stream * *Actors*. Inherited from *Base classification module* * *Overall functionality*. Extract clothes color information from detected people in a camera stream * *Detailed functionality*. NEED MORE SURVEY * *Note*. NEED MORE SURVEY * *Idea*. * Divide human body into top and bottom and formulate the problem as color classification * *Drawbacks*. Difficult to cope with clothes with multiple colors * Use metric learning, embed clothes colors into vectors and use clothes recognition **Human action recognition**. Inherited from *Base classification module* * *Purpose*. Recognize behaviors of people in a camera stream * *Actors*. Inherited from *Base classification module* * *Overall functionality*. Recognize behaviors of people in a camera stream * *Detailed functionality*. NEED MORE SURVEY * *Note*. NEED MORE SURVEY ## Secondary analysis (Feature modules) **Feature modules**. Carry out analysis based on perception outputs, without interacting with camera streams * *Main functionalities*. * Communicate with output communication channel and DB * Run module * Visualize results * Measure runtime ### Priority 1 **Human searching by ID**. Inherited from *Base module* * *Purpose*. Search for people in a camera stream based on their IDs * *Actors*. * *Overall functionality*. * Search for people according to the following I/O * *Input*. A list of IDs for searching * *Output*. A list of histories (trajectories and behaviors) of the desired people * *Detailed functionality*. * *Initialization functions*. Inherited from *Base module* * *Iteration functions*. Inherited from *Base module* * *Termination functions*. Inherited from *Base module* * *Debug functions*. Inherited from *Base module* * Stream out histories of the desired people to a file * Stream out measured time * Stream out log messages **Human searching by clothes color**. Inherited from *Base module* * *Purpose*. Search for people in a camera stream based on clothes color * *Actors*. * *Overall functionality*. * Search for people according to the following I/O * *Input*. A list of clothes colors for searching * *Output*. A list of histories (trajectories and behaviors) of the desired people * *Detailed functionality*. NEED MORE SURVEY **Human searching by behavior**. Inherited from *Base module* * *Purpose*. Search for people in a camera stream based on behavior * *Actors*. * *Overall functionality*. * Search for people according to the following I/O * *Input*. A list of behaviors for searching * *Output*. A list of histories (trajectories and behaviors) of the desired people * *Detailed functionality*. NEED MORE SURVEY ### Priority 2 **People counting**. Inherited from *Base module* * *Purpose*. Count the number of people appeared in the video stream * *Actors*. * *Overall functionality*. * Count the number of people appeared in the video stream * *Input*. A list of tracked boxes with IDs * *Output*. Number of people in framedesired people * *Detailed functionality*. * *Initialization functions*. Inherited from *Base module* * *Iteration functions*. Inherited from *Base module* * *Termination functions*. Inherited from *Base module* * *Debug functions*. Inherited from *Base module* * Stream out the number of people on the camera stream * Stream out measured time * Stream out log messages **Crowd detection**. Inherited from *Base module* * *Purpose*. Detect crowds in a camera stream * *Actors*. * *Overall functionality*. * Detect crowds in a camera stream * *Input*. A list of tracked boxes with IDs * *Output*. A list of detected crowds * *Detailed functionality*. * *Initialization functions*. Inherited from *Base module* * *Iteration functions*. Inherited from *Base module* * *Termination functions*. Inherited from *Base module* * *Debug functions*. Inherited from *Base module* * Stream out the crowds information * Stream out measured time * Stream out log messages **Violence detection**. Inherited from *Base module* * *Explain*. PENDING **Abandoning position detection**. Inherited from *Base module* * *Purpose*. Detect abandoning positions in a camera stream * *Actors*. * *Overall functionality*. * Detect abandoning positions in a camera stream * *Input*. A list of tracked boxes with IDs * *Output*. A list of abandoning positions * *Detailed functionality*. * *Initialization functions*. Inherited from *Base module* * *Iteration functions*. Inherited from *Base module* * *Termination functions*. Inherited from *Base module* * *Debug functions*. Inherited from *Base module* * Stream out the abandoning positions * Stream out measured time * Stream out log messages **Human instrusion detection by virtual bonudary**. Inherited from *Base module* * *Idea*. * *Option 1*. Have a virtual boundary and any box crossing the boundary is an invalid box * *Pros*. Easy to implement * *Cons*. Not very accurate * *Option 2*. Map the virtual boundary and the detected boxes to ground plane and any box crossing the boundary is an invalid box * *Pros*. Hard to implement * *Cons*. More accurate than option 1 * *Purpose*. Detect human instrusion in a camera stream * *Actors*. * *Overall functionality*. * Detect human instrusion in a camera stream * *Input*. A list of tracked boxes with IDs * *Output*. A list of valid and invalid boxes, i.e. boxes which violate the virtual boundary * *Detailed functionality*. * *Initialization functions*. Inherited from *Base module* * *Iteration functions*. Inherited from *Base module* * *Termination functions*. Inherited from *Base module* * *Debug functions*. Inherited from *Base module* * Stream out tracked boxes with valid-or-invalid predictions * Stream out measured time * Stream out log messages **Human instrusion detection by restricted region**. Inherited from *Base module* * *Idea*. * *Option 1*. Have a virtual boundary and any box crossing the boundary is an invalid box * *Pros*. Easy to implement * *Cons*. Not very accurate * *Option 2*. Map the virtual boundary and the detected boxes to ground plane and any box crossing the boundary is an invalid box * *Pros*. Hard to implement * *Cons*. More accurate than option 1 * *Purpose*. Detect human instrusion in a camera stream * *Actors*. * *Overall functionality*. * Detect human instrusion in a camera stream * *Input*. A list of tracked boxes with IDs * *Output*. A list of valid and invalid boxes, i.e. boxes which violate the restricted region * *Detailed functionality*. * *Initialization functions*. Inherited from *Base module* * *Iteration functions*. Inherited from *Base module* * *Termination functions*. Inherited from *Base module* * *Debug functions*. Inherited from *Base module* * Stream out tracked boxes with valid-or-invalid predictions * Stream out measured time * Stream out log messages ### Priority 3 **Human tracing**. Inherited from *Base module* * *Note*. PENDING