State Estimation using Force and Vision
Back to the homepage
In robot assembly, accurate contact detection is critical for ensuring precise positioning, force-controlled assembly, and adaptability to variations in object alignment and surface properties. Unlike quadruped robots that switch between swing and stance phases, an assembly robot switches between free motion and constrained contact with its environment. To reliably detect contact, robots integrate contact force measurements and vision-based localization using sensor fusion and probabilistic inference.
A robotic manipulator or end-effector alternates between two key phases:
- Free motion phase: The tool or gripper moves freely without external constraints.
- Contact phase: The tool makes contact with an object or surface, requiring force control.
- Uncertain contact conditions: The force required for contact varies based on material properties.
- Sensor noise: Force-torque sensors and vision systems have inherent noise and drift.
- Slippage and misalignment: Contact may be unstable due to misalignment, surface roughness, or lubrication.
- External disturbances: Unexpected forces (e.g., vibrations, mispositioning) can influence detection.
To accurately detect contact, robots integrate multiple sensors:
(1) Force-Torque Sensors (FTS)
- Measures contact forces and torques at the end-effector.
- If force , the robot is likely in contact.
- Useful but may have drift or low sensitivity to small forces.
(2) Vision-Based Localization
- Stereo cameras, depth sensors, or LiDAR estimate object pose and contact location.
- Tracks the position and orientation of the tool relative to the assembly object.
- Helps anticipate contact regions and predict required force adjustments.
(3) Joint Encoders & Motor Current Feedback
- Measures joint angles and motor torques.
- Detects if the robot is exerting resistance against an obstacle (contact) or moving freely.
- Higher torque → likely contact phase.
(4) Acoustic or Tactile Sensors (Optional)
- Microphones or contact microphones detect impact sounds.
- Tactile arrays provide additional surface feedback.
Sensor Fusion Process:
By fusing these measurements, the system improves contact detection accuracy:
-
Extended Kalman Filter (EKF)
- Fuses force sensor + vision + encoders to estimate contact state.
-
Bayesian Inference
- Uses probability models to infer contact likelihood based on noisy data.
-
Machine Learning / Neural Networks
- Uses historical force and vision data to predict contact transitions.
Since sensor data is noisy and contact conditions vary, probabilistic models help infer whether the robot is in free motion or contact.
- The free/contact phase is treated as a hidden state .
- Observations come from fused sensor data.
- The transition probability depends on:
- Force readings.
- Vision-based position error.
- Joint torque.
HMM State Transition Matrix Example:
(2) Gaussian Mixture Models (GMM)
- Clusters force-vision data into contact and free-motion phases.
- Uses likelihood estimation to classify the current phase.
- State → Sensor measurements.
- Action → Adjust force control strategy.
- The model learns optimal contact force profiles for various assembly tasks.
To determine whether the robot is in contact, we compute:
Where:
- is the likelihood of observing the sensor data given contact.
- is the prior probability of contact (e.g., during insertion, contact probability is high).
- is the normalization factor.
Decision Rule:
- If , switch to contact mode.
- Else, remain in free motion.
5. Implementation in MATLAB
Here’s a MATLAB example using Extended Kalman Filter (EKF) for contact detection:
6. Summary
✅ Sensor Fusion: Combines force sensors, vision, and torque feedback for accurate contact detection.
✅ Probability Models: Uses HMM, Bayesian Inference, and GMM to infer contact likelihood.
✅ Adaptive Control: Machine Learning or Reinforcement Learning adjusts force during assembly.
✅ Practical Use: Implemented in ABB YuMi, KUKA iiwa, and Universal Robots for high-precision tasks.
With sensor fusion and probabilistic models, robotic assembly systems achieve stable, precise, and efficient contact detection, crucial for automation in manufacturing and assembly. 🚀🔧