3D Vision Guidance Solution Introduction
1. Solution Introduction
The 3D vision guidance solution captures and understands the three-dimensional information of Target Objects and Scene Objects in the real environment through a 3D camera, determines the relative pose relationship between the 3D camera and the Robot through eye-hand calibration, and relies on the Pickwiz vision system to establish real-time and reliable data transmission with the 3D camera and the Robot, enabling precise Robot picking and meeting the needs of various industrial scenarios such as sack/carton depalletizing, random picking, ordered loading and unloading, and positioning assembly.
Dexforce uses 2D recognition and detection methods to extract instances from the Scene, applies 3D positioning to extract feature information from instances and match them accurately, and configures functions related to recognition and picking according to actual needs, providing suitable vision guidance solutions customized for different operational scenarios.
The currently supported 2D recognition solutions and 3D matching solutions are shown in the table below:
| Solution | Model | Features | Drawbacks | Examples |
|---|---|---|---|---|
| 2D recognition solutions: Segment instances from the Scene | General-purpose Model |
| The General-purpose Model will detect unwanted Target Objects. To keep only the Target Objects you want to recognize, instance filtering is required. |    |
| Synthetic Data Training Based on CAD (One-click Connectivity) |
|
| Data Rendering:    Real-world Inference:   | |
| Point Cloud Segmentation |
|
|   | |
| [In Development] Instance Detection Based on CAD |
| Non-standard product capability, not open to users |   | |
| 3D matching solutions: Calculate the pose of instances in the Camera coordinate system | Mask-based Pick Point Generation |
|  | |
| Basic Geometric Shape Fitting |
|       | ||
| Rigid Transformation |
|
|        Unsuitable Scene: identical front and back shapes   | |
| 3D Registration | Suitable for Target Objects of different types, shapes, or sizes, and applicable to almost all scenarios |
|   | |
| [In Development] Keypoint Generation & Rigid Transformation | Suitable for Target Objects with clear edges in depth images or colored texture features in 2D images |
|   |
In addition, Dexforce fully considers the special requirements of different industrial scenarios and provides several optional features for the 3D vision guidance solution:
Vision Classification: uses 2D depth features to classify instances. For usage, refer to Vision Classification User Guide
Principle: when calling PickWiz recognition, images are manually selected and categorized to extract their depth feature information and build a category database. When calling PickWiz recognition, the features of the input image are extracted and compared with the features in the database to find the category that best matches the instance image.
Application scenarios: sack/carton classification, Target Object orientation classification, front/back classification of Target Objects, etc.
Recognition Type: uses one-click connectivity to train a Deep Learning model capable of recognizing multiple types of materials, suitable for scenarios with many types of Target Objects and similar shapes. For usage, refer to Recognition Type User Guide
Collision Detection: detects collisions between the Tool and the bin to prevent collisions between the Robot and the bin during the picking process. For usage, refer to Collision Detection User Guide
Principle: extracts the bin Point Cloud of the actual Scene, computes the template Point Cloud of the bin by using the input bin dimensions in the Scene Object together with quadrilateral fitting, and then performs precise matching between the fitted template Point Cloud and the actual bin Point Cloud of the Scene to obtain the final bin pose.
Large-range bin movement can be tolerated.
Deformed bins (concave inward or bulging outward) are not supported.
Front/Back Recognition (via Point Cloud Templates): sets two Target Object template Point Clouds, estimates the poses of the two templates, and finally outputs the pose with the highest score. This is often used in scenarios where the orientation of incoming Target Objects is inconsistent and the front side or the back side may face upward. For usage, refer to Front/Back Recognition (via Point Cloud Templates) User Guide.
Local Feature Recognition: if a certain local feature on the Target Object remains basically unchanged among different Target Objects and under different deformations of the same Target Object, that local feature can be used to locate the Target Object. For usage, refer to Local Feature Recognition User Guide
Instance Optimization: uses a large model to optimize the Mask after model recognition, commonly used in scenarios where the instance Mask is incomplete or excessive after recognition by the General-purpose Model/one-click connectivity vision model. For usage, refer to Instance Optimization User Guide
Separator and Bottom Pallet Detection: performs picking of separators and bottom pallets after Target Object picking is completed. For usage, refer to Separator and Bottom Pallet Detection User Guide
2. Solution Selection
Please select an appropriate task type in PickWiz according to the production scenario and the Target Object type:
The 2D recognition solutions and 3D matching solutions applied to each task type are shown in the table below:
| Scene | task type | On-site Requirements | 2D Recognition Solution | 3D Matching Solution | |
|---|---|---|---|---|---|
| Depalletizing | Single Sack Depalletizing | Single-type sacks with internal filling; avoid empty sacks | General-purpose Sack Model | Mask-based Pick Point Generation | |
| Single Carton Depalletizing* | Single-type cartons; avoid tight contact and highly reflective tape | General-purpose Single Carton Depalletizing Model | Mask-based Pick Point Generation | ||
| Mixed Carton Depalletizing | Multiple types of cartons; avoid tight contact and highly reflective tape | General-purpose Mixed Carton Depalletizing Model | Mask-based Pick Point Generation | ||
| Ordered Loading and Unloading | General-purpose Target Object Ordered Loading and Unloading | Target Objects with large front/back differences, arranged in order, single-type incoming materials, high consistency of Target Objects, and Target Object CAD available | Synthetic Data Training Based on CAD (One-click Connectivity) | Rigid Transformation | |
| Ordered Loading and Unloading Based on Circular Surfaces | Circular-surface Target Objects, arranged in order, with high Point Cloud quality and clear edges | General-purpose Circle Model | Circular Shape Fitting | ||
| Surface-type Target Object Ordered Loading and Unloading | Surface-type Target Objects, arranged in order, with high consistency of Target Objects and Target Object CAD available | Synthetic Data Training Based on CAD (One-click Connectivity) | 3D Registration | ||
| Surface-type Target Object Ordered Loading and Unloading (Parallelized) | Surface-type Target Objects, arranged in order, with high consistency of Target Objects and Target Object CAD available | Synthetic Data Training Based on CAD (One-click Connectivity) | 3D Registration | ||
| Ordered Loading and Unloading Based on Cylinders | Cylindrical Target Objects, arranged in order, with high Point Cloud quality and clear edges | The ratio of cylinder height to radius does not exceed 4:1 | General-purpose Short Cylinder Model | Cylindrical Shape Fitting | |
| The ratio of cylinder height to radius exceeds 4:1 | General-purpose Long Cylinder Model | Cylindrical Shape Fitting | |||
| Ordered Loading and Unloading Based on Quadrilaterals | Planar quadrilateral Target Objects, arranged in order, with single-type incoming materials, flat planes, and clear edges | General-purpose Quadrilateral Model | Quadrilateral Fitting | ||
| Random Picking | General-purpose Target Object Random Picking | Target Objects with large front/back differences, randomly placed, high consistency of Target Objects, and Target Object CAD available | Synthetic Data Training Based on CAD (One-click Connectivity) | Rigid Transformation | |
| Surface-type Target Object Random Picking | Surface-type Target Objects, randomly placed, with high consistency of Target Objects and Target Object CAD available | Synthetic Data Training Based on CAD (One-click Connectivity) | 3D Registration | ||
| Surface-type Target Object Loading and Unloading (materials isolated from each other) | Target Objects are separated from each other, and the Point Cloud of the Target Object has no adhesion at all | Point Cloud Segmentation | 3D Registration/ Mask-based Pick Point Generation | ||
| Random Picking Based on Circular Surfaces | Circular-surface Target Objects, randomly placed, with high Point Cloud quality and clear edges | General-purpose Circle Model | Circular Shape Fitting | ||
| Random Picking Based on Cylinders | Cylindrical Target Objects, randomly placed, with high Point Cloud quality and clear edges | The ratio of cylinder height to radius does not exceed 4:1 | General-purpose Short Cylinder Model | Cylindrical Shape Fitting | |
| The ratio of cylinder height to radius exceeds 4:1 | General-purpose Long Cylinder Model | Cylindrical Shape Fitting | |||
| Positioning Assembly | General-purpose Target Object Positioning Assembly | Only a single Target Object is within the Camera field of view, the front and back sides of the Target Object differ greatly, and Target Object CAD is available | Synthetic Data Training Based on CAD (One-click Connectivity) | Rigid Transformation | |
| Surface-type Target Object Positioning Assembly | Only a single surface-type Target Object is within the Camera field of view, the position varies greatly, and Target Object CAD is available | Synthetic Data Training Based on CAD (One-click Connectivity) | 3D Registration | ||
| Surface-type Target Object Positioning Assembly (Matching Only) | Only a single surface-type Target Object is within the Camera field of view, and the position varies slightly | / | 3D Registration | ||
*Notes:
Top-corner picking solution:
- The model recognizes the carton and obtains the Mask of the carton
- Image/Point Cloud preprocessing adjusts the Mask and removes noisy regions
- Perform quadrilateral fitting on the Mask to obtain the fitted quadrilateral information (vertices, length, and width information)
- Extract the four vertices of the quadrilateral and sort them by the x and y coordinates of the image to distinguish upper-left, upper-right, lower-left, and lower-right
- If Carton Body Vertex is checked, the upper-left vertex on the long side of the carton is set as the upper-left vertex of the body, and the vertices are re-sorted according to the length and width information
- Perform plane fitting on the Mask to obtain the plane equation
- Project the image coordinates onto the fitted plane based on Intrinsic Parameter to obtain the final 3D coordinates of the vertex to be picked