Introduction to 3D Vision Guidance Solutions
About 3613 wordsAbout 12 min
1. Solution Overview
The 3D vision guidance solution uses a 3D camera to capture and interpret three-dimensional information about target objects and scene objects in the physical environment. Eye-hand calibration establishes the relative pose between the camera and the robot. The PickWiz vision system provides reliable real-time data transmission between them, enabling precise robotic picking. The solution supports industrial applications including sack and carton depalletizing, random picking, loading and unloading of neatly arranged target objects, and positioning and assembly.
Dexforce uses 2D recognition and detection to extract instances from a scene, then uses 3D positioning to extract their features and match them precisely. Recognition and picking functions can be configured for the application, providing a suitable vision guidance solution for each task scenario.
The currently supported 2D recognition and 3D matching approaches are listed below:
| Approach | Model | Strengths | Limitations | Examples |
|---|---|---|---|---|
| 2D recognition approaches: Segment instances from the scene | General model |
| The general model may detect unwanted target objects. Instance filtering is needed to retain only the target objects you want to recognize. | ![]() ![]() ![]() |
| CAD-based synthetic data training (One-Click Connection) |
|
| Data rendering:![]() ![]() ![]() ![]() Real-world inference: ![]() ![]() | |
| Point cloud segmentation |
|
| ![]() ![]() | |
| 3D matching approaches: Calculate instance poses in the camera coordinate system | Mask-based pick point generation |
| / | ![]() |
| Basic geometric shape fitting |
| / | ![]() ![]() ![]() ![]() ![]() ![]() | |
| Rigid transformation |
|
| ![]() ![]() ![]() ![]() ![]() ![]() ![]() Unsuitable scenario: identical front and back shapes ![]() ![]() | |
| 3D registration | Suitable for target objects of different types, shapes, or sizes, and applicable to almost all scenarios |
| ![]() ![]() |
Dexforce also provides optional functions for 3D vision guidance to meet the needs of different industrial scenarios:
Visual Classification: Classifies instances using 2D deep features. See the Visual Classification User Guide.
How it works: When PickWiz recognition is invoked, an operator selects and classifies images. The system extracts deep feature information from those images to build a category database. When PickWiz recognition is invoked for an input image, the system extracts its features, compares them with the database, and finds the category that best matches the instance image.
Applications: Classifying sacks or cartons, target object orientations, and target object fronts and backs.
Type Identification: Uses One-Click Connection to train a deep learning model that recognizes multiple material types. It is suitable when there are many types of similarly shaped target objects. See the Type Identification User Guide.
Collision Detection: Detects collisions between the end-effector and the material bin to prevent the robot from striking the bin during picking. See the Collision Detection User Guide.
How it works: The system extracts the material bin point cloud from the actual scene. Using the bin dimensions entered for the scene object, it calculates a template point cloud through quadrilateral fitting. It then performs fine matching between the fitted template point cloud and the bin point cloud from the actual scene to obtain the bin's final pose.
Allows the material bin to move over a relatively large range.
Does not support deformed material bins, whether concave or convex.
Front-Back Identification (using point cloud templates): Sets up two target object template point clouds, estimates the pose of each, and outputs the pose with the highest score. This is commonly used when incoming target objects may arrive either front side up or back side up. See the Front-Back Identification (Using Point Cloud Templates) User Guide.
Local Identification: If a local feature remains largely unchanged across different target objects or as the same target object deforms, the system can use that feature to locate the target object. See the Local Identification User Guide.
Instance Optimization: Uses a large model to refine masks after model recognition. This is commonly used when masks produced by a general model or a One-Click Connection visual model are incomplete or include too much of the scene. See the Instance Optimization User Guide.
Partition and Pallet Detection: Enables picking of the partition and pallet after target object picking is complete. See the Partition and Pallet Detection User Guide.
2. Selecting a Solution
Choose the appropriate task type in PickWiz based on the production scenario and target object type:

The 2D recognition and 3D matching approaches used by each task type are listed below:
| Scenario | Task type | Site requirements | 2D recognition approach | 3D matching approach | |
|---|---|---|---|---|---|
| Depalletizing | Individual sack depalletizing | A single type of sack, filled rather than empty | General sack model | Mask-based pick point generation | |
| Individual carton depalletizing | A single type of carton; avoid cartons packed tightly together and highly reflective tape | General individual-carton depalletizing model | Mask-based pick point generation | ||
| Loading and unloading of neatly arranged target objects | Loading and unloading of neatly arranged general target objects | Target objects with substantially different fronts and backs, arranged neatly; a single incoming material type, highly consistent target objects, and available target object CAD data | CAD-based synthetic data training (One-Click Connection) | Rigid transformation | |
| Loading and unloading of neatly arranged circular-surface target objects | Circular-surface target objects arranged neatly, with high-quality point clouds and clear edges | General circle model | Circle fitting | ||
| Loading and unloading of neatly arranged planar target objects | Planar target objects arranged neatly, with high consistency and available target object CAD data | CAD-based synthetic data training (One-Click Connection) | 3D registration | ||
| Loading and unloading of neatly arranged planar target objects (parallel) | Planar target objects arranged neatly, with high consistency and available target object CAD data | CAD-based synthetic data training (One-Click Connection) | 3D registration | ||
| Loading and unloading of neatly arranged planar target objects (image matching) | Planar target objects arranged neatly, with high consistency and available target object CAD data | Example-based instance segmentation | Feature matching | ||
| Loading and unloading of neatly arranged cylindrical target objects | Cylindrical target objects arranged neatly, with high-quality point clouds and clear edges | The ratio of cylinder height to radius does not exceed 4:1 | General short-cylinder model | Cylinder fitting | |
| The ratio of cylinder height to radius exceeds 4:1 | General long-cylinder model | Cylinder fitting | |||
| Loading and unloading of neatly arranged quadrilateral target objects | Planar quadrilateral target objects arranged neatly; a single incoming material type, flat surfaces, and clear edges | General quadrilateral model | Quadrilateral fitting | ||
| Random picking | Picking randomly arranged general target objects | Target objects with substantially different fronts and backs, arranged randomly, with high consistency and available target object CAD data | CAD-based synthetic data training (One-Click Connection) | Rigid transformation | |
| Picking randomly arranged planar target objects | Planar target objects arranged randomly, with high consistency and available target object CAD data | CAD-based synthetic data training (One-Click Connection) | 3D registration | ||
| Loading and unloading of planar target objects (objects separated) | Target objects separated from one another, with no contact or merging between their point clouds | Point cloud segmentation | 3D registration/ Mask-based pick point generation | ||
| Picking randomly arranged circular-surface target objects | Circular-surface target objects arranged randomly, with high-quality point clouds and clear edges | General circle model | Circle fitting | ||
| Picking randomly arranged cylindrical target objects | Cylindrical target objects arranged randomly, with high-quality point clouds and clear edges | The ratio of cylinder height to radius does not exceed 4:1 | General short-cylinder model | Cylinder fitting | |
| The ratio of cylinder height to radius exceeds 4:1 | General long-cylinder model | Cylinder fitting | |||
| Positioning and assembly | Positioning and assembly of general target objects | Only one target object in the camera's field of view; substantially different fronts and backs; available target object CAD data | CAD-based synthetic data training (One-Click Connection) | Rigid transformation | |
| Positioning and assembly of planar target objects | Only one planar target object in the camera's field of view, with substantial position variation and available target object CAD data | CAD-based synthetic data training (One-Click Connection) | 3D registration | ||
| Positioning and assembly of planar target objects (point cloud matching only) | Only one planar target object in the camera's field of view, with limited position variation | / | 3D registration | ||
| Positioning and assembly of planar target objects (image matching only) | Only one planar target object in the camera's field of view, with limited position variation | / | Feature matching | ||
*Note:
Top-corner picking approach:
- The model recognizes a carton and obtains its mask.
- Image/point cloud preprocessing adjusts the mask and removes noise.
- Quadrilateral fitting is applied to the mask to obtain the fitted quadrilateral's vertices, length, and width.
- The quadrilateral's four vertices are extracted and sorted by their image x and y coordinates to distinguish the upper-left, upper-right, lower-left, and lower-right vertices.
- If the Carton Body Vertex option is selected, the upper-left vertex along the carton's long side is set as the body's upper-left vertex, and the remaining vertices are reordered using the length and width information.
- Plane fitting is applied to the mask to obtain the plane equation.
- The image coordinates are projected onto the fitted plane using the intrinsic parameters, yielding the final 3D coordinates of the vertices to be picked.




























