Common Recognition and Picking Issues

About 11116 wordsAbout 37 min

1. Stereo Camera

Stereo Camera Windows PickWiz Deployment (Application)

1.1 Image Acquisition Takes Too Long with Automatic Exposure

Issue: When using a stereo Camera, how can the image capture Takt Time be further optimized under auto exposure or multiplier-based HDR mode 【≥1.7.1】:

Solution: The upper limit of the auto exposure time can be used to control the maximum exposure time of a single image (it is recommended to set it greater than 200000μs, that is, 200ms. Setting it lower will not improve the frame rate either. The maximum frame rate is 5 frames, and it will instead damage image quality). 200000μs is already the fastest Takt Time. The algorithm will automatically compensate brightness according to the gain, so the overall effect remains unchanged, but the image capture time is reduced. Under HDR mode, this parameter can also be used to control the maximum exposure time, thereby reducing the overall HDR time.

1.2 Crash During Photo Capture

Issue: The software crashes when photographing with a stereo Camera

Possible cause: The stereo Camera is not using Gigabit Ethernet

Solution: The stereo Camera must use Gigabit Ethernet. First check whether the network cable supports Gigabit Ethernet. If not, replace it.

1.3 Image capture failed error

Issue: The stereo Camera reports an image capture failed error

Solution: Create a new stereo Camera configuration, and then capture the image again.

1.4 Image capture failed error

Issue: The stereo Camera reports an image capture failed error

Possible cause: Severe packet loss

Solution:

Use the MVS software to connect to the stereo Camera and start acquisition to check whether a pop-up indicates severe packet loss
If yes
1. Ensure that the Camera switch is dedicated to Camera use to keep the network stable, and set the Camera exposure to 5000 to facilitate subsequent bandwidth testing;
2. Open the bandwidth management tool and connect to the Camera you want to use;
3. Confirm the maximum network bandwidth: slide the Camera bandwidth to the maximum and confirm this maximum value, for example 964M. According to testing, part of the bandwidth (tentatively 64M) should be reserved as margin, so the total available bandwidth is 900M;
4. Allocate bandwidth according to the number of devices in use: for example, if 4 cameras need to be used at the same time, set the Camera bandwidth to 900/4=225M;
5. Click Start Analysis. If the bandwidth for (sent data volume) is not fully utilized or the Camera still experiences packet loss, continue locating the cause;

1.5 Abnormal Point Cloud

Issue: The Point Cloud imaging of the stereo Camera is collapsed / distorted

Possible cause: The 2D imaging is overexposed / too dark, the long side of the workpiece is not perpendicular to the image, and there is a visual blind spot

Solution:

First check whether the 2D imaging is overexposed / too dark. If yes, adjust the exposure first.

Status	2D Image	Point Cloud	Remarks	Handling Method
Before exposure adjustment (too dark)			Collapsed	Add a light source so that the surface of the Target Object is clearly visible
After exposure adjustment			Normal	/

Status	Point Cloud	Remarks	Handling Method
Before exposure adjustment (too dark)		Collapsed
After exposure adjustment		Normal	/

Check whether the long side of the workpiece is perpendicular to the image. This involves the stereo principle issue of "corresponding points". It is recommended to rotate the Camera installation angle by 90 degrees, as shown below.
If the object has a "visual blind spot" in the left and right images, the model will not be able to correctly predict the Point Cloud, causing distortion or collapse
How should this visual blind spot be understood? Is it that only one Camera can capture it?: The left Camera can see it while the right Camera cannot (and vice versa)
Why "can't it be solved by the model"?: Usually, the way to solve this issue is Shadow mode, but this area happens to be an occluded area, and such an area will not enter Shadow mode learning. Therefore it is ineffective.
Example Imaging
If the red-line image is normal but the Point Cloud imaging is abnormal, modifying either the zoom ratio or the minimum distance between the object and the Camera can produce a normal Point Cloud. In practice, the minimum distance between the object and the Camera should be modified first.

Example	Red-line Image	Point Cloud
Abnormal
Normal

1.6 Partial overexposure in the 2D image of the stereo Camera

Issue: The stereo Camera has a "split lighting face" effect, causing partial overexposure in the 2D image

Possible cause: HDR exposure mode is not enabled, and the lighting is abnormal

Solution:

For general simple large-object scenes, such as sacks and cartons

Enable HDR high dynamic range. You can see that the partially overexposed areas on the workpiece surface can show texture, and the Point Cloud improves.

For complex-shaped workpieces with split lighting, such as cylinders and specific shapes
Currently, stereo imaging cannot solve this, as in the following scenarios

1.7 Takt Time optimization for high-resolution stereo photographing

Issue: When using a high-resolution stereo Camera, the Takt Time for a single photograph is too long (a single photograph is nearly more than 8~10s)

Possible cause:

The zoom ratio is set to 1.0, making the image too large for a high-resolution Camera
High dynamic range is enabled, which is equivalent to triggering multiple image captures
The model Inference uses the default 16 iterations, which takes too long for high-resolution images

Solution:

Check whether the zoom ratio setting on the photographing interface is 0.5

Check whether high dynamic range is turned off

By modifying the "refine_iters" field in the C:\Users\dex\.dexforce\kuawei_data\deepmodel\iris\sack_carton\config.json file from 16 to 5, restart the software after the change.

Note that the Point Cloud effect may worsen at this time. You can use the stereo Shadow mode in the product to adapt to it (requires >=1.7.6).

1.8 Black lines appear in stereo Camera image capture

Issue: Black lines appear in stereo Camera image capture, as shown below.

Possible cause: The customer replaced the network cable with a longer one, causing unstable network speed. Network issues cause black lines in images captured by the KINGFISHER stereo Camera

Solution: Prioritize checking the on-site network conditions or the status of the network cable.

2. Mask mode & Shadow mode

2.1 The difference between Mask mode and Shadow mode

Issue: What is the difference between Mask mode and Shadow mode, and what scenarios are they suitable for?

Answer:

Mask mode: refers to data quickly constructed based on texture + synthetic data and directly used for training. Its detection capability is equivalent to the given texture (if multiple textures are provided, it can detect cartons or sacks with multiple textures) in the corresponding scene. It is generally used directly for cases where actual material detection fails (for example, the recent Yushi case, where the detection performance was too poor); versions after 132 all support it. The training duration on the IPC is expected to be within 1 hour.
Shadow mode: refers to real data accumulated after the model runs in the actual scene and then reused for training. The typical use scenario is when a detection success rate of over 95% has already been achieved and the goal is to reach 99.9%. Versions after 130 all support it. The training duration on the IPC is expected to be within 1 hour (with an initial data volume of more than 100 samples).

2.2 Save path of texture data in Mask mode

Issue: Where is the texture data saved after training with Mask mode?

Answer: In the corresponding Project configuration folder

2.3 Training new workpieces in Mask mode

Issue: After using Mask mode to train and obtain model A, a different carton is added on site. Can Mask mode continue to be used for training on the basis of A?

Answer: Yes. You only need to replace the model used during Mask mode training with the corresponding model and enter the new texture.

2.4 Mask mode training has no effect

Issue: After using Mask mode, there is no change in the effect. What is the reason?

Possible cause: The workpiece model has not been updated, and the selected texture is incomplete

Solution:

2.5 Mask mode training failed

Issue: Mask mode training failed

Possible cause:

Solution:

It may be caused by a code bug that prevented the folder from being created successfully
If not, create it and rerun Mask mode or Shadow mode
It may be caused by file corruption due to a direct power-off or direct restart
1. Check these 5 folders under this path. Each folder contains info.yaml. If you open info.yaml and find that the file is corrupted and cannot be opened normally, proceed to step 2
- Find the corresponding folder at the location shown below on the NAS (note that one folder needs to be decompressed). After decompression, replace the folder with the same name in the data projects shown above. (First delete the folder in data projects, then update it)

2.6 Shadow data save path

Issue: After enabling Shadow mode, where can the saved shadow data be viewed?

Answer: By default, it is under the kuawei\_data path. The save path can also be modified (it is recommended to use the default path)

2.7 Shadow mode training has no effect

Issue: After using Shadow mode, there is no change in the effect. What is the reason?

Possible cause: The model was not updated, or the workpiece type does not match

Solution:

2.8 Location to enable Shadow mode

Issue: Where is Shadow mode enabled in unordered and ordered scenes?

Answer: Shadow mode is enabled by default in unordered and ordered scenes. If you are not sure, you can check according to the following path

2.9 Modify the Shadow mode save node 【1.6.0-1.7.0】 for all scenes

Issue: How to modify the Shadow mode save node?

Answer: Enter the Project directory and find the task configuration according to the task ID

Enter the configuration and modify the contents of the ShadowMode section

3. Depalletizing scenarios

3.1 2D recognition

3.1.1 Quickly obtain the most suitable zoom ratio

Issue: How to quickly obtain the most suitable zoom ratio

Answer:

Use run.py locally for Inference (Windows)

3.1.2 Extract top-layer texture - some instances cannot be recognized

Issue: For cartons of the same size arranged in order, some cartons cannot be detected. Besides optimizing the model, are there any other methods?

Possible cause:

Solution: You can modify the parameters for extracting the top-layer texture so that only the top-layer cartons are retained

3.2 Pick Point processing

3.2.1 Pose adjustment

Issue: The carton coordinates output by the software by default have x pointing to the long side and y pointing to the short side. If y is expected to point to the long side, how should the parameters be set?

Answer: Configure according to two cases

The RZ angle is not fixed (accepts ±180 rotation)
1. Just add a fixed 90° rotation in the grasping strategy
The Rz angle is fixed
1. First add a fixed rotation angle to the grasp to make y point to the short side
- Then add the setting for the grasp angle range (set the angle range according to the actual situation)

3.3 Picking accuracy

3.3.1 The Pick Point is not at the center - resolution

Issue: The carton instance segmentation is normal and the Mask is complete, but the Pick Point is not at the center. How should it be adjusted?

Possible cause: This issue may occur because the grasp calculation - resolution parameter is relatively large, resulting in large spacing in the calculated Mask

Solution: You can modify the size of the grasp - resolution parameter. The tuning suggestion is as follows

Parameter tuning
- Use Meshlab to open the Point Cloud in the folder where the historical data is located. The file path of the historical data is C:\Users\dex\kuawei\_data\PickLight
  Meshlab is a powerful 3D display and manipulation tool. Download address: https://www.meshlab.net/#download
- After opening the corresponding Point Cloud in Meshlab, click

Measure the distance between adjacent points and fill in a nearby value

In actual scenarios, the Point Cloud precision of upper-layer objects is smaller, while that of lower-layer objects is larger, so it is recommended to fill in a slightly larger value

As shown below, the measured Point Cloud value is 0.0016, so you can fill in a nearby value such as 0.002 as the Point Cloud precision

3.3.2 Excessive grasp angle

Issue: The sack instance segmentation is normal and the Mask is complete, but the sack bulges, causing the grasp angle to be too large. How can the grasp angle be made smaller?

Solution:

The size of the patch can be adjusted
Enter the visualized Workflow, and add "box\_width" :100 and "box\_height" :100 (representing the patch size, which can be adjusted according to the actual sack) to the dof4 algorithm

3.3.3 Pick Point not centered - segmented into the side

Issue: During detection, segmentation is correct, but the side of the sack is also segmented, causing the grasp to deviate and not land at the center.

Possible cause: The side was also included in the calculation.

Solution: In the vision parameters, change Grasp Calculation - Select Coordinate System from 1 (roi coordinate system) -> 2 (the coordinate system perpendicular to the sack)

3.3.4 Pick Point not centered - large Pick Point deviation

Issue: The segmented Mask is correct and complete, but the Point Cloud output by the Pick Point generation node is incorrect?

Possible cause: This is generally caused by incorrect parameter settings in the optional functions. For example, if the average distance ratio parameter in the Point Cloud removal function is set too small, a large number of sack Point Cloud points will be removed, causing the Pick Point to shift.

Solution: Adjust the parameter settings.

3.3.5 Pick Point not centered - rough segmentation

Issue: The sack segmentation box is too large and the Mask sticks to the lower layer, causing the Pick Point to shift. How can this be resolved?

Solution: Because the Mask is too large, the main body Point Cloud is connected to the lower-layer Point Cloud. The lower-layer Point Cloud needs to be removed so that only the upper-layer Point Cloud is retained.

In the vision parameters, check the optional function to filter out Point Clouds whose object distance exceeds the limit (based on the roi axis direction) (v1.4.2)

3.3.6 Pick Point not centered

Issue: From the segmentation result image, both the segmentation and Mask are normal, but the Pick Point is not at the center. How can this be resolved?

Possible cause: In this example, too many noise points are attached.

Solution: Check the output of the grasp node in the corresponding historical data. You can enable Point Cloud noise removal.

3.3.7 Pick Point mode: 4-axis/6-axis (≤1.4.1)

Issue: When the robot arm is 4-axis, where is the corresponding coordinate output configured?

Answer: If the robot arm is 4-axis, set 4dof to true in the dof4 algorithm configuration of the grasp generation node; if it is 6-axis, set it to false (you need to enter the visual Workflow).

3.4 Optional functions

3.4.1 `fill_hole` Point Cloud noise appears after hole filling

Version dependency	Add location	Source data	Target data
PickLight >= 1.5.0	Post-processing of foreground node or pre-processing of instance generation node	depthImage	depthImage

Recommended for use with glia-0.3.3 and later.

Issue: In the visual Workflow, when using fille_hole to fill holes, pyramid-like stepped noise appears, for example:

Possible cause: Areas with depth 0 in the depth image affect hole filling. This was fixed in PickWiz versions >= 1.5.0.

Solution:

It is recommended to contact product/R&D support to update the on-site software version.
- Confirm or upgrade glia >= 0.3.3. Refer to Windows Development Environment Setup - PickLight. As shown below, find the corresponding download version for glia-0.3.3.
- Confirm or upgrade pickwiz >= 1.5.0 or upgrade the backend whl package to release-1.4.5.1 (commit version >=xxx, PR pending merge )
Restart the software after the update, then rerun the historical data.

3.4.2 Using the `filter_by_color` function

Issue: False positives are generated during instance generation, detecting the bottom pallet or other parts by mistake.

Solution: When false positives occur during instance generation and the bottom pallet or other areas are mistakenly detected as workpieces to be picked, first observe the differences between these detected instances and normal workpieces.

For scenes like the one above where the sack and pallet differ greatly, enter the visual Workflow and add the filter_by_color function to filter according to the specified upper and lower color bounds.

Add filter_by_color in the instance filter node.

Parameter description

Parameter	Setting value	Description
min_c	0.1	Minimum value for color filtering; 0 is black and 1 is white
max_c	2	Maximum value for color filtering; cannot be smaller than `min_c`
threshold	0.7	Filtering Threshold. If the ratio of the number of pixels within the filtering range (`min_c`, `max_c`) in the instance Mask to the Mask area is lower than this Threshold, the instance is filtered out
inverse	false	Whether to invert. If `true`, points in the Mask whose color is less than `min_c` or greater than `max_c` are used to calculate the Threshold

With the above parameter settings, pallet instances containing black can be filtered out, leaving only sack instances.

3.4.3 `is_top_safety` Whether there is other occlusion above the Pick Point

Issue: How can you determine whether the Target Object to be picked is safe and unobstructed in some scenarios?

Answer: You can add the is_top_safety function in the instance filter node.

Parameter description

Parameter	Default value	Description
x_length	1.0	Required parameter for cuboid generation, used to determine whether there is an occlusion within the length range of the Pick Point along the pose x direction
y_width	1.0	Required parameter for cuboid generation, used to determine whether there is an occlusion within the width range of the Pick Point along the pose y direction
z_depth	2.0	Required parameter for cuboid generation, used to determine whether there is an occlusion within the width range of the Pick Point along the pose z direction
bindwidth	0.1	Distance Threshold in the z direction, in m. Point Clouds whose height difference from the Pick Point in the z direction exceeds this distance are considered occlusion points
num_threshold	1000	If the number of occlusion points exceeds this Threshold, the Pick Point is considered occluded
debug	false	Whether to save Point Cloud data, used to observe whether the cuboid generation is reasonable
save_path	""	Data save path. Note that on Windows, paths are separated with \\

Usage log

3.4.4 `is_safety` Whether the gripper at the Pick Point collides

Issue: What is the purpose of the is_safety function?

Answer: It is used to filter out Pick Points that may result in collisions, and is suitable for cases where a robot arm grips an object and the gripper model is known.

Parameter description

Parameter	Default value	Description
prewin	100	Unit: pixels. Generates a Mask within a specified pixel range around the Pick Point for subsequent collision-range evaluation. If this value is too large, it may increase runtime and cause calculation anomalies. It is recommended to set it slightly larger than the pixel length of the long side of the workpiece Mask itself. You can evaluate it by checking the data in `debug_vis`
threshold	1.0	Threshold for determining whether the grasp pose is safe. If the number of Point Cloud points inside the gripper exceeds the Threshold, it is judged unsafe
roi_crop	true	Whether to crop the Point Cloud and retain only the Point Cloud within the `roi3d` range
debug_vis	""	Debug data save path. Saves information from the debugging process to the specified path: `mask.png`: Mask range generated according to the `prewin` parameter `object_mask.png`: instance object Mask `shell_mask.png`: result of subtracting the above two Masks, used to determine the surrounding range for collision checking `crop.ply`: surrounding Point Cloud used to determine collision
use_projected_mask	false	Whether to use model projection. If `false`, the `mask` passed in from the instance generation node is used. Enable this only when the workpiece itself to be picked has an accurate model

Usage log

4. Loading and unloading scenarios (ordered/unordered)

4.1 Deep Learning model

4.1.1 Incomplete model recognition

Issue: The model recognition is incomplete. The Inference count across multiple images is capped at a maximum value (generally 20).

Possible cause: When the model was exported, the maximum recognizable count was set to (20).

Solution: Contact the model provider and export the model again.

4.1.2 Abnormal keypoints in model detection

Issue: Abnormal keypoints appear in model detection.

Possible cause: This issue occurs because when we use one-click connectivity for model training, the keypoints are generated by our own downsampling. When we receive the trained model, new keypoints are generated.

Solution: The keypoints need to be replaced with the new ones.

4.1.3 Method for creating dual/multi-templates

Use the Camera to capture the Point Cloud files of the front and back sides of the workpiece, color them separately, align them with the model CAD, and then merge the colored front and back Point Cloud files together as the template Point Cloud file; perform point-by-point coloring on the keypoint Point Cloud file so that the two colors are evenly spaced.

Procedure:

Operation	Result
Connect the Camera in pickwiz, place the front- and back-side workpieces under the Camera, take photos and save the Point Cloud files, and remove noise points other than the workpiece Point Clouds to obtain Point Clouds containing only the front- and back-side workpieces	Point Cloud file after removal:
Color the front- and back-side workpieces separately
`Spitialransc` algorithm: align the colored front- and back-side Point Cloud files with the model CAD (rotate/move the colored Point Cloud files) For rotation/movement, refer to: Common Function Instructions for `meshlab`/TCP Debugging Assistant `cpfv` algorithm: (this step is the end; the remaining steps all apply to the `Spitialransc` algorithm)
Merge the colored front- and back-side Point Cloud files together Open the colored front- and back-side Point Cloud files in Meshlab (open only these two), right-click and select Flatten Visible Layers, check all 4 options, then click Apply; finally save the merged Point Cloud file
Perform point-by-point coloring on the 128 keypoint Point Cloud file so that the two colors are evenly spaced (make sure the two colors are the same two colors previously used for the front and back sides)	Like this:

4.1.4 Method for creating and using dual templates with `SpitialRansc` registration for flat workpieces with identical front and back shapes

Prepare three types of files
1. Keypoint ply file (keypoint_128.ply)
2. Front and back Point Cloud model files. These are generally cut out from the scene Point Cloud, with the front template colored green and the back template colored red. (point_front.ply, point_back.ply)
3. Mesh model (model.ply)

【For 3D parts, use layered dual-color keypoints, and the Point Cloud template only needs the Point Cloud visible to the Camera】

After aligning the two Point Cloud templates to the CAD, merge the Point Clouds to generate a dual-template Point Cloud ply file.【After aligning the Point Cloud files to the CAD, save each template separately to avoid having to realign them when adjusting templates later】

Import the three files into pickwiz
【Note】: Check whether the dual Point Cloud templates are already aligned and colored in two colors, and check whether the keypoints are locally scattered in two colors. The dual colors of the templates and keypoints must correspond.

Create Pick Points
【Note】: Create the Pick Points in order, and read the requirements for creating each Pick Point carefully.

Open the visualization interface and add the dual-template binding information

{
    "model_to_pick_binding": {
        "0": [
            1,
            2
        ],
        "1": [
            3,
            4
        ]
    }
}

Add the option to select "Pick Point index" in the Robot settings

Click Run. The returned Pick Point index has 4 possible values (1,2,3,4). If high accuracy is required, use a script to teach all 4 Pick Points.

4.1.5 Using Auto Enhancement: Improve prediction performance through data augmentation

Please see the detailed introduction in General Workpiece Vision Parameter Tuning Guide

4.1.6 Correctness of the CAD model used for data rendering

Current ways to obtain the CAD model:

The customer directly provides the original CAD
Scan the actual workpiece

Whichever method is used, please ensure the CAD model is correct (dimensions, orientation, type, etc.)

Counterexample 1: The scanned workpiece Point Cloud is inconsistent with the actual workpiece (opening angle is inconsistent)

4.1.7 TRT model export

When exporting the onnx model, add the use-trt option

python -m mixedai.scripts.export \
        --config-file config_path \
        --model-file model_path \
        --sample-image image_path \
        --use-trt

Use glia to export the trt model

The following operations are all in the pickwiz\_py39 environment, conda activate pickwiz\_py39

rcnn

python -m glia.dl.utils.trt_converter \
        --onnx ONNX_file_path, e.g. xxx.onnx \
        --image image_file_path, e.g. xxx.png \
        --model-type rcnn \ 
        --saveEngine xxx.trt \

yolo

python -m glia.dl.utils.trt_converter \
        --onnx ONNX_file_path, e.g. xxx.onnx \
        --model-type yolo \ 
        --saveEngine xxx.trt \

stereo

python -m glia.dl.utils.trt_converter \
        --onnx ONNX_file_path \
        --model-type stereo \ 
        --saveEngine xxx.trt \
        --optShapes=image1:1x3x480x640,image2:1x3x480x640 \
        --fp32 \
        --workspace=6000 \
        --tacticSources=-CUDNN,+CUBLAS \
        –-builderOptimizationLevel=3

For more detailed export instructions, see: http://doc.open3dv.site/glia/doc/master/text/application/trt_converter.html

Use glia to run Inference on the onnx model and trt model, and check whether the results are consistent

python -m glia.dl.utils.inference --image-file "./*.png" \
        --model-file .trt .engine or .onnx model \
        --output-dir ./ \
        --bbox-mode aabb (or obb)\
            --scale 1.0

4.1.8 `extract_roi3d_rgb` settings

Remove the corresponding image content outside the ROI3D area to remove the Background

Parameter: fill kernel size
Before 1.5.1: kernel_size

After 1.5.1:

*******Note*******If Point Cloud data is missing in the area to be detected, holes may appear in the image after Background removal, affecting detection performance. In this case, increase the fill kernel size to fill the holes. The specific effect is as follows

kernel_size	Original image	Detection result
5	The default configuration is 5. If Point Cloud data is missing in the area to be detected, there may be many holes in that area, affecting detection results
7
9
11
13
15
17
19	After increasing the fill kernel size, the black holes in the detection area are clearly reduced, and detection performance improves

4.1.9 Model not recognized

Issue: When picking the last remaining layer of workpieces, the Deep Learning model cannot recognize them

Affected version: 1.6.1

Possible cause: The glia version is glia 0.4.1, which is not compatible with the model

Solution: Download the fix package from the NAS system. After updating the glia version, the model can recognize the last layer

4.2 CPFV

Surface Workpiece Vision Parameter Tuning Guide

4.2.1 No CPFV detection result

Issue: Detection failed. CPFV has no detection result.

Solution:

Check whether there are noise points unrelated to the main body in the template Point Cloud. The viewing method in meshlab is shown below. If the bounding box is larger than the expected template Point Cloud, it indicates that undeleted noise exists
When prompted that no pose was detected, first focus on the bootstrap_percentage parameter. Set it to 1. Once matching is accurate, then consider reducing cycle time through parameters such as downsampling
Check whether the order of magnitude of the template Point Cloud matches the scene instance Point Cloud (instancepcd) before downsampling (it is recommended to use the undownsampled scene Point Cloud as the template Point Cloud; when the scene Point Cloud is downsampled, the input Point Cloud for pose estimation should also be downsampled accordingly)
Check whether the model_voxel parameter is consistent with the scene Point Cloud downsampling parameter in preprocessing
Try lowering both model_voxel and the scene Point Cloud downsampling parameter in preprocessing to increase the number of matchable points
When the downsampling of the template Point Cloud and scene Point Cloud is consistent, and both Point Cloud counts are still 200+, but detection still fails, the issue may be due to differences in Normal between the template Point Cloud and scene Point Cloud (mainly possible when using contour mode, and fixed in version 1.5.2). You can open meshlab and load the Point Clouds to check the consistency of Normals between the template and instance Point Clouds. If the Normal consistency is high but detection still fails, contact product/R&D support
- How to view Point Cloud Normals in meshlab
  - Load the Point Cloud
  - Select the Point Cloud whose Normals you want to view, then click render in the upper left
  - In the render dropdown list, click show normal
  - The color size can be set in the lower right

4.2.2 Poor model localization accuracy / poor repeat localization accuracy

Problem: Accuracy is poor. Model localization accuracy is poor / repeat localization accuracy is poor

Solution:

When matching is inaccurate, first focus on the bootstrap_percentage parameter. Set it to 1. Once matching is accurate, then consider reducing cycle time
Check whether a direction prior is set, and test whether disabling the direction prior is effective
In backend PickLight versions after 1.4.0, CPFV does not perform ICP fine registration by default. Check whether the icp_process function exists in the post-processing of the pose estimation node. For how to use the function, refer to Surface Workpiece Vision Parameter Tuning Guide
- Software version <= 1.4.2, note! When the icp_process post-processing function exists, to view the pose estimation result you need to enable the icp_process parameter "save_data": true and inspect it in the pose_refine folder. In other words, the Point Cloud registration result in the existing Pose folder ≠ the final pose estimation result, so you cannot determine that pose estimation is inaccurate based on the Point Cloud at that location
- Software version >= 1.4.3: ICP registration results are saved in the two Point Cloud files with the poserefine suffix in Pose/output
If the visualized accuracy result is still unsatisfactory after using the ICP function, you can try changing icp_mode="Plane" in the icp_process function
When the localized object is small, reduce icp_dist=0.xxx in the icp_process function according to the log prompt
When the same scene is photographed and matched multiple times but the results differ, check the consistency of the Point Clouds across the multiple captures (for example, whether multi-angle photography causes spatial changes in key feature regions of the Point Cloud)
Check and remove noise points in the Point Cloud template (parts that appear intermittently in the scene Point Cloud)

4.2.3 Rotation exists in model localization

Issue: Angle accuracy is poor / angle accuracy is required, and rotation exists in model localization

Solution:

When the workpiece is rotationally symmetric and consistent, rotation may cause RZ to rotate by 180 degrees. If the incoming workpiece orientation is always consistent, it is recommended to set the CPFV direction prior to [1,0,0,0] to maintain registration consistency. If the incoming workpiece orientation is not reliably consistent, then do not add a prior, and instead add and use the bound_euler_angle function. For the function description, refer to document 3.1 Grasping Strategy Tuning Guide【To Be Updated】
- "prior_direction": [[1,0,0,0]]
Add the refine_pose_by_z_axis_rotation function in the post-processing section after the Pick Point generation node to correct the rotated pose (after 1.4.2)
For the function Parameter usage, refer to Surface Workpiece Vision Parameter Tuning Guide

4.2.4 Registration cycle time does not match the expected cycle time

Issue: Poor cycle time. The registration cycle time does not match the expected cycle time

Note: CPFV cycle time is currently generally expected to be controlled within 2 seconds (1 second for detection, 1 second for fine registration)

Solution: On the premise that matching accuracy is acceptable, adjust the following parameters (it is recommended to test matching accuracy or check visualization results after each adjustment)

Check the detection time in the logs as shown below. The detection time represents CPFV runtime. Pose estimation runtime - detection time can be regarded as the post-processing ICP runtime
When CPFV runtime is high:
- Adjust the downsampling parameters so that the number of template Point Cloud points and scene Point Cloud points in the reference data is around 200, which gives better results
- bootstrap_percent5age 0-1, speed: 1 is the slowest, 0.1 is the fastest; stability: 1 is the best, 0.1 is the worst; it is recommended to reduce by 0.1 each time, and in general the minimum should be adjusted to 0.3-0.5. If it is too low, occasional missed pose detections are likely
- Check relative_allowed_diam Parameter = 0.04
When post-processing ICP runtime is high:
- Check whether the number of template Point Cloud points is higher than 200,000. Without affecting accuracy requirements, appropriately reduce the number of template Point Cloud points (downsample in meshlab and then update the workpiece Point Cloud information)
- Check whether the InstancePCD Point Cloud has been downsampled. It is recommended to use meshlab to inspect the template Point Cloud spacing and set the corresponding downsampling parameters

4.2.5 Flat workpiece matching produces offset sliding

Issue: Matching offset. Flat workpiece matching produces offset sliding

Solution: First confirm whether the dimensions of the target Point Cloud will change in each direction. If they do, you need to confirm the unchanged part of the main body

Use the template shown in the figure, and fill in the coarse registration template in the workpiece properties

Add the icp_process Parameter mode_path="fine_registration_template_file_path"

For Windows paths, double backslashes or forward slashes are recommended

{
    "icp_dist": 0.01,
    "icp_mode": "Plane",
    "save_data": true,
    "model_path": "C:\\Users\\dex\\Documents\\refine_model.ply"
}

Scene Point Cloud (the dimensions inside the red box will change)	Coarse registration template (the right side is recommended to be larger than the scene)	Fine registration template (take the unchanged region)

Add extraction of boundary points

Add the function get_boundary_points in cpfv preprocessing

{
    "angle_threshold": 90,
    "max_nn": 30,
    "radius": 0.005
}

Warning! When using get_boundary_points, pay attention to changing the downsampling input data type

Add the boundary_type type to the cpfv parameters

{
    "angle_num": 40,
    "bootstrap_percentage": 1.0,
    "boundary_type": 1,
    "dist_num": 100,
    "faster_nms": true,
    "icp_dist": 0.01,
    "icp_method": "Plane",
    "model_voxel": 0.005,
    "nms_thresh": 0.5,
    "num_results": 1,
    "prior": {
        "angle_thresh": 30,
        "distance_thresh": 0.3,
        "is_flat": false,
        "pair_criterion": 1,
        "pair_mode": 0,
        "prior_direction": []
    },
    "process_in_roi": true,
    "refine": true,
    "relative_allowed_diam": 0.04,
    "scene_ratio": 1,
    "score_thresh": 0.5,
    "update_prior": false
}

When flat parts on site still show smooth offset after using boundary_type, you can use the contour refine feature (currently exists in branch cwy/add-boundary-refine, and will later be merged into 1.4.x). The usage is to add the parameters boundary_icp, boundary_thresh

Parameter description:

When boundary_icp is true, contour refine is enabled. The default is false

boundary_thresh is the radius parameter used to calculate the model contour. The default value is 0.005, the same as the radius parameter in preprocessing get_boundary_points. (The contour radius parameter is recommended to be half of the downsampling coefficient, and should be adjusted according to the contour Point Cloud status of the template and scene in Builder/pose/input)

{
    "angle_num": 40,
    "bootstrap_percentage": 1.0,
    "boundary_type": 1,
    "boundary_icp": true,
    "boundary_thresh": 0.005,
    "dist_num": 100,
    "faster_nms": true,
    "icp_dist": 0.01,
    "icp_method": "Plane",
    "model_voxel": 0.005,
    "nms_thresh": 0.5,
    "num_results": 1,
    "prior": {
        "angle_thresh": 30,
        "distance_thresh": 0.3,
        "is_flat": false,
        "pair_criterion": 1,
        "pair_mode": 0,
        "prior_direction": []
    },
    "process_in_roi": true,
    "refine": true,
    "relative_allowed_diam": 0.04,
    "scene_ratio": 1,
    "score_thresh": 0.5,
    "update_prior": false
}

4.2.6 Experience: debugging single-target precise positioning

Project name: Suzhou Samsung - Washing machine inner tub bottom screw hole positioning

Workflow characteristics

The Instance Segmentation node has no algorithm and only uses the ROI to frame part of the target workpiece area
It relies only on CPFV+ICP to register and precisely position the target

Project requirements

Positioning accuracy <=1 mm
Total runtime requirement <=2 s

Theoretical debugging process

First satisfy positioning accuracy, then consider reducing runtime. Refer to 【Detection failed】 and 【Poor accuracy】
1. It is recommended to first debug the Camera until the Point Cloud quality is stable, and then collect a scene Point Cloud dataset of different angle positions within a fixed ROI
2. Uniformly compare the scene dataset, confirm the feature region that appears 100% of the time, and select the relatively "3D" part in space as the Point Cloud template. If it is a flat workpiece, it is recommended to use contour registration
3. According to the log feedback in actual use, adjust the scene Point Cloud downsampling and CPFV+ICP parameters until the matching effect is stable
4. Test the repeat positioning accuracy. If it meets the conditions, end the positioning debugging; otherwise continue adjusting parameters
Accelerate CPFV+ICP on the basis that positioning accuracy meets the conditions
1. Refer to 【Poor cycle time】
2. If the pose estimation node is adjusted to about 1s but the overall runtime still exceeds expectations, refer to Algorithm process runtime analysis to analyze the cycle time, and then submit a request and consult product/R&D for other possible acceleration operations

Specific debugging issues and experience

Issue 1: Matching offset + high runtime

The on-site feedback on the pose estimation matching result is shown below. Green is the template matching result, and white is the scene Point Cloud

First, check the overlap between the template Point Cloud and the scene Point Cloud.
1. It can be seen that the main body of the pipe at the very top of the figure is largely missing in the scene Point Cloud.
2. The larger main body is the plane in the middle of the figure, and there are similar planar Point Clouds on both sides of that planar section.
3. Because the relative relationship between the ROI and the target is not fixed, the planar portion of the template on one side may extend beyond the scene Point Cloud.
Determine the cause of the matching offset.
1. Due to ICP's point-pair matching characteristics, a greater number of similar points can cause the head of the template T-shape to "sway" left and right at the scene Point Cloud position, while the pipe main body at the tail of the T-shape cannot be effectively "anchored" by the scene Point Cloud, resulting in possible deviation.
2. In addition, the icp_dis Threshold is set to 0.01 by default, causing the template Point Cloud to treat points within about 10mm as "its own," which further increases the randomness of matching.

Solution 1:

Set icp_dis=0.002 to reduce the randomness of movement of the template Point Cloud.
Adjust plane removal for the Point Cloud template, and add remove_or_preserve_plane in the post-processing after the instance filtering node to reduce the abnormal impact of missing planar regions on matching. The Parameter settings are shown in the figure. When actually creating it, there will be one extra return_params Parameter; just delete it.
1. Set dist_thres according to the fluctuation thickness of the planar Point Cloud.
2. When inverse is set to true, the plane is removed; otherwise, the plane is preserved.
Create a new template based on the Point Cloud with the plane removed.
Adjust CPFV and preprocessing downsampling parameters to improve speed. You can directly refer to 【Poor cycle time】.

Issue 2: Low repeat positioning accuracy in matching

The pose estimation result is severely misaligned.

Check the coarse registration Point Cloud condition.
1. The CPFV coarse registration result of the new template may be poor, causing ICP to fail to produce the correct result.
2. It was found that there is a severe difference in point spacing between the template Point Cloud and the scene Point Cloud. Inspection showed a large difference between the parameters model_voxel=0.01 and voxel_size=0.02.

Solution 2:

Adjust the preprocessing downsampling Parameter to 0.015 so that it is close to CPFV's "model_voxel"=0.01.

After correction, tests on 13 sets of historical data matched normally. At the site, the Camera was used to take 10 shots, and the repeatability met the requirements. The algorithm runtime was about 1s.

Notes

The main body in the new template has been changed from a plane to the front end of the black pipe, so it is still necessary to ensure that this part is always within the ROI regardless of the incoming part angle; otherwise, it may still cause matching deviation!

4.3 PNP

4.3.1 Inaccurate matching / reversed matching

Issue: Inaccurate matching / reversed matching occurs during the matching process.

Solution:

Check the template Point Cloud and delete interfering Point Clouds.
Check whether the template Point Cloud and the scene Point Cloud are exactly reversed by 180°. If so, flip the template Point Cloud by 180°.

Key point of the issue: The bottom Point Cloud accounts for a large proportion, while the top Point Cloud accounts for a small proportion, and priority registration is required for the top Point Cloud with the smaller proportion.
Analysis: The Point Cloud template has many invalid points, and the object pose correction range is set too large.
Method:
1. The CAD dimensions are basically the same as those of the real workpiece. Remove the invisible regions from the CAD, then downsample it to improve the registration score.
- Use the scene Point Cloud as the template, and the template must overlap with the CAD. If the effect is still poor, reduce the number of Point Clouds at the bottom of the template and increase the proportion of Point Clouds at the top to improve the top registration result.
- The object pose correction range is large. You can check the logs for a recommended value and modify the object pose correction range accordingly.

4.3.2 Matching offset

Issue: In random picking based on CAD, pose matching is offset when using the SpatialRansac algorithm.

Solution:

Use surface workpieces + the cpfv algorithm, and obtain the model Point Cloud file by photographing with the Camera and extracting it.

If inaccurate matching still exists after using cpfv, refer to the following solutions:

Adjust Camera parameters to obtain high-quality Point Cloud images.
Process the Point Cloud image after Instance Segmentation and use it as the Point Cloud template.
Adjust the number of points in the template Point Cloud and the scene Point Cloud, and control the number of points between 200\~400.
Adjust the bootstrap_percentage Parameter to 1 to remove the pose angle prior in pose estimation.

4.4 Cylinder fitting PMFE

4.4.1 Pick Point angle not vertical

Issue: How can the Z-axis Pick Point angle be kept in a state that appears more vertical to the human eye?

Solution: Reduce the rotation interval in the cylinder workpiece properties so that there is one Pick Point every small angle around the cylinder surface. This makes the Pick Points denser, meaning the angle difference between the most vertical Pick Point and the ROI Z-axis or Camera Z-axis will be smaller, as shown below:

Perform front/back classification on a single instance. First, add the processor deep_feature_classify in the post-processing after Pick Point generation, with the parameters shown below.

Create a folder to store the front/back data, and under the created folder create two folders named "0" and "1", as shown below.

Then run the Workflow once. After deep_featrue_classify is called, it will automatically crop the front/back images and store them in the historical data.
View the latest historical data in picklight.

Copy x_search.png, then under the specified path create a folder named with the size of the part you want to add. The most direct way is to copy an existing folder and then clear its contents. Inside are the 0 and 1 folders. After clearing them, put in the images classified by orientation (Tip: in this step, the operator must distinguish the front and back by themselves).

Go to this path and execute these two commands. Remember to change the path to the part you want to add. In this way, the front/back templates will be created. (How to find the path: open the folder on the desktop named New Folder 2, find the folder named site-packages on the left and click to enter it, then create the folder according to the path shown in the figure.)

If you want to add a new part, just repeat this step starting from step 2.

4.6 Wire harness Workflow

4.6.1 Camera configuration

Adjust the 2D exposure. The exposure needs to be high enough so that the wire harness does not appear "completely black."

4.6.2 Workpiece template

Use the extracted Point Cloud as the template, and align it with keypoints, mesh, etc. This can be done using meshlab.
O-shaped wire harnesses generally have dimensional differences. You can scale both the Cad and the feature points at the same time; refer to Common Meshlab Operations, form a model with dimensions equivalent to the actual wire harness, and register it into the software.

4.6.3 Bin detection

Extract the bin opening, and refine the extraction of the bin opening.
Keep the Point Cloud storage path well managed; bin templates for different workstations are independent of each other.
In the backend logs, pay attention to backend.log and search for "container score" to check whether the matching score is normal (generally it should be above 0.3-0.4).
Use Points for the matching mode.
If the material has not passed the bin opening, note that additional handling is required.
In the output of the container node, the item with the icp suffix indicates the final bin opening matching result; init indicates the initial bin opening pose.

For detailed usage documentation, see Circle3D, IcpEstimator and CPFV Parameter Description

4.6.4 Pose estimation

In the wire harness scenario, use Points for the matching mode.
After changing the Camera exposure, be sure to change the Processor as well, because there may be color filtering that causes abnormal Point Cloud filtering.
The runtime branch for the wire harness is the specific branch cwy/collection_add_collision_score(only for version 1.2.x; version 1.4.x does not require this). To prevent an empty bin report during mid-process picking, fitness needs to be set to 0.

4.6.5 Collision detection

Special handling is required when configuring the gripper; otherwise, it is easy to encounter the "gripping the bin wall" phenomenon.
The bin height setting must be reasonable; otherwise, "false collision detection" can easily occur, causing the materials at the bottom of the bin to become ungraspable.
When the gripper scrapes the edge of the bin, it is recommended to determine whether the root cause is a positioning issue or a gripper size issue. The gripper can be expanded. For the method, see Common Meshlab Operations.

4.6.6 No Pick Point

4.6.7 Pick Point settings

For the x and y axes, the gripper coordinate system needs to be set at the center of the wire harness Point Cloud.
The z axis needs to be set on the surface of the Point Cloud; it must not be too high or too low.
The rz angle needs to be adjusted correctly to match the actual grasping pose.
The rx and ry poses need to be set perpendicular to the Point Cloud.
In the case of an eccentric gripper, collision detection filters out more candidates, so symmetric Pick Points need to be set to avoid collision detection.

4.7 Pick sorting

4.7.1 Sorting abnormality: not picking in the expected order

Solution:

Check whether the instance filtering function has taken effect and reduced the number of instances participating in pick sorting. All instances need to be sorted to obtain the correct pick order.

4.7.2 Grasping strategy: picking the lower layer

Solution: Troubleshoot step by step.

Based on the visualization of the instance generation node, determine whether the upper-layer instances have been segmented correctly.
Confirm at which node the instance Point Cloud is generated; the height-sorted grasping strategy will be executed after that node.
If the grasping strategy is executed earlier than instance filtering, the existing instances may be correctly sorted, but the instance filtering node filters out some upper-layer instances (for example, they are filtered out due to low Confidence). In that case, those upper-layer instances will not be provided later, and the visible effect is that the lower layer gets picked.
Solution: remove some unnecessary filtering functions; train a new model in Shadow mode.

How to Use Shadow Mode

Common Recognition and Picking Issues