Guide to Adjusting Vision Parameters for Curved-Surface Target Objects
This article mainly introduces how to adjust vision parameters according to actual scenarios in ordered loading and unloading and random picking based on curved surfaces.
1. 2D Recognition
1.1 Preprocessing
The preprocessing for 2D Recognition processes 2D images before the Deep Learning model performs computation.
1.1.1 Bilateral Filtering
- Function
Image smoothing based on bilateral filtering
- Parameter Description
| Parameter | Description | Default Value | Value Range |
|---|---|---|---|
| Maximum Depth Difference | The maximum depth difference for bilateral filtering | 0.03 | [0.01, 1] |
| Filter Kernel Size | The convolution kernel size for bilateral filtering | 7 | [1, 3000] |
1.1.2 Depth to Normal Map
- Function
Calculate pixel normals from the depth map and convert the image into a normal map
1.1.3 Image Enhancement
- Function
Common image enhancement, such as color saturation, contrast, brightness, and sharpness
- Parameter Description
| Parameter | Description | Default Value | Value Range |
|---|---|---|---|
| Image Enhancement Type | Enhance a certain element of the image | Contrast | Color saturation, contrast, brightness, sharpness |
| Image Enhancement Threshold | How much to enhance a certain element of the image | 1.5 | [0.1, 100] |
1.1.4 Histogram Equalization
- Function
Improve the contrast of the image
- Parameter Description
| Parameter | Description | Default Value | Value Range |
|---|---|---|---|
| Local Mode | Local or global histogram equalization. If selected, local histogram equalization is used; if not selected, global histogram equalization is used | Selected | / |
| Contrast Threshold | Contrast Threshold | 3 | [1,1000] |
1.1.5 Filter Depth Map by Color
- Function
Filter the depth map based on color values
- Parameter Description
| Parameter | Description | Default Value | Value Range |
|---|---|---|---|
| Fill Kernel Size | The size for color filling | 3 | [1,99] |
| Filter Depth by hsv - Maximum Color Range Value | Maximum color value | [180,255,255] | [[0,0,0],[255,255,255]] |
| Filter Depth by hsv - Minimum Color Range Value | Minimum color value | [0,0,0] | [[0,0,0],[255,255,255]] |
| Keep Areas Within the Color Range | If selected, keep the areas within the color range; if not selected, keep the areas outside the color range | / | / |
1.1.6 Gamma Image Correction
- Function
Gamma correction changes the image brightness
- Parameter Description
| Parameter | Description | Default Value | Value Range |
|---|---|---|---|
| Gamma Compensation Coefficient | When this value is less than 1, the image becomes darker; when this value is greater than 1, the image becomes brighter | 1 | [0.1,100] |
| Gamma Correction Coefficient | When this value is less than 1, the image becomes darker and is suitable for images with excessive brightness; when this value is greater than 1, the image becomes brighter and is suitable for images with insufficient brightness | 2.2 | [0.1,100] |
1.1.7 Fill Depth Map Holes
- Function
Fill hollow areas in the depth map and smooth the filled depth map
- Use Case
Due to issues such as occlusion caused by the structure of the target object itself and uneven lighting, parts of the target object may be missing from the depth map
- Parameter Description
| Parameter | Description | Default Value | Value Range |
|---|---|---|---|
| Fill Kernel Size | The size for hole filling | 3 | [1,99] |
Only odd numbers can be entered for Fill Kernel Size
- Parameter Tuning
Adjust according to the detection results. If the filling is excessive, reduce the parameter; if the filling is insufficient, increase the parameter
- Example
1.1.8 Edge Enhancement
- Function
Set the edge portions of the texture in the image to the Background color or a color that differs greatly from the Background color, so as to highlight the edge information of the target object
- Use Case
Mutual occlusion or overlap of target objects causes unclear edges
- Parameter Description
| Parameter | Description | Default Value | Parameter Range | Tuning Recommendation |
|---|---|---|---|---|
| Normal Z-direction Filtering Threshold | The angle filtering threshold between the normal corresponding to each point in the depth map and the positive Z-axis direction of the camera coordinate system. If the angle between the normal of a point and the positive Z-axis direction of the camera coordinate system is greater than this threshold, the color at the corresponding position of that point in the 2D image will be set to the Background color or a color that differs greatly from the Background color | 30 | [0,180] | For flat target object surfaces, this threshold can be smaller. For curved-surface target objects, increase it appropriately according to the degree of surface inclination |
| Background Color | The RGB color threshold of the Background color | 128 | [0,255] | |
| Automatically Adjust Contrast Background | Selected After Automatically Adjust Contrast Background is enabled, the color of points in the 2D image whose angle is greater than the filtering threshold is set to a color that differs greatly from the Background color; not selected When Automatically Adjust Contrast Background is disabled, the color of points in the 2D image whose angle is greater than the filtering threshold is set to the color corresponding to the Background color | Not selected | / |
- Example
1.1.9 Extract Top-Layer Texture
- Function
Extract the texture of the topmost or bottommost target object, while setting other regions to the Background color or a color that differs greatly from the Background color.
- Use Case
Factors such as poor lighting conditions, similar color textures, dense stacking, interleaved stacking, or occlusion may make it difficult for the model to distinguish texture differences between upper and lower target objects, which can easily lead to false detections.
- Parameter Description
| Parameter | Description | Default Value | Parameter Range | Unit | Tuning Recommendation |
|---|---|---|---|---|---|
| Distance Threshold (mm) | If the distance from a point to the topmost plane (bottommost plane) is lower than this threshold, it is regarded as a point within the topmost plane (bottommost plane) and should be retained. Otherwise, it is regarded as a point on the lower layer (upper layer), and the color of points on the lower layer (upper layer) is set to the Background color or a color that differs greatly from the Background color | 50 | [0.1, 1000] | mm | Generally adjust it to 1/2 of the target object height |
| Cluster Point Cloud Count | The expected number of points participating in clustering, that is, the number of sampled Point Cloud points in the ROI 3D area | 10000 | [1,10000000] | / | The more Cluster Point Cloud Count, the lower the model Inference speed but the higher the accuracy; the fewer Cluster Point Cloud Count, the higher the model Inference speed but the lower the accuracy |
| Minimum Number of Category Points | The minimum number of points used to filter categories | 1000 | [1, 10000000] | / | / |
| Automatically Calculate Contrast Background | Selected After Automatically Calculate Contrast Background is enabled, other regions outside the topmost (bottommost) layer in the 2D image are set to a color that differs greatly from the Background color threshold; not selected When Automatically Calculate Contrast Background is disabled, other regions outside the topmost (bottommost) layer in the 2D image are set to the color corresponding to the Background color threshold | Selected | / | / | / |
| Background Color Threshold | Background color RGB threshold | 128 | [0,255] | / | / |
- Example
1.1.10 Remove Image Background Outside roi3d
- Function
Remove the background in the 2D image outside the ROI3D area
- Use Case
Excessive image background noise affects the detection results
- Parameter Description
| Parameter Name | Description | Default Value | Value Range |
|---|---|---|---|
| Fill Kernel Size | The size for hole filling | 5 | [1,99] |
| Iterations | The number of iterations for image Dilation | 1 | [1,99] |
| Automatically Calculate Contrast Background | Selected After Automatically Calculate Contrast Background is enabled, areas outside roi in the 2D image are set to a color that differs greatly from the Background color threshold; not selected When Automatically Calculate Contrast Background is disabled, areas outside roi in the 2D image are set to the color corresponding to the Background color threshold | Selected | / |
| Background Color Threshold | The RGB color threshold of the Background color | 128 | [0,255] |
Only odd numbers can be entered for Fill Kernel Size
- Parameter Tuning
If more background noise needs to be removed from the image, the Fill Kernel Size should be reduced
- Example
1.2 Instance Segmentation
1.2.1 Scaling Ratio
- Function
Scale the original image proportionally before Inference to improve the accuracy and recall of 2D Recognition.
- Use Case
If the detection result is unsatisfactory (for example, no instance is detected, missed recognition occurs, a bounding box covers multiple instances, or a bounding box does not fully cover an instance), this function should be adjusted.
Parameter Description
Default Value: 1.0
Value Range: [0.01, 3.00]
Step Size: 0.01
Parameter Tuning
- Run with the default value and check the detection result in the visualization window. If no instance is detected, missed recognition occurs, a bounding box covers multiple instances, or a bounding box does not fully cover an instance, this function should be adjusted.
If a good detection result still cannot be obtained after trying all Scaling Ratios, the ROI area can be adjusted
As shown in the figure below, when the Scaling Ratio is 0.7, the detection result improves significantly, so 0.7 can be determined as the lower limit of the Scaling Ratio range
When the Scaling Ratio is 2, the detection result decreases significantly, so 2 can be determined as the upper limit of the Scaling Ratio range
1.2.2 Lower Confidence Threshold
- Function
Retain only recognition results whose scores from the Deep Learning model are higher than the Lower Confidence Threshold
- Use Case
If the instances boxed in the detection results do not match expectations, this function can be adjusted
- Parameter Description
Default Value: 0.5
Value Range: [0.01, 1.00]
Parameter Tuning
If the model detects too few instances, this threshold should be reduced; if the value is too small, it may affect the accuracy of image recognition.
If an excessively small Lower Confidence Threshold causes incorrect instances to be detected and those incorrect instances need to be removed, this threshold should be increased; if the value is too large, it may result in zero retained detection results and no output.
1.2.3 Enable Auto Enhancement
- Function
Combine all input values of Scaling Ratios and rotation angles for Inference, and return all results from the combinations that are greater than the set Lower Confidence Threshold, which can improve model Inference accuracy, but will increase the processing time.
- Use Case
A single Scaling Ratio cannot meet the requirements of the actual scenario, resulting in incomplete detection, or the object placement inclination is relatively large.
- Example
If Auto Enhancement - Scaling Ratio is set to [0.8, 0.9, 1.0], and Auto Enhancement - Rotation Angle is set to [0, 90.0] , then the values in the Scaling Ratios and rotation angles will be combined in pairs. The model will automatically generate 6 images internally for Inference, and finally merge these 6 Inference results together, outputting the results that are greater than the Lower Confidence Threshold.
Auto Enhancement - Scaling Ratio
- Function
Scale the original image multiple times and perform Inference multiple times to output comprehensive Inference results
- Use Case
A single Scaling Ratio cannot meet the requirements of the actual scenario, resulting in incomplete detection
- Parameter Description
Default Value: [1.0]
Value Range: The range of each Scaling Ratio is [0.1, 3.0]
Multiple Scaling Ratios can be set, separated by English commas
- Parameter Tuning
Enter multiple 1.2.1 Scaling Ratio Scaling Ratios that achieve good detection results
Auto Enhancement - Rotation Angle
- Function
Rotate the original image multiple times and perform Inference multiple times to output comprehensive Inference results
- Use Case
Use it when the object placement deviates greatly from the coordinate axes
- Parameter Description
Default Value: [0.0]
Value Range: The value range of each rotation angle is [0, 360]
Multiple rotation angles can be set, separated by English commas
- Parameter Tuning
Adjust Auto Enhancement - Rotation Angle according to the object angle in the actual scenario. The inclination angle can be judged based on sack patterns and bag opening shapes, as well as carton edges and brand logos
1.3 Point Cloud Generation
| Instance Point Cloud Generation Mode | Mask Mode (After Segmentation) | - | Use the segmented instance Mask to generate the Point Cloud |
| Bounding Box Mode (After Segmentation) | Bounding Box Scaling Ratio (After Segmentation) | Use the segmented instance bounding box to generate the Point Cloud | |
| Whether Color Is Needed for Point Cloud Generation (After Segmentation) | Whether the generated instance Point Cloud needs color attached | ||
| Mask Mode (After Filtering) | - | Use the filtered instance Mask to generate the Point Cloud | |
| Bounding Box Mode (After Filtering) | Bounding Box Scaling Ratio (After Filtering) | Use the filtered instance bounding box to generate the Point Cloud | |
| Whether Color Is Needed for Point Cloud Generation (After Filtering) | Whether the generated instance Point Cloud needs color attached |
If acceleration is not needed, there is no need to use the Instance Filtering function. Use Mask Mode (After Segmentation) or Bounding Box Mode (After Segmentation) to generate the instance Point Cloud, which can be viewed in the generated instance Point Cloud folder under the project storage folder \Project Name\data\PickLight\Historical Data Timestamp\Builder\pose\input folder;
If acceleration is needed, the Instance Filtering function can be used to filter instances. Use Mask Mode (After Filtering) or Bounding Box Mode (After Filtering) to generate the instance Point Cloud, which can be viewed in the generated instance Point Cloud folder under the project storage folder \Project Name\data\PickLight\Historical Data Timestamp\Builder\pose\input folder.
1.4 Instance Filtering
1.4.1 Filter Based on Bounding Box Area
- Function Description
Filter based on the pixel area of the bounding box of the detected instance.
- Use Case
Applicable to scenarios where the areas of instance bounding boxes differ greatly. By setting the upper and lower limits of the bounding box area, noise in the image can be filtered out, improving the accuracy of image recognition and avoiding additional processing time caused by noise in subsequent processing.
- Parameter Description
| Parameter | Description | Default Value | Parameter Range | Unit |
|---|---|---|---|---|
| Minimum Area (Pixels) | This parameter is used to set the minimum filtering area of the bounding box. Instances whose bounding box area is lower than this value will be filtered out | 1 | [1, 10000000] | Pixels |
| Maximum Area (Pixels) | This parameter is used to set the maximum filtering area of the bounding box. Instances whose bounding box area is higher than this value will be filtered out | 10000000 | [2, 10000000] | Pixels |
- Example
Run with the default values, and you can view the bounding box area of each instance in the log, as shown in the figure below.
Adjust **Minimum Area ** and Maximum Area according to the bounding box area of each instance. For example, if Minimum Area is set to 20000 and Maximum Area is set to 30000, instances with pixel areas less than 20000 or greater than 30000 can be filtered out, and the instance filtering process can be viewed in the log.
1.4.2 Filter Based on Bounding Box Aspect Ratio
- Function Description
Instances whose bounding box aspect ratio is outside the specified range will be filtered out
- Use Case
Applicable to scenarios where the aspect ratios of instance bounding boxes differ greatly
- Parameter Description
| Parameter | Description | Default Value | Parameter Range |
|---|---|---|---|
| Minimum Aspect Ratio | The minimum value of the bounding box aspect ratio. Instances whose bounding box aspect ratio is lower than this value will be filtered out | 0 | [0, 10000000] |
| Maximum Aspect Ratio | The maximum value of the bounding box aspect ratio. Instances whose bounding box aspect ratio is higher than this value will be filtered out | 10000000 | [0, 10000000] |
| Use X/Y Axis Side Length as Aspect Ratio | Not selected by default. The ratio of the longer side to the shorter side of the bounding box is used as the aspect ratio, which is suitable when the lengths of the longer and shorter sides of the bounding box differ greatly; After it is selected, the ratio of the side length of the bounding box on the X-axis to that on the Y-axis in the pixel coordinate system is used as the aspect ratio, which is suitable when the longer-side/shorter-side ratio of most normal instance bounding boxes is similar, but the ratio of the length on the X-axis to the length on the Y-axis of some abnormally recognized instance bounding boxes differs greatly. | Not selected | / |
1.4.3 Filter Instances Based on Category ID
- Function Description
Filter according to the instance category
- Use Case
Applicable to scenarios where incoming materials contain multiple types of target objects
- Parameter Description
| Parameter | Description | Default Value |
|---|---|---|
| Retained Category IDs | Retain instances whose category IDs are in the list. Instances whose category IDs are not in the list will be filtered out | [0] |