Vision Parameter Adjustment Guide for Ordered Loading/Unloading of Planar Workpieces (Parallelized)
Getting Started
Background Introduction
The new "Ordered Loading/Unloading of Planar Workpieces (Parallelized)" scenario is a vision acceleration solution for ordered planar scenes. Its main feature is that it improves visual computation Takt Time, reduces computation time through parallel computing, and directly generates Pick Points.
Unlike the way visual acceleration mode is enabled in other scenarios, the ordered planar scenario is enabled by creating a new task.
Task Scenario Selection
In the current ordered planar scenario, there are two available task scenarios. Their comparison is shown in the table below.
| Workflow | Ordered Loading/Unloading of Planar Workpieces | Ordered Loading/Unloading of Planar Workpieces (Parallelized) | Remarks |
|---|---|---|---|
| Accuracy | High, depends on Point Cloud density and scene Point Cloud consistency | Relatively high, depends on image features, Point Cloud density, and scene Point Cloud consistency | / |
| Speed | Relatively fast (single instance) | Fast (multiple instances) | Only the parallelized mode can output all valid results in the scene |
| Parameter tuning | Medium, requires matching parameter tuning experience | Simple, relatively simple and fixed (similar to general workpieces) | Some advanced parameters will be hidden in the parallelized mode in the future |
| Applicability | Strong, suitable for all planar workpieces | Relatively weak; the current version supports cases where incoming material orientations are inconsistent | The parallelized mode has been updated with multi-template mode and supports multiple incoming directions |
| Template creation difficulty | Average, software-supported | Relatively complex, the current version requires scripts | Template creation for the parallelized mode will be integrated into PickWiz in the future |
| Template characteristics | For reference | Select a complete scene instance Point Cloud relatively centered in the Camera field of view | / |
Build the Project
(1) Create a new ordered loading/unloading project for planar workpieces (parallelized) (the project name and project path can be customized, but the project name cannot contain Chinese characters)
Workpiece type: planar workpiece (not circular, cylindrical, or quadrilateral, and with relatively small differences between the front and back sides)
(2) Camera and Robot configuration
(3) Add the workpiece
- Workpiece Information
The workpiece name can be customized. The workpiece type defaults to standard workpiece and cannot be changed. The workpiece ID can be customized and is used to automatically switch workpieces during robot picking.
Point cloud file: the workpiece point cloud template. In the planar workpiece ordered loading/unloading (parallelized) scenario, the point cloud file is special. For the creation method, refer to 2.2.1 Template File Path
Fine matching point cloud template: used for fine matching
Camera parameters: not required
- Model Information
Vision model: the 2D recognition solution used for planar workpiece applications is CAD-based synthetic data training (One-click Connection). The vision model for different planar workpiece applications must be obtained through one-click connection training.
Mesh file: normally upload the workpiece CAD. To remove some noise, a standardized mesh file is required. The mesh file can also be standardized in Point Cloud Template Creation.
Workpiece attributes: elongated, symmetrical, highly reflective, and low-solidity
Incoming material type: custom incoming material type — enter the incoming material type; tightly fitted — the range of the number of workpieces in each row and column
task environment: enter the environment file. In one-click connection, the environment used for data generation is automatically replaced with the entered environment to improve recognition results.
Workpiece texture: enter the workpiece texture. During model training in one-click connection, the entered workpiece texture is used for data augmentation to improve recognition results.
Mixed random-scene data: when enabled, one-click connection generates synthetic data for both random and ordered scenes during model training to improve recognition results.
Maximum number of model recognitions: the default is 20; modify it according to scene requirements.
- Pick Point: configure the Pick Point according to the workpiece
Absolute coordinate system: uses the initial point as the origin. The initial point is built into the workpiece point cloud and CAD.
Pick Point coordinate system (offset): uses the current Pick Point as the origin.
(4) Add Tool, eye-hand calibration, and ROI
(5) Optional functional options: instance optimization, collision detection, visual classification, point cloud denoising, and bin/workpiece collision detection
Instance optimization: optimizes instances generated by the model and processes the instance masks.
Collision detection: this function is used to detect collisions between the Tool and the container, and to filter out picking poses that may collide. Collision Detection User Guide
Visual classification: used to identify features such as different textures and different orientations of the same workpiece. Visual Classification User Guide
Point cloud denoising: you can import the workpiece point cloud template to remove noise points from the instance workpiece point cloud. Point Cloud Denoising User Guide
Bin and workpiece collision detection: detects collisions between the Tool and the scene (bin, scene point cloud), and filters out picking poses that may collide. Bin and Workpiece Collision Detection User Guide
Vision Parameters
- 2D recognition: recognize and segment instances from the actual scene
Preprocessing: process the 2D image before instance segmentation (commonly used: fill holes in the depth map & edge enhancement & extract the top-layer texture & remove image background outside ROI 3D)
Instance segmentation: segment instances (scaling ratio & lower Confidence threshold & auto enhancement). To accelerate processing, you can clear the checkbox for Return Mask.
Point cloud generation: the method for generating instance point clouds — generate instance point clouds using segmented instance masks or bounding boxes / generate instance point clouds using filtered instance masks or bounding boxes
Instance filtering: filter the segmented instances
Instance sorting: sort instances
- 3D computation: calculate the pose of the instance in the Camera coordinate system and generate Pick Points
Preprocessing: preprocess the 3D point cloud before calculating Pick Points
Pose estimation: calculate the pose of the instance in the Camera coordinate system (coarse matching and fine matching) and generate Pick Points
- Pick Point processing: filter, adjust, and sort Pick Points
Pick Point filtering: filter Pick Points
Pick Point adjustment: adjust Pick Points
Pick Point sorting: sort Pick Points
1. 2D识别
1.1 预处理
2D识别的预处理是在实例分割之前对2D图像进行处理
1.1.1 双边滤波
- 功能
基于双边滤波的图像平滑功能
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 最大深度差值 | 双边过滤的最大深度差值 | 0.03 | [0.01, 1] |
| 过滤核大小 | 双边过滤卷积核大小 | 7 | [1, 3000] |
1.1.2 深度转法向量图
- 功能
通过深度图计算像素法向量，并把图片转换成法向量图
1.1.3 图像增强
- 功能
常用图像增强，如色彩饱和度、对比度、亮度、锐利度
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 图像增强类型 | 对图像的某个元素进行增强 | 对比度 | 色彩饱和度、对比度、亮度、锐利度 |
| 图片增强阈值 | 对图像的某个元素增强多少 | 1.5 | [0.1, 100] |
1.1.4 直方图均衡
- 功能
提高图像的对比度
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 局部模式 | 局域或全局直方图均衡，勾选则局域直方图均衡，取消勾选则全局直方图均衡 | 勾选 | / |
| 对比度阈值 | 对比度阈值 | 3 | [1,1000] |
1.1.5 通过颜色过滤深度图
- 功能
根据颜色值过滤深度图
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 填充核大小 | 颜色填充的大小 | 3 | [1,99] |
| 根据hsv过滤深度-色域最大值 | 最大颜色值 | [180,255,255] | [[0,0,0],[255,255,255]] |
| 根据hsv过滤深度-色域最小值 | 最小颜色值 | [0,0,0] | [[0,0,0],[255,255,255]] |
| 保存颜色范围内的区域 | 勾选则保存颜色范围内的区域，不勾选则保存颜色范围外的区域 | / | / |
1.1.6 伽马图片校正
- 功能
gamma校正改变图片亮度
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| gamma补偿系数 | 该值小于1，图片变暗该值大于1，图片变量 | 1 | [0.1,100] |
| gamma校正系数 | 该值小于1，图片变暗，适用于亮度过高的图片该值大于1，图片变亮，适用于亮度过低的图片 | 2.2 | [0.1,100] |
1.1.7 填充深度图空洞
- 功能
对深度图中的空洞区域进行填充，并对填充后的深度图进行平滑处理
- 使用场景
因工件本身结构遮挡、光照不均匀等问题，深度图可能缺失工件的部分区域
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 填充核大小 | 空洞填充的大小 | 3 | [1,99] |
填充核大小只能填入奇数
- 调参
根据检测结果调整，如果填充过度，应调小参数；如果填充不足，应调大参数
- 示例
1.1.8 边缘增强
- 功能
把图像中纹理的边缘部分置为背景色或者与背景色相差较大的颜色，以凸显工件的边缘信息
- 使用场景
工件相互遮挡或重叠导致边缘不清晰
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 | 调参建议 |
|---|---|---|---|---|
| 法向z方向过滤阈值 | 深度图每个点对应的法向量与相机坐标系Z轴正方向的角度过滤阈值，若点的法向量与相机坐标系的Z轴正方向的角度大于此阈值，则2D图中该点对应位置的颜色将会被置为背景色或者与背景色相差较大的颜色 | 30 | [0,180] | 对于平整工件表面，该阈值可以小一些，曲面工件根据表面倾斜程度适当增大 |
| 背景色 | 背景色的RGB颜色阈值 | 128 | [0,255] | |
| 自动调节反差背景 | 勾选 自动调节反差背景后，将2D图中角度大于过滤阈值的点的颜色设置为与背景色相差较大的颜色不勾选 自动调节反差背景，将2D图中角度大于过滤阈值的点的颜色设置为背景色对应的颜色 | 不勾选 | / |
- 示例
1.1.9 提取最上层纹理
- 功能
提取最上层或最底层的工件纹理，而把其他区域置为背景色或者与背景颜色相差较大的颜色。
- 使用场景
光照条件不佳、颜色纹理相近、紧密堆叠、交错堆叠或遮挡等因素可能导致模型难以区分上层和下层工件的纹理差异，因此容易误检测。
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 | 单位 | 调参建议 |
|---|---|---|---|---|---|
| 距离阈值(mm) | 点与最上层平面（最底层平面）的距离低于该阈值，则被认为是最上层平面（最底层平面）内的点，应当保留，否则认为是下层（上层）的点，将下层（上层）的点的颜色置为背景色或与背景色相差较大的颜色 | 50 | [0.1, 1000] | mm | 一般调整为工件高度的1/2 |
| 聚类点云数量 | 期望参与聚类的点数量，在ROI 3D区域内采样点云的数量 | 10000 | [1，10000000] | / | 聚类点云数量越多，模型推理速度下降而精度提升；聚类点云数量越少，模型推理速度提升而精度下降 |
| 类别点最小数量 | 用于过滤类别的最小点数 | 1000 | [1, 10000000] | / | / |
| 自动计算反差背景 | 勾选 自动计算反差背景后，将2D图中最上层（最底层）外的其他区域设置为与背景色阈值相差较大的颜色不勾选 自动计算反差背景，将2D图中最上层（最底层）外的其他区域设置为背景色阈值对应的颜色 | 勾选 | / | / | / |
| 背景色阈值 | 背景色RGB颜色阈值 | 128 | [0,255] | / | / |
1.1.10 去除roi3d外的图片背景
- 功能
去除2D图像中除了ROI3D区域以外的背景
- 使用场景
图像背景噪声较多影响检测结果
- 参数说明
| 参数名 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 填充核大小 | 空洞填充的大小 | 5 | [1,99] |
| 迭代次数 | 图片膨胀的迭代次数 | 1 | [1,99] |
| 自动计算反差背景 | 勾选 自动计算反差背景后，将2D图中roi以外的区域设置为与背景色阈值相差较大的颜色不勾选 自动计算反差背景，将2D图中roi以外的区域设置为背景色阈值对应的颜色 | 勾选 | / |
| 背景色阈值 | 背景色的RGB颜色阈值 | 128 | [0,255] |
填充核大小只能填入奇数
- 调参
如果需要去除图像中更多背景噪声，应当调小填充核大小
- 示例
1.2 实例分割
1.2.1 缩放比例
- 功能
通过等比放缩原始图像后再推理以提升2D识别的准确率与召回率。
- 使用场景
检测效果不佳（如未检测到实例、漏识别、包围盒框到多个实例或框不满实例)应调整该函数。
- 参数说明
默认值：1.0
取值范围：[0.01, 3.00]
步长：0.01
调参
- 使用默认值运行，在可视化视窗查看检测结果，若出现未检测到实例、漏识别、包围盒框到多个实例或框不满实例的情况，应调整该函数。
2D识别中实例的百分数为置信度分数，数字为实例ID（实例的识别顺序）。
2D识别中实例上有颜色的阴影是掩膜，包围实例的矩形框是包围盒。
- 尝试不同的缩放比例，观察检测结果的变化，逐步确定缩放比例的范围。如果在某个缩放比例下，检测效果显著提高，则将该缩放比例作为下限；在某个缩放比例下，检测效果显著降低，则将该缩放比例作为上限。
若尝试所有缩放比例都无法获得较好检测结果，可调整ROI区域
如下图所示，缩放比例为0.33时检测效果显著提高，因此可以确定0.33为缩放比例范围的下限
缩放比例为3时检测效果依然较好，因此可以确定3为缩放比例范围的上限
- 如果实际场景对抓取精度要求不高，可以在[0.33,3]区间内选择一个检测效果较好的缩放比例；如果实际场景对抓取精度要求较高，应当进一步细化缩放比例范围，按更小步长来调整，直至找到检测效果最佳的缩放比例。
1.2.2 置信度下限阈值
- 功能
仅保留深度学习模型识别结果分数高于置信度下限阈值的识别结果
- 使用场景
检测结果框取的实例不符预期时，可调整该函数
- 参数说明
默认值：0.5
取值范围：[0.01, 1.00]
调参
- 如果模型检测出的实例较少，应当调小该阈值；取值过小，可能会影响图像识别的准确度。
- 若因为置信度下限阈值过小导致检测出错误的实例，而需要去除这些错误的实例，应当调大该阈值；取值过大，可能会导致保留的检测结果为零，没有结果输出。
1.2.3 启用自动增强
- 功能
将输入的缩放比例和旋转角度中所有的值进行组合后推理，返回组合后大于设定置信度下限阈值的所有结果，可以提升模型推理精度, 但会增加耗时。
- 使用场景
单个缩放比例无法满足实际场景需求导致检测不完整或物体摆放倾斜度较高。
- 示例
如果自动增强-缩放比例设置为 [0.8, 0.9, 1.0] ，自动增强-旋转角度设置为 [0, 90.0] ，那么将缩放比例和旋转角度中的值两两组合，模型内部会自动生成6种图片进行推理，最后将这6种推理结果统一到一起，输出大于置信度下限阈值的结果。
自动增强-缩放比例
- 功能
对原始图像进行多次缩放并进行多次推理，输出综合的推理结果
- 使用场景
单个缩放比例无法满足实际场景需求导致检测不完整
- 参数说明
默认值：[1.0]
取值范围：每个缩放比例的范围为[0.1, 3.0]
可设置多个缩放比例，每个缩放比例之间用英文逗号隔开
- 调参
填入多个 1.2.1 缩放比例 获得的检测效果较好的缩放比例
自动增强-旋转角度
- 功能
对原始图像多次旋转并进行多次推理，输出综合的推理结果
- 使用场景
物体摆放偏离坐标轴较多时使用
- 参数说明
默认值：[0.0]
取值范围：每个旋转角度的取值范围为[0, 360]
可设置多个旋转角度，每个旋转角度之间用英文逗号隔开
- 调参
根据实际场景中的物体角度调整自动增强-旋转角度，可根据麻袋的图案和袋口形状、纸箱的棱边和品牌标志判断倾斜角度
1.3 点云生成
| 实例点云生成形式 | 掩膜形式（分割后） | — | 使用分割后的实例掩膜生成点云 |
| 包围盒形式（分割后） | 包围盒缩放比例（分割后） | 使用分割后的实例包围盒生成点云 | |
| 生成点云是否需要颜色（分割后） | 生成的实例点云是否需要附着颜色 | ||
| 掩膜形式（过滤后） | — | 使用过滤后的实例掩膜生成点云 | |
| 包围盒形式（过滤后） | 包围盒缩放比例（过滤后） | 使用过滤后的实例包围盒生成点云 | |
| 生成点云是否需要颜色（过滤后） | 生成的实例点云是否需要附着颜色 |
如果不需要加速，无需使用实例过滤函数，使用掩膜形式（分割后）或包围盒形式（分割后）生成实例点云，可在 项目存储文件夹\Project Name\data\PickLight\Historical Data Timestamp\Builder\pose\input folder containing the generated instance point cloud；
如果需要加速，可使用实例过滤函数对实例进行过滤，使用掩膜形式（过滤后）或包围盒形式（过滤后）生成实例点云，可在 项目存储文件夹\Project Name\data\PickLight\Historical Data Timestamp\Builder\pose\input folder containing the generated instance point cloud
1.4 实例过滤
1.4.1 基于包围盒面积过滤
- 功能介绍
根据检测出实例的包围盒的像素面积来进行过滤。
- 使用场景
适用于实例包围盒面积相差较大的场景，通过设置包围盒面积的上限和下限来过滤图像中的噪声，提升图像识别的准确度，避免噪声给后续处理增加耗时。
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 | 单位 |
|---|---|---|---|---|
| 最小面积（像素） | 该参数用于设置包围盒的最小过滤面积，包围盒面积低于这个值的实例会被过滤 | 1 | [1, 10000000] | 像素点 |
| 最大面积（像素） | 该参数用于设置包围盒的最大过滤面积，包围盒面积高于这个值的实例会被过滤 | 10000000 | [2, 10000000] | 像素点 |
- 示例
按默认值运行，可在日志中查看每个实例的包围盒面积，如下图所示。
根据每个实例的包围盒面积调整 最小面积 和 最大面积，如将最小面积设置为20000，将最大面积设置为30000，即可将像素面积为小于20000或大于30000的实例过滤掉，可在日志中查看实例过滤过程。
1.4.2 基于包围盒长宽比过滤
- 功能介绍
包围盒长宽比在指定范围外的实例将被过滤掉
- 使用场景
适用于实例的包围盒长宽比相差较大的场景
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 |
|---|---|---|---|
| 最小长宽比 | 包围盒长宽比的最小值，包围盒长宽比低于该值的实例会被过滤 | 0 | [0, 10000000] |
| 最大长宽比 | 包围盒长宽比的最大值，包围盒长宽比高于该值的实例会被过滤 | 10000000 | [0, 10000000] |
| 使用X/Y轴边长作长宽比 | 默认不勾选，使用包围盒的较长边/较短边的长度比值作为长宽比，适用于包围盒的长短边长度相差大的情况； 勾选后，则使用像素坐标系下包围盒在X轴/Y轴上的边的长度比值作为长宽比，适用于大部分正常实例包围盒的长边/短边比值近似，但部分异常识别的实例包围盒在X轴上的长度/在Y轴上的长度的比值相差较大的情况。 | 不勾选 | / |
1.4.3 基于类别ID过滤实例
- 功能介绍
根据实例类别过滤
- 使用场景
适用于来料有多种类型工件的场景
- 参数说明
| 参数 | 说明 | 默认值 |
|---|---|---|
| 保留的类别ID | 保留类别ID在列表内的实例，类别ID不在列表内的实例将被过滤 | [0] |
- 示例
1.4.4 基于实例点云的边长过滤
- 功能介绍
根据实例点云的长边和短边过滤
- 使用场景
适用于实例点云在x轴或y轴的距离相差较大的场景，通过设置实例点云的距离范围来过滤图像中的噪声，提升图像识别的准确度，避免噪声给后续处理增加耗时。
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 | 单位 |
|---|---|---|---|---|
| 短边长度范围（mm） | 点云短边的边长范围 | [0, 10000] | [0, 10000] | mm |
| 长边长度范围（mm） | 点云长边的边长范围 | [0, 10000] | [0, 10000] | mm |
| 边缘去噪下限（%） | 提取实例点云中X/Y值（相机坐标系）的百分比下限，去除上下限外的点云，避免噪点影响长度计算 | 5 | [0, 100] | / |
| 边缘去噪上限（%） | 提取实例点云中X/Y值（相机坐标系）的百分比上限，去除上下限外的点云，避免噪点影响长度计算 | 95 | [0, 100] | / |
| 边长类型 | 按实例点云的长边、短边过滤，长边、短边的长度不在范围内的实例将被过滤 | 实例点云短边 | 实例点云短边；实例点云长边；实例点云长边和短边 | / |
- 示例
1.4.5 基于分类器的类别ID过滤
- 功能介绍
基于分类器的类别 ID 过滤实例，不在参考类别内的实例将被过滤。
- 使用场景
在多类工件场景中，视觉模型可能会检测出多种类型的工件，但实际作业可能仅需其中某一种类别的工件，此时就可以使用该函数过滤掉不需要的工件
- 参数说明
默认值为[0]，即默认保留类别 ID 为 0 的实例，类别 ID 不在列表内的实例将被过滤。
1.4.6 基于三通道颜色过滤
- 功能介绍
可通过三通道颜色阈值（HSV或者RGB）过滤掉实例。
- 使用场景
错误实例和正确实例颜色有明显区分的情况。
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 色域最大值 | 最大颜色值 | [180,255,255] | [[0,0,0],[255,255,255]] |
| 色域最小值 | 最小颜色值 | [0,0,0] | [[0,0,0],[255,255,255]] |
| 过滤百分比阈值 | 颜色通过率阈值 | 0.05 | [0,1] |
| 反向过滤 | 勾选则剔除颜色范围外的比例低于阈值的实例，不勾选则剔除实例图像中颜色范围内的比例低于阈值的实例 | 不勾选 | / |
| 颜色模式 | 颜色过滤中选择的颜色空间 | HSV色彩空间 | RGB色彩空间HSV色彩空间 |
- 示例
1.4.7 基于置信度过滤
- 功能介绍
根据实例的置信度分数过滤
- 使用场景
适用于实例的置信度相差较大的场景
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 |
|---|---|---|---|
| 参考置信度度 | 保留置信度大于阈值的实例，过滤置信度小于阈值的实例。 | 0.5 | [0,1] |
| 反转过滤结果 | 反转后，保留可见度置信度小于阈值的实例，过滤置信度大于阈值的实例。 | 不勾选 | / |
- 示例
1.4.8 基于点云数量过滤
- 功能介绍
根据降采样后的实例点云数量过滤
- 使用场景
实例点云带有较多噪声
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 |
|---|---|---|---|
| 点云数量最小值 | 点云数量的最小值 | 3500 | [1, 10000000] |
| 点云数量最大值 | 点云数量的最大值 | 8500 | [2, 10000000] |
| 过滤数量在区间内的实例 | 勾选则过滤点云数量在最小值和最大值区间内的实例，不勾选则过滤点云数量不在区间内的实例 | 不勾选 | / |
1.4.9 基于掩膜面积过滤
- 功能介绍
根据检测出实例的掩膜像素和（即像素面积）过滤图像掩膜。
- 使用场景
适用于实例掩膜面积相差较大的场景，通过设置掩膜的面积上限和下限来过滤图像掩膜中的噪声，提升图像识别的准确度，避免噪声给后续处理增加耗时。
- 参数设置说明
| 参数名 | 说明 | 默认值 | 参数范围 | 单位 |
|---|---|---|---|---|
| 参考最小面积 | 该参数用于设置掩膜的最小过滤面积，掩膜面积低于这个值的实例会被过滤 | 1 | [1, 10000000] | 像素点 |
| 参考最大面积 | 该参数用于设置掩膜的最大过滤面积，掩膜面积高于这个值的实例会被过滤 | 10000000 | [2, 10000000] | 像素点 |
- 示例
1.4.10 基于可见度过滤
- 功能介绍
根据实例的可见度分数过滤
- 使用场景
适用于实例的可见度相差较大的场景
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 |
|---|---|---|---|
| 参考可见度阈值 | 保留可见度大于阈值的实例，过滤可见度小于阈值的实例。可见度用于判断图像中的实例可见的程度，工件被遮挡越多，可见度越低。 | 0.5 | [0,1] |
| 反转过滤结果 | 反转后，保留可见度小于阈值的实例，过滤可见度大于阈值的实例。 | 不勾选 | / |
1.4.11 过滤包围盒重叠的实例
- 功能介绍
过滤包围盒交叉重叠的实例
- 使用场景
适用于实例的包围盒相互交叉的场景
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 |
|---|---|---|---|
| 包围盒重叠比例阈值 | 包围盒交叉的面积与实例包围盒的面积的占比阈值 | 0.05 | [0, 1] |
| 过滤包围盒面积较大的实例 | 勾选则过滤两个包围盒交叉的实例中面积较大的实例，不勾选则过滤两个包围盒交叉的实例中面积较小的实例 | 勾选 | / |
- 示例
新增 过滤被包围实例，以默认值运行，在日志中查看实例包围盒交叉的情况，实例过滤后剩余2个实例
由日志可知，12个实例因为包围盒交叉被过滤掉，剩余2个包围盒没有交叉的实例
将 包围盒重叠比例阈值 设置为0.1，勾选 是否过滤较大的实例，在日志中查看实例例过滤过程，9个实例因为包围盒交叉面积与实例包围盒面积的占比大于0.1被过滤掉，3个实例因为包围盒交叉面积与实例包围盒面积的占比小于0.1被保留，2个实例包围盒没有交叉。
将 包围盒重叠比例阈值 设置为0.1，取消勾选 是否过滤较大的实例，在日志中查看实例例过滤过程，9个实例的包围盒交叉面积与实例包围盒面积的占比大于0.1，但其中2个实例因为包围盒面积小于与其交叉的实例被保留，因此7个实例被过滤，3个实例因为包围盒交叉面积与实例包围盒面积的占比小于0.1被保留，2个实例包围盒没有交叉。
1.4.12 【大师】基于掩膜/掩膜外接多边形面积比，过滤掩膜凹凸的实例
- 功能介绍
计算掩膜/掩膜外接多边形的面积比值，若小于设置的阈值则会过滤掉实例
- 使用场景
适用于工件掩膜存在锯齿/凹凸的情况。
- 参数说明
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 面积比阈值 | 掩膜/凸包面积比阈值，若小于设置的阈值则会过滤掉实例。 | 0.1 | [0,1] |
1.4.13 【大师】基于点云平均距离过滤
- 功能介绍
基于点云中点到拟合平面的距离的平均值进行过滤，剔除不平整的实例点云
- 使用场景
适用于平面型工件点云弯曲的场景
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 | 单位 |
|---|---|---|---|---|
| 平面分割距离阈值（mm） | 在弯曲的实例点云中提取一个平面，与平面的距离小于该阈值的点视为该平面的点 | 10 | [-1000, 1000] | mm |
| 平均距离阈值（mm） | 实例点云中的点到提取平面的距离的平均值 | 20 | [-1000, 1000] | mm |
| 剔除平均距离小于阈值的实例 | 勾选则过滤点到提取平面的平均距离小于平均距离阈值的实例，不勾选则过滤点到提取平面的平均距离大于平均距离阈值的实例 | 不勾选 | / | / |
1.4.14 【大师】基于掩膜/包围盒面积比，过滤被遮挡的实例
- 功能介绍
计算掩膜/包围盒面积比值，比值不在最大最小范围内的实例将被过滤
- 使用场景
用于过滤被遮挡工件的实例
- 参数说明
，相反，代表可能被遮挡。
| 参数 | 说明 | 默认值 | 取值范围 |
|---|---|---|---|
| 最小面积比 | 掩膜/包围盒面积比例范围下限，比值越小，说明实例被遮挡程度越高 | 0.1 | [0,1] |
| 最大面积比 | 掩膜/包围盒面积比例范围上限，比值越接近1，说明实例被遮挡程度低 | 1.0 | [0,1] |
1.4.15 【大师】判断最上层实例是否全量检出
- 功能介绍
防呆机制之一，判断最上层的实例是否全部被检出，若存在没有被检出的最上层实例则会报错并结束工作流
- 使用场景
适用于拍一次抓多次或者必须要按顺序进行抓取的场景，防止因实例检出不完整造成漏抓影响后续作业
- 参数说明
| 参数 | 说明 | 默认值 | 参数范围 | 单位 | 调参 |
|---|---|---|---|---|---|
| 距离阈值 | 用于判断最上层的工件，点与工件点云最高点的距离小于距离阈值，则认为这个点是最上层点云，否则认为这个点不是最上层点云。 | 5 | [0.1, 1000] | mm | 应当小于工件的高度 |
1.5 实例排序
- 功能介绍
根据选择的策略对实例进行分组、排序、提取
- 使用场景
拆垛、无序抓取、有序上下料场景通用
如果不需要排序，可以不配置具体的策略。
1.5.1 基准坐标系
- 功能介绍
为所有实例设定一个统一的坐标系，进行实例的分组排序
- 使用场景
拆垛场景、无序抓取场景、有序上下料场景通用
使用坐标相关的策略应当先设置基准坐标系
- 参数说明
| 参数 | 说明 | 图示 |
|---|---|---|
| 相机坐标系 | 坐标系原点在物体上方，Z轴正方向朝下；XYZ取值是物体中心点在该坐标系下的值 |  |
| ROI坐标系 | 坐标系原点大致在垛中心，Z轴正方向朝上；XYZ取值是物体中心点在该坐标系下的值 |  |
| 机械臂坐标系 | 坐标系原点在机械臂自身，Z轴正方向一般朝上；XYZ取值是物体中心点在该坐标系下的值 |  |
| 像素坐标系 | 坐标系原点在RGB图的左顶点，是二维平面坐标系；X、Y取值是bbox识别框的x值、bbox识别框的y值，Z是0 |  |
1.5.2 通用抓取策略
- 参数说明
| 参数 | 说明 | 默认值 |
|---|---|---|
| 策略 | 选择依据哪个值进行分组排序以及如何排序，包括实例点云中心XYZ坐标值、包围盒长宽比、实例点云中心距离ROI中心等可叠加多条，按顺序依次执行 | 实例点云中心X坐标值从小到大(mm) |
| 分组步长 | 依据选择的策略、按照步长将实例划分为若干组，分组步长即两组实例之间的间隔如策略选择”实例点云中心Z坐标值从大到小(mm)“，则将所有实例点云中心的Z坐标从大到小排序，然后按照步长把Z坐标分组，相应的实例也划分为若干组 | / |
| 提取前几组 | 分组排序之后，需要保留多少组实例 | 10000 |
| 策略名* | 说明 | 分组步长 | 提取前几组 | |
|---|---|---|---|---|
| 默认值 | 取值范围 | 默认值 | ||
| 实例点云中心XYZ坐标值从大到小/从小到大(mm) | 使用每个实例的点云中心的XYZ坐标值来进行分组排序 使用该策略进行排序前应当先设置基准坐标系 | 200.000 | (0, 10000000] | 10000 |
| 从实例点云中心XY坐标轴向的中间到两侧/从实例点云中心XY坐标轴的两侧到中间(mm) | 使用每个实例的点云中心的 XY 坐标值，按照 “中间到两侧” 或 “两侧到中间” 的方向进行分组排序 使用该策略进行排序前应当先设置基准坐标系 | 200.000 | (0, 10000000] | 10000 |
| 包围盒中心XY坐标值从大到小/从小到大(mm) | 使用像素坐标系下，每个实例的包围盒中心点的XY 坐标值进行分组排序 | 200.000 | (0, 10000000] | 10000 |
| 包围盒长宽比从大到小/从小到大 | 使用包围盒的长边/宽边的比值进行分组排序 | 1 | (0, 10000] | 10000 |
| 从包围盒中心XY坐标轴向的中间到两侧/两侧到中间(mm) | 使用包围盒的中心点的XY坐标值，按照 “中间到两侧” 或 “两侧到中间” 的方向进行分组排序 | 200.000 | (0, 10000000] | 10000 |
| 工件类型ID从大到小/从小到大 | 使用工件类型的ID进行分组排序，适用于多类工件场景 | 1 | [1, 10000] | 10000 |
| 局部特征ID从大到小/从小到大 | 使用局部特征的ID进行分组排序 | 1 | [1, 10000] | 10000 |
| 置信度从大到小/从小到大 | 使用每个实例的置信度进行分组排序 | 1 | (0, 1] | 10000 |
| 可见度从小到大/从大到小 | 使用每个实例的可见度进行分组排序 | 1 | (0, 0.1] | 10000 |
| 掩膜面积从大到小/从小到大 | 使用每个实例的掩膜面积进行分组排序 | 10000 | [1, 10000000] | 10000 |
| 实例点云中心距离ROI中心近到远/远到近(mm) | 使用每个实例的点云中心与ROI坐标系的中心的距离进行分组排序 | 200.000 | (0, 10000000] | 10000 |
| 实例点云中心距离机器人坐标原点近到远/远到近(mm) | 使用每个实例的点云中心与机器人坐标系的原点的距离进行分组排序 | 200.000 | (0, 10000000] | 10000 |
- 示例
1.5.3 自定义抓取策略
（1）功能说明
将Grasping Strategy切换为Custom Grasping Strategy，点击 Add 可增加一条自定义抓取策略。
自定义每个工件按照什么顺序抓取，使用通用抓取策略很难实现抓取或者因为点云噪点等问题很难调到合适的参数，可以考虑使用自定义抓取策略
自定义抓取策略适用于拆垛场景、有序上下料场景，无序抓取场景不适用，因为自定义抓取策略的工件必须是有序的（即工件的顺序固定）
自定义抓取策略只能和单个通用抓取策略组合使用，且策略只能选择Z坐标从小到大
（2）参数说明
| 参数 | 说明 | 默认值 | 取值范围 | 调参 |
|---|---|---|---|---|
| IOU阈值 | 表示标注的bbox框和检测出来的bbox框的重叠度阈值，通过重叠度来确定当前工件实例排序时应该选择哪一张图片上的排序方式。 | 0.7 | [0,1] | 阈值越大，匹配越严格，抗干扰性会越差，微小的形状或位置变化都可能导致匹配失败，可能匹配到错误的自定义策略，按错误的顺序进行排序 |
| 像素距离阈值 | 表示可以匹配上的bbox框和检测出来的bbox框在尺寸上的差异性。 | 100 | [0,1000] | 阈值越小，匹配越严格，抗干扰性也会更好。如果不同层之间的工件摆放比较相似，也可能误匹配自定义策略，导致排序顺序错误。 |
（3）选择基准坐标系
使用自定义抓取策略，只能选择相机坐标系或像素坐标系
如果有多层工件，则选择相机坐标系；如果只有一层工件，则选择像素坐标系
（4）策略、分组步长、提取前几组
| 参数 | 说明 | 默认值 |
|---|---|---|
| 策略 | 只能选择实例点云中心Z坐标值从大到小/从小到大(mm) | / |
| 分组步长 | 依据Z坐标从小到大策略，将实例的Z坐标从小到大排序，按照步长将实例划分为若干组 | 10000 |
| 提取前几组 | 分组排序之后，需要保留多少组实例 | 10000 |
（5）拍照取图/添加本地图像
点击拍照取图从当前连接的相机获取图像，或点击添加本地图像从本地导入图像，有多少层或有多少种工件的不同摆放形式，就需要拍照取图或添加本地图像得到多少张图片，如果每一层相同，只需要一张即可。鼠标右键点击图像可删除。
在获取的图像上长按拖动鼠标左键标注bbox框，DELETE键可逐步删除标注的bbox框。
2. 3D计算
2.1 预处理
3D计算的预处理是在对实例进行姿态估计、生成抓取点之前对3D点云进行处理，面型工件有序上下料（并行化）场景无需对3D点云进行处理。
2.2 点云匹配姿态估计
2.2.1 模板文件路径
- 功能
上传点云模板与场景的实例点云进行匹配
- 使用场景
面型工件有序上下料（并行化）场景
- 调参说明
该点云模板需使用模板生成脚本制作，制作方法如下：
- 复制场景的2D图、深度图、点云图
（1）选择要复制的历史数据的时间戳
从PickLight历史数据文件夹中选取一个时间戳文件夹（如 /home/xxx/PickLight/20240718201333036），复制其完整路径备用。
（2）下载模板生成脚本
注意：
模板生成脚本需要下载与软件版本相对应的版本，否则版本不兼容会导致点云模板生成失败。
模板生成脚本的下载目录不能有中文或特殊字符，建议存储在默认的下载目录C:
\Users\dex\Downloads
PickWiz 版本 > =1.8.0 模版生成脚本：点击查看完整代码
Template generation script
import argparse
import json
import math
import os
import re
import shutil
from copy import deepcopy
import cv2
import numpy as np
import open3d as o3d
from tqdm import tqdm
from PickLight.Utils.Convertor import generate_mask_from_points
from PickLight.Utils.Convertor import get_ratio_from_mask
from PickLight.Utils.Utility import FileOperation
try:
import glia
if not glia.__version__ >= "0.2.4":
raise RuntimeError("Please upgrade glia to version 0.2.4 or later. The current version is {}".format(glia.__version__))
from glia.features.image_pair_matcher import ImagePairMatcher
except ImportError as e:
print(f"Warning: {e}")
ImagePairMatcher = None
try:
import torch
except ImportError:
torch = None
default_config_superpoint = {
'detector': {
'type': 'superpoint',
'params': {
'default': {
'keypoint_threshold': 0.01,
'max_keypoints': -1,
'use_edge': False,
'edge_kernel_size': 3,
},
'expand': {},
},
},
'matcher': {
'type': 'superglue',
'params': {
'default': {
'match_threshold': 0.01,
},
'expand': {},
},
},
'preprocess': {
'resize': False,
},
}
default_config_xfeat = {
'detector': {
'type': 'xfeat',
'params': {
'default': {
'keypoint_threshold': 0.01,
'top_k': 512,
},
'expand': {}
},
},
'matcher': {
'type': 'lighterglue',
'params': {
'default': {
'match_threshold': 0.01,
},
'expand': {}
},
},
'preprocess': {
'resize': False,
},
}
def file_transfer(args):
"""File copying feature, merged from two scripts"""
input_dir = args.data_dir
output_dir = args.output_dir
os.makedirs(output_dir, exist_ok=True)
# RGB+D images
try:
for root, dirs, files in os.walk(input_dir):
target_path = os.path.join(root, 'Builder', 'foreground', 'input')
if os.path.exists(target_path):
for file in os.listdir(target_path):
if file.endswith('.png') or file.endswith('.tiff'):
full_file_path = os.path.join(target_path, file)
shutil.copy(full_file_path, output_dir)
print(f'Copied: {file}')
except Exception as e:
raise ValueError(f"Check whether the Builder/foreground/input folder exists under the path {input_dir}, {e}")
# PCD point cloud files
try:
for root, dirs, files in os.walk(input_dir):
target_path = os.path.join(root, 'Builder', 'foreground', 'output')
if os.path.exists(target_path):
for file in os.listdir(target_path):
if file.endswith('.ply'):
full_file_path = os.path.join(target_path, file)
shutil.copy(full_file_path, output_dir)
print(f'Copied: {file}')
except Exception as e:
print(f'{e}, trying the path from the second script')
# Try the path from the second script
try:
for root, dirs, files in os.walk(input_dir):
target_path = os.path.join(root, 'Builder', 'foreground', 'input')
if os.path.exists(target_path):
for file in os.listdir(target_path):
if file.endswith('.ply'):
full_file_path = os.path.join(target_path, file)
shutil.copy(full_file_path, output_dir)
print(f'Copied: {file}')
except Exception as e2:
raise ValueError(f"Check whether the Builder/foreground/output or input folder exists under the path {input_dir}, {e2}")
# JSON configuration files
try:
for root, dirs, files in os.walk(input_dir):
target_path = os.path.join(root, 'Builder', 'foreground', 'input')
if os.path.exists(target_path):
for file in os.listdir(target_path):
# Compatible with the different requirements of the two scripts
if file.endswith('.json') and (args.type in ['feat', 'both'] or 'camera_param' in file):
full_file_path = os.path.join(target_path, file)
shutil.copy(full_file_path, output_dir)
print(f'Copied: {file}')
except Exception as e:
raise ValueError(f"Check whether JSON files exist under the path {input_dir}, {e}")
def _list_indices_by_json(dir_path):
"""List the existing indices of model_info_{i}.json in the directory"""
if not os.path.isdir(dir_path):
return []
pat = re.compile(r"model_info_(\d+)\.json$")
idxs = []
for f in os.listdir(dir_path):
m = pat.match(f)
if m:
idxs.append(int(m.group(1)))
return sorted(idxs)
def _get_next_index(dir_path):
"""Return the next available index"""
idxs = _list_indices_by_json(dir_path)
if not idxs:
return 0
return max(idxs) + 1
def _ensure_template_id_in_dir(dir_path, template_id_value):
"""Write template_id to all model_info_{i}.json files in the directory (fill in if missing)"""
for i in _list_indices_by_json(dir_path):
p = os.path.join(dir_path, f"model_info_{i}.json")
try:
d = json.load(open(p, "r", encoding="utf-8"))
except Exception:
continue
if "template_id" not in d:
d["template_id"] = int(template_id_value)
with open(p, "w", encoding="utf-8") as fw:
json.dump(d, fw, ensure_ascii=False, indent=4)
def _is_superglue_template_dir(dir_path):
"""Roughly validate whether this is a superglue template directory"""
if not os.path.isdir(dir_path):
return False
js = _list_indices_by_json(dir_path)
if not js:
return False
# At least temp_0.png or depth_model_0.tiff
has_temp = any(os.path.exists(os.path.join(dir_path, f"temp_{i}.png")) for i in js)
has_depth = any(os.path.exists(os.path.join(dir_path, f"depth_model_{i}.tiff")) for i in js)
return has_temp or has_depth
def _copy_one_template_block(src_dir, src_idx, dst_dir, dst_idx, template_id):
"""Copy one set of template files for an index from src to dst, rename them to the new index, and write template_id"""
patterns = [
("temp_{i}.png", "temp_{j}.png"),
("depth_model_{i}.tiff", "depth_model_{j}.tiff"),
("model_{i}.ply", "model_{j}.ply"),
("model_{i}_downsampled.ply", "model_{j}_downsampled.ply"),
("keypoints_{i}.ply", "keypoints_{j}.ply"),
("temp_vis_{i}.jpg", "temp_vis_{j}.jpg"),
("depth_mask_uint8_{i}.png", "depth_mask_uint8_{j}.png"),
]
os.makedirs(dst_dir, exist_ok=True)
for src_pat, dst_pat in patterns:
s = os.path.join(src_dir, src_pat.format(i=src_idx))
d = os.path.join(dst_dir, dst_pat.format(j=dst_idx))
if os.path.exists(s):
shutil.copy(s, d)
# Handle model_info
src_info = os.path.join(src_dir, f"model_info_{src_idx}.json")
dst_info = os.path.join(dst_dir, f"model_info_{dst_idx}.json")
if os.path.exists(src_info):
try:
data = json.load(open(src_info, "r", encoding="utf-8"))
except Exception:
data = {}
data["template_id"] = int(template_id)
with open(dst_info, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=4)
def merge_superglue_folders(target_dir, source_dirs):
"""Function 1: Merge multiple superglue template directories into target_dir, append them in order, and write template_id"""
if not _is_superglue_template_dir(target_dir):
raise RuntimeError(f"The target path is not a valid superglue template directory: {target_dir}")
for s in source_dirs:
if not _is_superglue_template_dir(s):
raise RuntimeError(f"The source path is not a valid superglue template directory: {s}")
# Write template_id=0 to existing templates in the target directory (if missing)
_ensure_template_id_in_dir(target_dir, 0)
next_idx = _get_next_index(target_dir)
# Assign template_id to subsequent directories starting from 1
for folder_tid, src in enumerate(source_dirs, start=1):
src_indices = _list_indices_by_json(src)
for i in src_indices:
_copy_one_template_block(src, i, target_dir, next_idx, folder_tid)
next_idx += 1
print(f"Merge completed, output at: {target_dir}")
def depth2img(
depth_map,
min_percentile=2,
max_percentile=98,
min_range=0.005):
"""Convert depth to rgb image.
Args:
depth_map (np.ndarray): The depth image.
Returns:
np.ndarray: The converted rgb image.
"""
valid_mask = np.isfinite(depth_map) & (depth_map > 0)
valid_depths = depth_map[valid_mask]
min_depth = np.percentile(valid_depths, min_percentile)
max_depth = np.percentile(valid_depths, max_percentile)
actual_range = max_depth - min_depth
# If the actual range is smaller than the minimum variation range, map all valid depths to a single color
if actual_range < min_range:
single_color_value = 0
gray_image = np.full_like(depth_map, single_color_value, dtype=np.uint8)
temp = cv2.applyColorMap(gray_image, cv2.COLORMAP_JET)
temp[~valid_mask] = 0
else:
depth_range = max(max_depth - min_depth, 1e-6)
normalized = np.clip((depth_map - min_depth) / depth_range, 0, 1)
gray_image = (normalized * 255).astype(np.uint8)
temp = cv2.applyColorMap(gray_image, cv2.COLORMAP_JET)
temp[~valid_mask] = 0
return temp
def _detect_group_dirs(root_dir):
"""Function 2: Detect whether multiple data groups exist under root_dir (each subdirectory is treated as one group)"""
group_dirs = []
for name in sorted(os.listdir(root_dir)):
p = os.path.join(root_dir, name)
if not os.path.isdir(p):
continue
# Must contain ply + any rgb + tiff + json(camera_param) at the same time
has_ply = any(fn.endswith(".ply") for fn in os.listdir(p))
has_img = any(fn.lower().endswith((".png", ".jpg", ".jpeg", ".bmp")) for fn in os.listdir(p))
has_tiff = any(fn.lower().endswith(".tiff") for fn in os.listdir(p))
has_json = False
for fn in os.listdir(p):
if fn.lower().endswith(".json"):
try:
d = json.load(open(os.path.join(p, fn), "r", encoding="utf-8"))
if "camera_param" in d:
has_json = True
break
except Exception:
pass
if has_ply and has_img and has_tiff and has_json:
group_dirs.append(p)
return group_dirs
def search_depth_values(indices, depth_mask, search_radius=3):
"""
Search for a nearby non-zero depth value and fill it into the depth mask
Args:
- indices: Index array of keypoints, shape (N, 2)
- depth_mask: Initial depth mask, shape (H, W)
- search_radius: Search radius, default is 3
Returns:
- depth_values: The depth value found for each keypoint
"""
depth_values = np.full(indices.shape[0], -1, dtype=np.float32)
for idx, (x, y) in enumerate(indices):
if depth_mask[y, x] == 0:
xmin, xmax = max(0, x - search_radius), min(depth_mask.shape[1], x + search_radius + 1)
ymin, ymax = max(0, y - search_radius), min(depth_mask.shape[0], y + search_radius + 1)
search_area = depth_mask[ymin:ymax, xmin:xmax]
non_zero = search_area[search_area != 0]
if non_zero.size > 0:
depth_values[idx] = non_zero[0]
else:
depth_values[idx] = depth_mask[y, x]
return depth_values
def rotate_pcd(pcd, angle, original_center=None):
"""Rotate the point cloud"""
if original_center is None:
original_center = pcd.get_center()
t_1 = np.array(
[[1, 0, 0, -original_center[0]], [0, 1, 0, -original_center[1]], [0, 0, 1, -original_center[2]], [0, 0, 0, 1]]
)
theta = np.radians(angle)
cos_theta = np.cos(theta)
sin_theta = np.sin(theta)
t_2 = np.array([[cos_theta, -sin_theta, 0, 0], [sin_theta, cos_theta, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
t_3 = np.array(
[[1, 0, 0, original_center[0]], [0, 1, 0, original_center[1]], [0, 0, 1, original_center[2]], [0, 0, 0, 1]]
)
transform_all = np.dot(t_3, np.dot(t_2, t_1))
pcd_final = deepcopy(pcd).transform(transform_all)
return pcd_final, transform_all
def project_pcd_to_rgb(
input_dir,
output_dir,
angle,
auto_scale,
mask_kernel_size,
edge_kernel_size,
index=1,
use_edge=False,
use_depth = False,
background_color=0,
foreground_color=255,
template_id=None,
base_index=0,
feat_type = "superpoint",
):
"""Project the point cloud to an RGB image (for SuperGlue templates)"""
os.makedirs(output_dir, exist_ok=True)
# Use base_index to read the reference template
cam_k_path = os.path.join(input_dir, f"model_info_{base_index}.json")
cam_k = json.load(open(cam_k_path))
cam_k = cam_k["camera_param"]
cam_k = np.asarray(cam_k).reshape(3, 3).astype(np.float32)
pcd_path = os.path.join(input_dir, f"model_{base_index}.ply")
pcd = o3d.io.read_point_cloud(pcd_path)
temp_img_path = os.path.join(input_dir, f"temp_{base_index}.png")
temp_img = cv2.imread(temp_img_path)
project_img = np.zeros(temp_img.shape, dtype=np.uint8)
project_depth = np.zeros(temp_img.shape[:2], dtype=np.float32)
aabb = pcd.get_axis_aligned_bounding_box()
aabb_center = (aabb.get_min_bound() + aabb.get_max_bound()) / 2
pcd_transformed, transform_all = rotate_pcd(pcd, angle, aabb_center)
o3d.io.write_point_cloud(os.path.join(output_dir, f"model_{index}.ply"), pcd_transformed)
f_json_data = open(os.path.join(output_dir, f"model_info_{index}.json"), "w+")
json_saved_data = {}
json_saved_data["camera_param"] = cam_k.tolist()
json_saved_data["transform"] = transform_all.tolist()
json_saved_data["angle"] = angle
json_saved_data["feat_type"] = feat_type
json_saved_data["mask_kernel_size"] = mask_kernel_size
json_saved_data["edge_kernel_size"] = edge_kernel_size
json_saved_data["use_edge"] = use_edge
json_saved_data["use_depth"] = use_depth
if template_id is not None:
json_saved_data["template_id"] = int(template_id)
json.dump(json_saved_data, f_json_data, indent=4)
f_json_data.close()
rvec = np.array([0.0, 0.0, 0.0])
tvec = np.array([0.0, 0.0, 0.0])
distortion_zeros = np.zeros((5, 1), dtype=np.float32)
pcd_np = np.array(pcd_transformed.points)
points_2d, _ = cv2.projectPoints(pcd_np, rvec, tvec, cam_k, distortion_zeros)
points_2d = points_2d.squeeze(1).reshape(-1, 2)
color_bgr = np.asarray(pcd_transformed.colors)[:, ::-1] * 255
for i, pt in enumerate(points_2d):
project_img[round(pt[1]), round(pt[0]), :] = color_bgr[i]
project_depth[round(pt[1]), round(pt[0])] = pcd_np[i][2]
project_img = cv2.morphologyEx(
project_img, cv2.MORPH_CLOSE, np.ones((mask_kernel_size, mask_kernel_size), np.uint8)
)
if use_edge:
mask = np.any(project_img != 0, axis=2)
project_img[project_img == 0] = background_color
project_img[mask] = foreground_color
if use_depth:
project_img = depth2img(project_depth)
project_img = cv2.morphologyEx(project_img, cv2.MORPH_CLOSE, np.ones((2, 2), np.uint8))
cv2.imwrite(os.path.join(output_dir, f"depth_model_{index}.tiff"), project_depth)
cv2.imwrite(os.path.join(output_dir, f"temp_{index}.png"), project_img)
concat_img = cv2.hconcat([project_img, temp_img])
cv2.imwrite(os.path.join(output_dir, f"project_img_concat_{index}.jpg"), concat_img)
# Output keypoints.ply
if ImagePairMatcher is not None and torch is not None:
config = default_config_superpoint if feat_type == "superpoint" else default_config_xfeat
if feat_type == "superpoint":
config['detector']['params']['default']['use_edge'] = use_edge
config['detector']['params']['default']['edge_kernel_size'] = edge_kernel_size
image_pair_matcher = ImagePairMatcher(config)
image_pair_matcher.register([project_img])
keypoints_model_2d = image_pair_matcher.kpts_info_t['keypoints'][0].cpu().numpy().astype(np.int32)
Keypoint_3D_model = np.zeros((len(keypoints_model_2d), 3), dtype=np.float32)
depth_values = search_depth_values(keypoints_model_2d, project_depth, 5)
def get_rainbow_color(index, total_points):
hsv = np.array([[[int(255 * index / total_points), 255, 255]]], dtype=np.uint8)
bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)[0][0]
return tuple(map(int, bgr))
temp_vis = deepcopy(project_img)
for i in range(len(keypoints_model_2d)):
x, y = keypoints_model_2d[i]
z = depth_values[i]
if z == -1:
Keypoint_3D_model[i] = [0, 0, 0]
continue
Keypoint_3D_model[i] = np.linalg.inv(cam_k) @ (np.array([x, y, 1]) * z)
color = get_rainbow_color(i, len(keypoints_model_2d))
center = (int(round(x)), int(round(y)))
cv2.circle(temp_vis, center, radius=1, color=color, thickness=-1)
cv2.imwrite(os.path.join(output_dir, f"temp_vis_{index}.jpg"), temp_vis)
rotated_model_kpts = o3d.geometry.PointCloud()
rotated_model_kpts.points = o3d.utility.Vector3dVector(Keypoint_3D_model)
rotated_model_kpts.colors = o3d.utility.Vector3dVector([[0, 1, 0] for _ in range(Keypoint_3D_model.shape[0])])
o3d.io.write_point_cloud(os.path.join(output_dir, f"keypoints_{index}.ply"), rotated_model_kpts)
if torch.cuda.is_available():
torch.cuda.empty_cache()
return True
def generate_ism_template(args):
"""Generate the ISM template (function of the first script)"""
data_dir = args.data_dir
rotation_range = args.rot_range
rotation_interval = args.rot_interval
rotation_range_xy = args.rot_range_xy
depth_range = args.depth_range
depth_interval = args.depth_interval
template_path = None
ply_path = None
pcd = None
# Search for required files
for f in os.listdir(data_dir):
template_path = os.path.join(data_dir, "ism")
os.makedirs(template_path, exist_ok=True)
f_path = os.path.join(data_dir, f)
if f.endswith('.ply'):
ply_path = f_path
pcd = o3d.io.read_point_cloud(ply_path)
pcd.paint_uniform_color([0, 1.0, 0])
o3d.io.write_point_cloud(f"{template_path}/model.ply", pcd)
if f.endswith('.png') or f.endswith('.jpg') or f.endswith('.bmp'):
rgb = cv2.imread(f_path)
if len(rgb.shape) == 2:
rgb = cv2.cvtColor(rgb, cv2.COLOR_GRAY2RGB)
h, w, _ = rgb.shape
if f.endswith('.tiff'):
depth = cv2.imread(f_path, -1)
if f.endswith('.json'):
resource_manager = FileOperation.load_json(f_path)
cam_k = resource_manager["camera_param"]
cam_k = np.asarray(cam_k).reshape(3, 3).astype(np.float32)
# Check whether the files exist
if ply_path is None:
raise ValueError(f"Error! No point cloud file was found under the path {data_dir}. Supported suffix: ply.\n")
if pcd is None:
raise ValueError(f"Error! Failed to read the point cloud file at path {ply_path}.\n")
try:
rgb
except NameError:
raise ValueError(f"Error! Failed to correctly read a color image file under the path {data_dir}. Supported suffixes: png/jpg/bmp.\n")
try:
depth
except NameError:
raise ValueError(f"Error! Failed to correctly read a depth image file under the path {data_dir}. Supported suffix: tiff.\n")
try:
cam_k
except NameError:
raise ValueError(f"Error! No configuration file was found under the path {data_dir}. Supported suffix: json.\n")
# ISM prompt
pcd_center = pcd.get_center()
angles_z = range(-rotation_range, rotation_range, rotation_interval)
mask = generate_mask_from_points(np.array(pcd.points), cam_k, h, w, auto_scale=1)
# ISM prompt
mask_uint8 = cv2.normalize(mask, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
rgb_mask = deepcopy(rgb)
rgb_mask[mask == 0] = 0
# Translate the foreground centroid of mask and rgb to the image center
coords = np.column_stack(np.where(mask_uint8 > 0))
if coords.size > 0:
cy, cx = coords.mean(axis=0)
center_x = w / 2.0
center_y = h / 2.0
dx = center_x - cx
dy = center_y - cy
T = np.float32([[1, 0, dx], [0, 1, dy]])
rgb_mask = cv2.warpAffine(
rgb_mask, T, (w, h), flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0)
)
mask_uint8 = cv2.warpAffine(
mask_uint8, T, (w, h), flags=cv2.INTER_NEAREST, borderMode=cv2.BORDER_CONSTANT, borderValue=0
)
if template_path is not None:
center = (w / 2, h / 2)
new_w = int(math.sqrt(w**2 + h**2))
new_h = new_w
main_i = 0
for angle in tqdm(angles_z, desc="Generating ISM rotation templates"):
M = cv2.getRotationMatrix2D(center, angle, 1.0)
M[0, 2] += (new_w - w) / 2
M[1, 2] += (new_h - h) / 2
# Generate rgb
rotated_rgb = cv2.warpAffine(
rgb_mask, M, (new_w, new_h), borderMode=cv2.BORDER_REPLICATE, borderValue=(127, 127, 127)
)
save_name_rgb = f"rgb_{main_i}.png"
save_path_rgb = os.path.join(template_path, save_name_rgb)
cv2.imwrite(save_path_rgb, rotated_rgb)
# Generate mask
rotated_mask = cv2.warpAffine(
mask_uint8, M, (new_w, new_h), borderMode=cv2.BORDER_REPLICATE, borderValue=(127, 127, 127)
)
save_name_mask = f"mask_{main_i}.png"
save_path_mask = os.path.join(template_path, save_name_mask)
cv2.imwrite(save_path_mask, rotated_mask)
main_i += 1
angles_xy = range(-rotation_range_xy, rotation_range_xy + 1, rotation_range_xy) if rotation_range_xy > 0 else [0]
# Set the depth variation range (mm to meters)
if depth_interval > 0:
depth_shifts = np.arange(-depth_range, depth_range + depth_interval, depth_interval) / 1000.0
else:
depth_shifts = [0]
geometric_features_model = []
variants_meta = []
i = 0
for angle_z in tqdm(angles_z, desc="Generating ISM geometric features"):
for angle_x in angles_xy:
for angle_y in angles_xy:
pcd_rotated = deepcopy(pcd)
# Z-axis rotation
Rz = pcd_rotated.get_rotation_matrix_from_xyz((0, 0, np.deg2rad(angle_z)))
pcd_rotated.rotate(Rz, center=pcd_center)
# Small-angle rotation around the X-axis
if angle_x != 0:
Rx = pcd_rotated.get_rotation_matrix_from_xyz((np.deg2rad(angle_x), 0, 0))
pcd_rotated.rotate(Rx, center=pcd_center)
# Small-angle rotation around the Y-axis
if angle_y != 0:
Ry = pcd_rotated.get_rotation_matrix_from_xyz((0, np.deg2rad(angle_y), 0))
pcd_rotated.rotate(Ry, center=pcd_center)
for depth_shift in depth_shifts:
pcd_final = deepcopy(pcd_rotated)
if depth_shift != 0:
pcd_final.translate((0, 0, depth_shift))
# Generate the mask from the rotated and translated point cloud
mask = generate_mask_from_points(np.array(pcd_final.points), cam_k, h, w, auto_scale=2)
mask_uint8 = cv2.normalize(mask, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
obb_ratio = get_ratio_from_mask(mask_uint8)
mask_area = int(np.count_nonzero(mask_uint8 > 0))
if obb_ratio > 0 and mask_area > 0:
geometric_features_model.append([float(obb_ratio), float(mask_area)])
variants_meta.append(
{
"idx": i,
"angle_z": angle_z,
"angle_x": angle_x,
"angle_y": angle_y,
"depth_shift_m": float(depth_shift),
"obb_ratio": float(obb_ratio),
"mask_area": mask_area,
}
)
i += 1
meta = {"geometric_features_model": geometric_features_model, "variants": variants_meta, "num_variants": i}
with open(os.path.join(template_path, "prompt_meta.json"), "w", encoding="utf-8") as f:
json.dump(meta, f, ensure_ascii=False, indent=2)
print(f"ISM template generation completed: metadata written to {template_path}/prompt_meta.json")
return template_path
def generate_superglue_template(args, start_index=0, template_id=None):
"""Generate the SuperGlue template (function of the second script)"""
if ImagePairMatcher is None:
raise ImportError("Unable to import SuperGlueMatcher. Please ensure the glia version is correct")
if torch is None:
raise ImportError("Unable to import torch. Please install PyTorch")
data_dir = args.data_dir
auto_scale = args.auto_scale
mask_kernel_size = args.mask_kernel_size
edge_kernel_size = args.edge_kernel_size
background_color = args.background_color
foreground_color = args.foreground_color
voxel_size = args.down_sample
ply_path = None
pcd = None
# Search for required files
for f in os.listdir(data_dir):
f_path = os.path.join(data_dir, f)
if f.endswith('.ply'):
ply_path = f_path
pcd = o3d.io.read_point_cloud(ply_path)
if f.endswith('.png') or f.endswith('.jpg') or f.endswith('.bmp'):
rgb = cv2.imread(f_path)
h, w = rgb.shape[0], rgb.shape[1]
if f.endswith('.tiff'):
depth = cv2.imread(f_path, -1)
if f.endswith('.json') and 'camera_param' in f:
resource_manager = json.load(open(f_path))
cam_k = resource_manager["camera_param"]
cam_k = np.asarray(cam_k).reshape(3, 3).astype(np.float32)
# Check whether the files exist
if ply_path is None:
raise ValueError(f"Error! No point cloud file was found under the path {data_dir}. Supported suffix: ply.\n")
if pcd is None:
raise ValueError(f"Error! Failed to read the point cloud file at path {ply_path}.\n")
try:
rgb
except NameError:
raise ValueError(f"Error! Failed to correctly read a color image file under the path {data_dir}. Supported suffixes: png/jpg/bmp.\n")
try:
depth
except NameError:
raise ValueError(f"Error! Failed to correctly read a depth image file under the path {data_dir}. Supported suffix: tiff.\n")
try:
cam_k
except NameError:
raise ValueError(f"Error! No configuration file was found under the path {data_dir}. Supported suffix: json.\n")
mask = generate_mask_from_points(
np.array(pcd.points), cam_k, h, w, kernel_size=mask_kernel_size, auto_scale=auto_scale
)
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, np.ones((mask_kernel_size, mask_kernel_size), np.uint8))
rgb_mask = deepcopy(rgb)
rgb_mask[mask == 0] = background_color
depth_mask = deepcopy(depth)
depth_mask[mask == 0] = 0
if args.use_edge:
rgb_mask[mask != 0] = foreground_color
# Crop the image
coords = cv2.findNonZero(mask)
if coords is None or len(coords) == 0:
raise ValueError("There are no non-zero points in the mask, so cropping cannot be performed")
# Get the initial bounding box
x_bbox, y_bbox, w_bbox, h_bbox = cv2.boundingRect(coords)
# Use a circle with the diagonal as the diameter, then compute the circumscribed square to prevent the rotated template from going out of frame
diameter = math.ceil(np.sqrt(w_bbox*w_bbox + h_bbox*h_bbox))
rgb_mask_paded = np.zeros([diameter, diameter, 3], dtype=np.uint8)
depth_mask_paded = np.zeros([diameter, diameter], dtype=np.float32)
x_pad, y_pad = (diameter - w_bbox)//2, (diameter - h_bbox)//2
rgb_mask_paded[y_pad:y_pad + h_bbox, x_pad:x_pad + w_bbox] = rgb_mask[y_bbox : y_bbox + h_bbox, x_bbox : x_bbox + w_bbox]
depth_mask_paded[y_pad:y_pad + h_bbox, x_pad:x_pad + w_bbox] = depth_mask[y_bbox : y_bbox + h_bbox, x_bbox : x_bbox + w_bbox]
rgb_mask = deepcopy(rgb_mask_paded)
depth_mask = deepcopy(depth_mask_paded)
# Update the camera intrinsic matrix
cam_k[0, 2] = cam_k[0, 2] - x_bbox + x_pad # Adjust cx (principal point x-coordinate); the principal point moved by x_expanded in the x direction, so subtract it
cam_k[1, 2] = cam_k[1, 2] - y_bbox + y_pad # Adjust cy (principal point y-coordinate); the principal point moved by y_expanded in the y direction, so subtract it
# Unified output directory: prefer args.output_feat_dir, otherwise data_dir/feature matching
superglue_path = getattr(args, "output_feat_dir", None) or os.path.join(data_dir, "feature matching")
os.makedirs(superglue_path, exist_ok=True)
index = int(start_index)
base_index = index
use_depth = args.use_depth
use_edge = args.use_edge
# Output model.ply
o3d.io.write_point_cloud(f"{superglue_path}/model_{index}.ply", pcd)
if voxel_size is not None:
pcd_down_sampled = deepcopy(pcd).voxel_down_sample(voxel_size=voxel_size)
o3d.io.write_point_cloud(f"{superglue_path}/model_{index}_downsampled.ply", pcd_down_sampled)
# Register the template and output temp.png and keypoints.ply
temp = deepcopy(rgb_mask)
if use_depth:
temp = depth2img(depth_mask)
temp = cv2.morphologyEx(temp, cv2.MORPH_CLOSE, np.ones((2, 2), np.uint8))
cv2.imwrite(f"{superglue_path}/depth_model_{index}.tiff", depth_mask)
cv2.imwrite(f"{superglue_path}/temp_{index}.png", temp)
# Output keypoints.ply
config = default_config_superpoint if args.feat_type == "superpoint" else default_config_xfeat
if args.feat_type == "superpoint":
config['detector']['params']['default']['use_edge'] = use_edge
config['detector']['params']['default']['edge_kernel_size'] = edge_kernel_size
image_pair_matcher = ImagePairMatcher(config)
image_pair_matcher.register([temp])
keypoints_model_2d = image_pair_matcher.kpts_info_t['keypoints'][0].cpu().numpy().astype(np.int32)
Keypoint_3D_model = np.zeros((len(keypoints_model_2d), 3), dtype=np.float32)
depth_values = search_depth_values(keypoints_model_2d, depth_mask, 5)
temp_vis = deepcopy(temp)
def get_rainbow_color(index, total_points):
hsv = np.array([[[int(255 * index / total_points), 255, 255]]], dtype=np.uint8)
bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)[0][0]
return tuple(map(int, bgr))
for i in range(len(keypoints_model_2d)):
x, y = keypoints_model_2d[i]
z = depth_values[i]
if z == -1:
Keypoint_3D_model[i] = [0, 0, 0]
continue
Keypoint_3D_model[i] = np.linalg.inv(cam_k) @ (np.array([x, y, 1]) * z)
color = get_rainbow_color(i, len(keypoints_model_2d))
center = (int(round(x)), int(round(y)))
cv2.circle(temp_vis, center, radius=1, color=color, thickness=-1)
cv2.imwrite(f"{superglue_path}/temp_vis_{index}.jpg", temp_vis)
pcd_model_keypoints = o3d.geometry.PointCloud()
pcd_model_keypoints.points = o3d.utility.Vector3dVector(Keypoint_3D_model)
pcd_model_keypoints.colors = o3d.utility.Vector3dVector([[0, 1, 0] for _ in range(Keypoint_3D_model.shape[0])])
o3d.io.write_point_cloud(f"{superglue_path}/keypoints_{index}.ply", pcd_model_keypoints)
f_json_data = open(os.path.join(superglue_path, f"model_info_{index}.json"), "w+")
json_saved_data = {}
json_saved_data["camera_param"] = cam_k.tolist()
json_saved_data["transform"] = np.eye(4).tolist()
json_saved_data["angle"] = 0.0
json_saved_data["feat_type"] = args.feat_type
json_saved_data["mask_kernel_size"] = mask_kernel_size
json_saved_data["edge_kernel_size"] = edge_kernel_size
json_saved_data["use_edge"] = args.use_edge
json_saved_data["use_depth"] = args.use_depth
json_saved_data["downsample_size"] = args.down_sample
if template_id is not None:
json_saved_data["template_id"] = int(template_id)
json.dump(json_saved_data, f_json_data, indent=4)
f_json_data.close()
if args.multi_temp:
for k, angle in enumerate(args.angle):
project_pcd_to_rgb(
superglue_path,
superglue_path,
angle,
args.auto_scale,
mask_kernel_size,
edge_kernel_size,
index=base_index + 1 + k,
use_edge=args.use_edge,
use_depth=args.use_depth,
background_color=background_color,
foreground_color=foreground_color,
template_id=template_id,
base_index=base_index,
feat_type=args.feat_type,
)
# Save the point cloud of template 0 in the group as multi_model_{template_id}.ply (if multi-template is enabled)
if template_id is not None:
src = os.path.join(superglue_path, f"model_{base_index}.ply")
dst = os.path.join(superglue_path, f"multi_model_{template_id}.ply")
if os.path.exists(src):
shutil.copy(src, dst)
if voxel_size is not None:
# Generate multi_model_{template_id}_downsampled.ply
pcd0 = o3d.io.read_point_cloud(src)
pcd0_ds = deepcopy(pcd0).voxel_down_sample(voxel_size=voxel_size)
o3d.io.write_point_cloud(
os.path.join(superglue_path, f"multi_model_{template_id}_downsampled.ply"), pcd0_ds
)
if torch.cuda.is_available():
torch.cuda.empty_cache()
print(f"SuperGlue template generation completed: files saved in {superglue_path}")
# Return the next available index (calculated using base_index)
end_next_index = base_index + (len(args.angle) + 1 if args.multi_temp else 1)
return superglue_path, end_next_index
def generate_template(args=None):
"""
Main function for generating templates, supporting standalone execution and external calls
Args:
args: Parameters of type argparse.Namespace or dict
If None, parse parameters from the command line
If dict, convert it to Namespace
Returns:
tuple: (ism_path, feat_path) or a single path depending on the parameters
"""
if args is None:
# Command-line invocation mode
parser = create_parser()
args = parser.parse_args()
elif isinstance(args, dict):
# External call, parameters are provided as a dictionary
parser = create_parser()
args_ori = parser.parse_args()
for key, value in args.items():
if hasattr(args_ori, key):
setattr(args_ori, key, value)
args = args_ori
# Perform file copying
if args.file_transfer:
if not args.output_dir:
raise ValueError("--output_dir must be specified when performing file copying")
print("Starting file copying...")
file_transfer(args)
print(f"File copying completed, files saved in: {args.output_dir}")
return
# Generate ISM
arg_ism = deepcopy(args)
if args.type in ["ism", "both"]:
def _first_subdir(root: str) -> str:
try:
subs = sorted([e.path for e in os.scandir(root) if e.is_dir()])
return subs[0] if subs else root
except Exception:
return root
if getattr(args, "multi_template", False):
arg_ism.data_dir = _first_subdir(arg_ism.data_dir)
print("Starting ISM template generation...")
try:
ism_path = generate_ism_template(arg_ism)
print(f"ISM template generation completed, saved in: {ism_path}")
except Exception as e:
print(f"Error generating the ISM template: {e}")
if arg_ism.type == "ism":
raise
# Generate/process SuperGlue
if args.type in ["feat", "both"]:
print("Starting SuperGlue template generation/processing...")
try:
if args.multi_template:
# Function 1: merge multiple superglue directories
if len(args.feat_dirs) >= 2:
target_dir = args.feat_dirs[0]
source_dirs = args.feat_dirs[1:]
merge_superglue_folders(target_dir, source_dirs)
print(f"SuperGlue multi-directory merge completed, output: {target_dir}")
else:
# Function 2: multiple groups in a single directory -> write them uniformly into one directory and number them consecutively from the end
group_dirs = _detect_group_dirs(args.data_dir)
out_dir = args.output_feat_dir or os.path.join(args.data_dir, "feature matching")
os.makedirs(out_dir, exist_ok=True)
start_idx = _get_next_index(out_dir)
next_idx = start_idx
if len(group_dirs) >= 1:
for gid, gdir in enumerate(group_dirs):
args_one = deepcopy(args)
args_one.data_dir = gdir
args_one.output_feat_dir = out_dir # Unified output directory
_, next_idx = generate_superglue_template(args_one, start_index=next_idx, template_id=gid)
print(f"SuperGlue multi-group generation completed, output: {out_dir}")
else:
# Regular single-group generation, but write template_id=0; unified output directory is also supported
_, _ = generate_superglue_template(
args,
start_index=next_idx,
template_id=0,
)
print(f"SuperGlue single-group generation completed, output: {out_dir}")
args.data_dir = out_dir
else:
# Original logic: single-group generation (without writing the template_id field)
_ = generate_superglue_template(args) # Return value unused
print("SuperGlue template generation completed")
except Exception as e:
print(f"Error generating/processing the SuperGlue template: {e}")
if args.type == "feat":
raise
if args.type == "both":
print("All template generation is completed!")
print(f"ISM templates are saved in: {os.path.join(arg_ism.data_dir, 'ism')}")
print(f"Feat templates are saved in: {os.path.join(args.data_dir, 'feature matching')}")
def create_parser():
"""Create the argument parser"""
parser = argparse.ArgumentParser(description="Merged script: generate ISM and/or Feat templates")
parser.add_argument(
"--output_feat_dir",
type=str,
default=None,
help="Specify the unified output directory for Feat templates; if not specified, write to data_dir/feature matching",
)
# Basic parameters
parser.add_argument("--data_dir", type=str, default=None, help="Input data directory")
parser.add_argument(
"--type",
choices=["ism", "feat", "both"],
default="both",
help="Select the template type to generate: ism, feat, or both",
)
parser.add_argument("--file_transfer", action="store_true", default=False, help="Perform file copying")
parser.add_argument("--output_dir", help="Output directory when copying files")
# ISM parameters
parser.add_argument("--rot_range", type=int, default=60, help="Z-axis rotation range (degrees)")
parser.add_argument("--rot_interval", type=int, default=10, help="Z-axis rotation interval (degrees)")
parser.add_argument("--rot_range_xy", type=int, default=5, help="XY small rotation range (degrees)")
parser.add_argument("--depth_range", type=int, default=200, help="Z translation range (mm)")
parser.add_argument("--depth_interval", type=int, default=200, help="Z translation interval (mm)")
# SuperGlue parameters
parser.add_argument("--auto_scale", type=float, default=0.1, help="Scaling ratio from point cloud to mask")
parser.add_argument("--use_depth", action="store_true", default=False, help="Use depth images")
parser.add_argument("--use_edge", action="store_true", default=False, help="Use edge mode")
parser.add_argument("--feat_type", choices=["superpoint", "xfeat"], default="superpoint", help="Feature point extractor type")
parser.add_argument("--edge_kernel_size", type=int, default=3, help="Edge kernel size")
parser.add_argument("--mask_kernel_size", type=int, default=5, help="Morphological operation kernel size")
parser.add_argument("--multi_temp", action="store_true", default=False, help="Generate multi-angle templates")
parser.add_argument("--angle", type=float, nargs="+", default=[90, 180, 270], help="Rotation angles")
parser.add_argument("--down_sample", type=float, default=None, help="Point cloud downsampling voxel size (meters); set to None to disable")
parser.add_argument("--background_color", type=int, default=0, help="Edge mode background color")
parser.add_argument("--foreground_color", type=int, default=255, help="Edge mode foreground color")
# New: multi-template mode
parser.add_argument("--multi_template", action="store_true", default=False, help="Enable SuperGlue multi-template mode")
parser.add_argument(
"--feat_dirs",
nargs="+",
default=[],
help="Multiple superglue template directories (Function 1: merge multiple generated feat template directories)",
)
return parser
def main():
generate_template()
if __name__ == "__main__":
main()（3）利用模板生成脚本，复制历史数据
- 在模板生成脚本的下载目录中，鼠标右键点击空白处，出现"右键菜单"，在"右键菜单"中点击
Open in Terminal，打开Windows PowerShell终端，如下图所示。
- 在终端执行
conda activate pickwiz_py39命令进入到 pickwiz_py39 环境，如下图所示。
- 终端进入到 pickwiz_py39 环境后，继续执行以下命令，可根据模板生成脚本的名称、要复制的历史数据时间戳的路径、输出保存路径修改以下命令。
python generate_prompt_double_ism_feature.py # Call the Python script; you can modify it according to the template generation script name
--data_dir "C:\Users\dex\kuawei_data\PickLight\20240617150557809" # Path of the historical data timestamp to copy; you can modify it according to the historical data timestamp
--file_transfer --output_dir # File transfer and output command
"C:\Users\dex\Documents\PickWiz\new_project_22\superglue" # Output file save path; you can modify the save path as needed
--type feat # Indicates generating a feature matching template示例：模板生成脚本的名称是 "generate_prompt_superglue.py"；
要复制的历史数据的时间戳路径是 "D:\Pickwiz\new_project\data\PickLight\20250411144909289"；
输出文件保存路径是"C:\Users\dex\Documents\PickWiz\new_project_1\data\PickLight"。
python generate_prompt_double_ism_feature.py # Call the Python script "generate_prompt_superglue.py"
--data_dir "D:\Pickwiz\new_project\data\PickLight\20250411144909289" # Path of the historical data timestamp to copy
--file_transfer --output_dir # File transfer and output command
"C:\Users\dex\Documents\PickWiz\new_project_1\data\PickLight" # Output file save path
--type feat # Indicates generating a feature matching template执行命令时修改脚本名称、历史数据时间戳路径、输出文件保存路径，如下图所示。
（4）执行完命令后，在保存路径下生成了4个文件，分别是场景的2D图像、场景的深度图、场景点云和相机内参
- 裁剪点云
在 meshlab 软件打开场景点云.ply文件，裁剪掉场景点云的噪点直至仅保留工件点云，然后直接点击覆盖保存。
裁剪点云时需要仔细保留完整的工件点云。
| 裁剪前 | 裁剪后 | 覆盖保存 |
|---|---|---|
 |  |  |
生成点云模板
在终端执行
conda activate pickwiz_py39命令进入到 pickwiz_py39 环境。
- 在 pickwiz_py39 环境下，执行如下命令，可根据工件特征修改以下命令。
脚本提供了--use_edge 、--multi_temp 参数，用于调整模板生成方式。 --use_edge 表示使用边缘检测,--multi_temp表示生成多方向模版用于来料方向不固定的场景。这两个参数默认不增加，表示不启用多模版和边缘检测，使用2D图生成点云模板。
当工件表面纹理不明显时，增加 --use_edge 参数，启用边缘检测强化图像特征，生成边缘增强的点云模板。
python generate_prompt_double_ism_feature.py # Call the Python script
--data_dir "C:\Users\dex\Documents\lixin\unify_infer\superglue_model_gen\superglue" # Input the save path from step 3 above
--type feat # Indicates generating a feature matching template示例：当实际场景光照条件不稳定或工件表面纹理不明显、几何形状复杂时，可增加 --use_edge 参数，脚本会先对2D图像进行边缘检测，替代原本的2D图像进行匹配，匹配时会专注于工件的几何边缘特征，生成点云模板。
python generate_prompt_double_ism_feature.py # Call the Python script
--data_dir "C:\Users\dex\Documents\PickWiz\new_project_1\data\PickLight" --use_edge # Input the save path from step 3 above and add the --use_edge edge detection parameter
--type feat # Indicates generating a feature matching template
- 执行指令后，在保存路径下生成了模板文件夹"feature matching"。
检查该文件夹下是否存在以下4个文件。
（注意灰度模式下模板图为灰度图，边缘模式下为二值图）
| 灰度模式 |  |
|---|---|
| 边缘模式 |  |
- 若来料包含多个方向，启用多模版方式的示例如下，其中
--angle后指定相对主模版的旋转角度，运行后会生成根据主模版旋转若干个角度的模版，如下所示
python generate_prompt_double_ism_feature.py --data_dir superglue-compressor --type feat --use_edge --multi_temp --angle 45 90 180 225 270如下图所示
导入点云模板、选择模板文件路径
将保存路径下 "feature matching"文件夹中的
model.ply文件作为点云模板导入到工件界面的Point Cloud File中。
- 在
Template File Path选择保存路径下“feature matching”文件夹的路径（如C:\Users\dex\Documents\PickWiz\new_project_1\data\PickLight\feature matching）。
- 模版生成脚本参数说明
| 参数名称 | 参数说明 | 取值建议 | 简单说明 |
|---|---|---|---|
| data_dir | 先验文件夹路径 | \ | 与先前生成数据脚本使用的文件路径一致 windows复制路径为单反斜杠 "\"，报路径错误，需要将单反斜杠改为双反斜杠或改为取值建议中的正斜杠 "/" |
| file_transfer | 是否拷贝文件 | \ | \ |
| output_dir | 仅在 file_transfer 有效时起作用 | \ | \ |
| scale | 点云投影生成mask的缩放尺度 | 0.1 | 点云投影生成mask的缩放尺度 |
| use_edge | 是否启用边缘模式，不启用则为灰度模式 | \ | 多层堆叠时，下层物体与上层物体轮廓难以被模型区分，需使用灰度模式 |
| use_depth | 是否使用深度图模板 | \ | 开启后，会从深度图转彩图产生模板图，物体表面无明显纹理，且点云/深度图质量较好时可切换深度图模式 |
| multi_temp | 是否生成多方向模版 | \ | 默认为false，即默认仅生成单方向模版 |
| mask_kernel_size | 点云投影成mask时的闭运算卷积核大小 | 5 | 使用默认值 |
| edge_kernel_size | 边缘特征提取时的卷积核大小 | 3 |  数值越大，特征图越靠内 |
| angle | 多模版时，子模版相对主模版的旋转角度 | \ | 角度输入格式，可参考示例命令 |
| down_sample | 点云降采样倍数，降采样可降低时间节拍，但可能影响配准精度，需要验证后使用 | 0.001 | 表示点云体素降采样过程，点与点间隔大小参数 |
2.2.2 匹配置信阈值(mm)
- 功能
特征点匹配的可信度分数，分数越高，特征点质量越高，但是匹配到的特征点数量可能越少
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
默认值：0.01
取值范围：[0.001, 1.0]
2.2.3 特征扩增数量
- **功能 **
在原始特征点检测的基础上人工扩增特征点数量，防止因特征点过少导致匹配异常。
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
默认值：3
取值范围：[0, 99]
2.2.4 特征扩增范围
- **功能 **
特征点扩增的选取邻域范围
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
默认值：3
取值范围：[0, 99]
- 调参：
当场景点云质量较好时可适当调大该参数，反之场景点云质量差时调小该参数
2.2.7 最大迭代次数
- **功能 **
限制算法在粗匹配阶段的最大迭代次数，避免因无限循环或收敛过慢导致计算资源浪费
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
默认值：100
不建议修改，后续会隐藏该函数
2.2.8 包围盒尺寸系数
- **功能 **
动态调整包围盒的尺寸，控制检测到的包围盒的长宽缩放比例。
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
默认值：1.0
- 调参
减少背景干扰或相邻目标的误合并，缩小检测范围，系数 <1.0；
防止目标部分（如遮挡）被裁剪，放大检测范围，系数 >1.0；
可根据2D识别结果中的包围盒情况调整该系数
2.2.9 启用深度特征
- **功能 **
通过 SuperGlue 模型提取的特征替代传统的点云特征，用于解决复杂场景下的匹配异常。
- 使用场景
面型工件有序上下料（并行化）场景
- 调参
适用于表面光滑、重复纹理、光照变化大的工件，适用于多类工件混合上下料
2.2.10 启用边缘特征
- **功能 **
启用时，仅在物体边缘区域提取和匹配特征点
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
物体边缘区域（如轮廓、锐利过渡区），忽略平坦或纹理单一的区域。
模板制作时也需同步启用use_edge，保证特征一致性。
2.2.11 粗匹配评判阈值(mm)
- **功能 **
特征点粗匹配时，仅保留可信度高于此值的匹配点。
- 使用场景
面型工件有序上下料（并行化）场景
- 参数说明
默认值：10
取值范围：[0.1, 1000]
- 调参
低阈值（如5）：减少误匹配，但可能丢失有效点。
高阈值（如20）：保留更多匹配，但噪声增加。
2.2.12 物体姿态修正
精匹配搜索半径(mm)
- 功能
在精匹配过程中，模板点云与实例点云进行匹配，模板点云中每个点都需要在实例点云中搜索最邻近点。精匹配搜索半径既表示在实例点云中的搜索半径，又表示模板点云中的每个点与实例点云中的最邻近点的距离阈值。若点与最邻近点的距离小于精匹配搜索半径，则认为这两个点能够匹配，否则认为两个点不能匹配。
- 使用场景
面型工件有序上下料、面型工件无序抓取、面型工件定位装配场景
- 参数说明
默认值：10
取值范围：[1, 500]
单位：mm
- 调参
通常不更改
精匹配搜索模式
- 功能
在精匹配过程中，模板点云在实例点云中检索最邻近点的方式
- 使用场景
模板点云与实例点云的精匹配效果不佳，应调整该函数
- 参数说明
| 参数 | 说明 |
|---|---|
| 点到点 | 模板点云中的每个点在实例点云中搜索最邻近点（搜索半径内直线距离最短的点）适用于所有工件 |
| 点到面 | 模板点云中的每个点沿着其法向量在实例点云中搜索最邻近点适用于几何特征明显的工件 |
| 点到点和点到面两种方式结合 | 先采用点对点模式优化实例点云中的工件姿态，再采用点对面模式优化实例点云中的工件姿态适用于几何特征明显的工件
|
使用轮廓模式
- 功能
提取模板点云和实例点云中的轮廓点云进行粗匹配
- 使用场景
面型工件有序上下料、面型工件无序抓取、面型工件定位装配场景，若使用关键点进行粗匹配的结果不佳，应当勾选该函数使用轮廓点云再次进行粗匹配
- 调参
粗匹配的结果会影响精匹配结果，如果精匹配结果不佳，可勾选 使用轮廓模式
轮廓搜索范围(mm)
- 功能
在模板点云和实例点云中提取轮廓点云的搜索半径
- 使用场景
通用工件有序上下料、通用工件无序抓取、通用工件定位装配场景
- 参数说明
默认值：5
取值范围：[0.1, 500]
单位：mm
- 调参
取值较小，搜索轮廓点云的半径较小，适合提取细致的工件轮廓，但提取的轮廓可能包含离群点噪声；
取值较大，搜索轮廓点云的半径较大，适合提取较宽的工件轮廓，但提取的轮廓可能会忽略一些细节特征。
保存姿态估计[精匹配]数据
- 功能
勾选则保存精匹配数据
- 使用场景
面型工件有序上下料、面型工件无序抓取、面型工件定位装配、面型工件定位装配（仅匹配）
- 示例
精匹配数据保存在项目保存路径\Project Folder\data\PickLight\Historical Data Timestamp\Builder\pose\output folder。
2.3 空ROI判断
- 功能
判断ROI 3D内是否还有工件（点云）剩余，如果ROI 3D内的3D点的数量小于该值，表示没有工件点云剩余，此时不返回点云
- 参数说明
默认值：1000
取值范围：[0, 100000]
- 使用流程
设置ROI 3D最小点数判断阈值，小于该阈值即ROI 3D中工件点云不足，从而判断为无工件在ROI 3D中；
机器人配置中，新增视觉状态码，便于后续机器人进行信号处理。
3. Pick Point Processing
This section mainly explains functions related to Pick Point filtering and adjustment, along with Parameter tuning recommendations.
3.1 Pick Point Adjustment
3.1.1 Rotate the Picking Pose when it is outside the angle range
- Function description
When the Picking Pose is outside the configured angle range, it is rotated counterclockwise by a certain angle around a fixed axis. If it is still outside the configured angle range after rotation, a warning is issued.
- Usage scenario
This function is only applicable to depalletizing scenarios. It can keep the robot's approach direction stable during picking and prevent the end effector from repeatedly rotating during the picking process. In 180° cases, it can prevent exceptions such as cable twisting.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Fixed axis | An axis of the Picking Pose. The pose is rotated counterclockwise around this fixed axis | Z-axis | X/Y/Z-axis | / |
| Rotation angle | The angle by which the pose is rotated counterclockwise around the fixed axis. Adjust this angle so the Picking Pose satisfies the angle range | 0 | [-360,360] | degree |
| Angle range | The angle range of the Picking Pose. Set the angle range according to factors such as material placement, end effector type, and cycle time | [0,180] | [-180,180] | degree |
| Use current robot Euler Angles | By default, pose calculation uses Euler Angles "XYZ". When selected, the Euler Angles configured for the current robot are used so the pose remains consistent with the robot teach pendant. | Unchecked | / | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Robot arm coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system | / |
- Example
Without using this function, the generated Pick Points are shown below.
When this function is used with the default values, the RZ angles of the Picking Poses for instances 0, 1, and 2 are all within the angle range [0,180], so no processing is performed. The RZ angle of the Picking Pose for instance 4 is -90°, which is outside the angle range [0,180], so the Picking Pose of instance 4 is rotated by 0° around the fixed Z-axis.
If you want to adjust the RZ angle of the Picking Pose for instance 4 into the angle range, you can change the rotation angle to 180 and rotate the Picking Pose of instance 4 by 180° around the fixed Z-axis.
3.1.2 Rotate the Picking Pose so the rotation axis direction matches the target axis direction
- Function description
Rotate the Picking Pose once around the fixed axis so that the direction of the rotation axis (determined by the right-hand rule) matches the positive or negative direction of the target axis in the target coordinate system.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Rotation axis | An axis of the Picking Pose. Determined by the right-hand rule, the Picking Pose is rotated counterclockwise once around the fixed axis so that the direction of the rotation axis matches the positive or negative direction of the target axis in the target coordinate system | X-axis | X/Y/Z-axis |
| Fixed axis | The Picking Pose is rotated counterclockwise once around the fixed axis so that the direction of the rotation axis matches the positive or negative direction of the target axis in the target coordinate system | Z-axis | X/Y/Z-axis |
| Target axis | An axis of the target coordinate system. The Picking Pose is rotated counterclockwise once around the fixed axis so that the direction of the rotation axis matches the positive or negative direction of the target axis in the target coordinate system | X-axis | X/Y/Z-axis |
| Negative target axis direction | If selected, the direction of the rotation axis is aligned with the negative direction of the target axis in the target coordinate system; otherwise, it is aligned with the positive direction of the target axis in the target coordinate system | Unchecked | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Default coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system |
- Example
3.1.3 Rotate the Picking Pose so the angle between the rotation axis and the target axis is minimized
- Function description
Rotate the Picking Pose around the fixed axis by 0, 90, 180, and 270 degrees respectively, calculate the angle between the rotated rotation axis and the positive or negative direction of the target axis in the camera coordinate system, and finally output the Picking Pose with the smallest angle after rotation.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Fixed axis | An axis of the Picking Pose. Rotate the pose counterclockwise around this fixed axis | Z-axis | X/Y/Z-axis |
| Rotation axis | An axis of the Picking Pose. When rotating the pose, calculate the angle between this rotation axis and the positive or negative direction of the target axis | X-axis | X/Y/Z-axis |
| Target axis | An axis of the camera coordinate system. When rotating the pose, calculate the angle between the rotation axis and the positive or negative direction of this target axis | X-axis | X/Y/Z-axis |
| Negative target axis direction | If selected, calculate the angle between the rotation axis and the negative direction of the target axis; otherwise, calculate the angle between the rotation axis and the positive direction of the target axis | Selected | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Default coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system |
- Example
3.1.4 Flip the Picking Pose so the angle between the rotation axis and the target axis is minimized
- Function description
Rotate the Picking Pose once around the fixed axis so that the angle formed between the rotation axis and the positive or negative direction of the target axis in the ROI coordinate system is acute.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Fixed axis | An axis of the Picking Pose. Rotate the Picking Pose counterclockwise around this fixed axis | Z-axis | X/Y/Z-axis |
| Rotation axis | An axis of the Picking Pose. Rotate the Picking Pose so that the direction of this rotation axis matches the positive or negative direction of the target axis | X-axis | X/Y/Z-axis |
| Target axis | An axis in the ROI coordinate system. Rotate the Picking Pose so that the direction of the rotation axis matches the positive or negative direction of this target axis | X-axis | X/Y/Z-axis |
| Negative target axis direction | If selected, rotate the Picking Pose so that the direction of the rotation axis matches the negative direction of the target axis; otherwise, rotate the Picking Pose so that the direction of the rotation axis matches the positive direction of the target axis | Selected | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Default coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system |
- Example
3.1.5 Point a Picking Pose axis toward the ROI center
- Function
Rotate the Picking Pose around a fixed axis so that the pointing axis of the Picking Pose points to the ROI center.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Pointing axis | The axis in the Picking Pose that needs to be adjusted | X-axis | X/Y/Z-axis |
| Fixed axis | The axis that remains unchanged during rotation | Z-axis | X/Y/Z-axis |
| Reverse align | If selected, reverse-align the pointing axis to the ROI center; otherwise, align the pointing axis to the ROI center | Selected | / |
| Strict pointing | If selected, force the Picking Pose to rotate so the pointing axis points to the ROI center | Unchecked | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Default coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system |
- Example
3.1.6 Rotate the Picking Pose so the Z-axis direction matches the Z-axis of the target coordinate system
- Function description
Rotate the Picking Pose so that its Z-axis direction matches the Z-axis of the target coordinate system.
- Usage scenario
Usually this is used by default only in depalletizing scenarios and cannot be deleted. It is used to make the Z-axis of the Picking Pose perpendicular to the Z-axis of the ROI coordinate system (4-axis) or consistent with the direction of the Target Object surface (6-axis).
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Robot configuration | Set according to the on-site robot configuration. You can choose 4-axis or 6-axis. If a 6-axis robot is actually used as a 4-axis robot, it should be set to 4-axis | 4-axis | 4-axis/6-axis |
| Use ROI Z-axis as target direction | When the robot configuration is set to 4-axis, if selected, the pose is rotated around the X-axis so that the Z-axis direction of the rotated pose matches the positive direction of the ROI Z-axis ; if not selected, the pose is rotated around the X-axis so that the Z-axis direction of the rotated pose matches the positive direction of the Z-axis of the camera coordinate system . When the robot configuration is set to 6-axis, regardless of whether it is selected, the pose is rotated around the X-axis so that the Z-axis direction of the rotated pose matches the positive direction of the Z-axis of the object's own coordinate system | Unchecked | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Camera coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system |
- Example
3.1.7 Rotate the Picking Pose around a fixed axis
- Function description
Rotate the Picking Pose by a certain angle around a fixed axis.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Rotation angle | The angle by which the pose is rotated counterclockwise around the fixed axis | 90 | [-360, 360] | degree° |
| Fixed axis | An axis of the Picking Pose. Rotate the pose counterclockwise around this fixed axis | Z-axis | X/Y/Z-axis | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Default coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system | / |
- Example
3.1.8 Translate the Picking Pose
- Function description
Move the Picking Pose by a certain distance along the translation axis.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Translation amount (mm) | The distance the Picking Pose moves along the translation axis. A positive translation amount means translating in the positive direction of the translation axis, and a negative translation amount means translating in the negative direction of the translation axis | 0 | [-1000, 1000] | mm |
| Translation axis | The direction in which the Picking Pose moves | X-axis | X/Y/Z-axis | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Robot arm coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system | / |
- Example
3.1.9 Pick Point Teaching
- Function description
Record the Pick Point coordinates generated by the software and the Pick Point coordinates taught under the current operating condition, then output the transformed Picking Pose based on the offset between the two.
- Usage scenario
When the Pick Points generated by the vision system have an obvious systematic offset and the robot TCP coordinate accuracy is limited or difficult to calibrate, this method can be used to directly map the same offset pattern to subsequent Pick Points, thereby avoiding robot TCP calibration.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Vision Pose | Pick coordinates of the detection result | ||
| X(mm) | X coordinate of the Vision Pose | 0.00 | ±10000000, meaning no limit. |
| Y(mm) | Y coordinate of the Vision Pose | 0.00 | ±10000000, meaning no limit. |
| Z(mm) | Z coordinate of the Vision Pose | 0.00 | ±10000000, meaning no limit. |
| RX(°) | X-axis rotation amount of the Vision Pose | 0.00 | ±180 |
| RY(°) | Y-axis rotation amount of the Vision Pose | 0.00 | ±180 |
| RZ(°) | Z-axis rotation amount of the Vision Pose | 0.00 | ±180 |
| Picking Pose | Manually taught Pick Point | ||
| X(mm) | X coordinate of the Picking Pose | 0.00 | ±10000000, meaning no limit. |
| Y(mm) | Y coordinate of the Picking Pose | 0.00 | ±10000000, meaning no limit. |
| Z(mm) | Z coordinate of the Picking Pose | 0.00 | ±10000000, meaning no limit. |
| RX(°) | X-axis rotation amount of the Picking Pose | 0.00 | ±180 |
| RY(°) | Y-axis rotation amount of the Picking Pose | 0.00 | ±180 |
| RZ(°) | Z-axis rotation amount of the Picking Pose | 0.00 | ±180 |
3.1.10 Refine Object Pose based on plane Normal
- Function description
Correct the Object Pose by fitting the plane Normal so that the Z-axis direction of the Object Pose remains consistent with the direction of the plane Normal of the Target Object.
- Usage scenario
When the Target Object contains a plane and there is a tilt deviation in the plane when the template Point Cloud is matched with the actual Point Cloud, use this function to fine-tune the Target Object plane and improve picking accuracy.
Not applicable to depalletizing scenarios
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Distance Threshold | Distance Threshold for fitting a plane from the Point Cloud | 10 | [-1000, 1000] | mm |
| Save visualization data | If selected, the visualization data will be saved under the historical data timestamp | Selected | / | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Camera coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system | / |
- Example
3.1.11 Sort Pick Points by inter-axis angle
- Function
Sort Pick Points according to the angle between an axis of the Picking Pose and the target axis of the ROI.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Axis selection | An axis of the Picking Pose | Z-axis | X/Y/Z-axis |
| Target axis selection | An axis of the ROI coordinate system | Z-axis | X/Y/Z-axis |
| Select reverse direction | If selected, calculate the angle with the negative direction of the target axis; otherwise, calculate the angle with the positive direction of the target axis | Unchecked | / |
| Select descending order | If selected, sort Pick Points from small to large by angle; otherwise, sort Pick Points from large to small by angle | Unchecked | / |
3.1.12 [Advanced] Rotate the Picking Pose and automatically compensate for excessive angles to the specified axis
- Function description
Determine whether the angle formed between the specified axis of the Picking Pose and the target axis is within the specified range. If not, adjust the Picking Pose into the specified range.
- Usage scenario
Avoid collisions between the robot end effector and the bin.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Angle range | Adjust the Picking Pose into the angle range | 30 | [0, 180] | degree° |
| Specified axis | An axis of the Picking Pose. Adjust this axis so that it falls within the angle range relative to the target axis of the ROI coordinate system | Z-axis | X/Y/Z-axis | / |
| Target axis | An axis of the ROI coordinate system. Compare the angle range with the specified axis of the Picking Pose | Z-axis | X/Y/Z-axis | / |
| Compare with the negative half-axis of the ROI | If not selected, compare the angle range with the positive direction of the target axis of the ROI coordinate system; if selected, compare the angle range with the negative direction of the target axis of the ROI coordinate system | Unchecked | / | / |
| Custom coordinate system | The coordinate system of the Picking Pose | Default coordinate system | Default coordinate system; camera coordinate system; ROI coordinate system; robot arm coordinate system | / |
3.1.13 [Advanced] Symmetry center Object Pose optimization
- Function
Search for the symmetry center of the Target Object based on the instance Mask, then combine it with the plane of the instance or the pose of the ROI 3D center point to calculate the optimal Picking Pose.
Before using this function, first make sure the instance Mask is symmetrical
- Usage scenario
Applicable when the instance Mask of a symmetrical Target Object is also symmetrical, but the Picking Pose is not near the expected center; at the same time, the Target Object has a plane that can be used as a reference, for example, there is a plane on the top of the object, or ROI 3D can be used as a reference for the projected pose.
Applicable project scenarios include brake discs (general circles), refractory bricks (depalletizing), symmetrical irregular parts, fuel fillers, and so on.
- Parameter description
| Parameter | Description | Default | Range | Tuning recommendation |
|---|---|---|---|---|
| Target Object Symmetry type | Target Object Symmetry type of the instance Mask | Rotational symmetry | Rotational symmetry: after the Target Object rotates by a certain angle around the center point, its shape completely overlaps with the original position; mirror symmetry: the Target Object uses a certain axis / plane as the mirror, and the left-right or upper-lower sides are completely symmetrical. | Circles and rectangles are both rotationally symmetrical and mirror-symmetrical, so rotational symmetry is preferred; for trapezoids and other shapes that are symmetrical only along a certain axis or plane, choose mirror symmetry. |
| Gaussian blur level | Tolerance for determining whether the actual Point Cloud overlaps after rotation | 3 | \[1,99\] |
|
| Rotation angle setting | When the symmetry mode is rotational symmetry, it indicates the rotation angle interval, that is, the angle difference between two adjacent rotations. When the symmetry mode is mirror symmetry, it indicates the rotation range, that is, the angle interval within which the Point Cloud can rotate around the symmetry axis. | 180 | \[1,360\] |
|
| Image scaling ratio | Adjusts the size of the Point Cloud image. The larger this ratio, the smaller the Point Cloud image size and the lower the GPU memory usage, but image detail loss increases, resulting in reduced calculation accuracy . | 2 | \[1,10000000\] | |
| Search range | Based on the initially determined center of the Target Object, this defines the range expanded outward to search for Point Cloud features. The actual range is (search range*2*image scaling ratio) | 10 | \[1,10000000\] | For example, for a square Target Object, the initially determined center position of the Target Object is point O. If the search range is set to 10 and the image scaling ratio is 1, then the actual search range is a square region centered at point O with a side length of 10×2×1=20. Point Cloud features are searched within this region to further determine the symmetry center of the Target Object and the optimal Picking Pose. As another example, for a circular Target Object, if the search range is set to 8 and the image scaling ratio is 2, then the actual search range is a circular region centered at the initially determined center of the Target Object with a diameter of 8×2×2=32. Point Cloud features are searched within this region to further determine the Object Pose of the Target Object and the optimal Picking Pose. |
| Use ROI3D as the reference projection plane | If selected, ROI3D is used as the reference projection plane | Unchecked | / | Select this when the Point Cloud has no obvious plane and the projection plane is difficult to determine; leave it unchecked when the Point Cloud has a clear plane. |
| Save symmetry center process data | If selected, the debug data generated during the symmetry center process is saved. You can view it in the `\ProjectName`\data`\PickLight`\HistoricalDataTimestamp`\find`\_symmetry_center folder | Unchecked | / | Select this when you need to inspect the detailed process images |
| Symmetry axis prior type | Effective in ``{=html}mirror symmetry``{=html} mode. Specifies the known Target Object Symmetry type and fixes the asymmetric orientation | Automatic search | Automatic searchSymmetric along the long axisSymmetric along the short axis | If the symmetry axis of the Target Object is the long axis, choose "Symmetric along the long axis". If the symmetry axis of the Target Object is the short axis, choose "Symmetric along the short axis". If uncertain, choose "Automatic search" |
| Pose adjustment type | Whether to inherit pose-related information from the input pose | Default pose | Default poseInherit rotationInherit translation | / |
| Symmetry score Threshold | Symmetry results with a symmetry score lower than this Threshold are abnormal results. When set to 0, no filtering is performed | 0.0 | \[0.0, 1.0\] | / |
- Example
3.2 Pick Point Filtering
3.2.1 Filter by fine matching score
- Function description
Filter Pick Points based on the pose fine matching score.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Score Threshold | Retain Pick Points whose fine matching score is greater than this Threshold | 0.5 | [0, 1] |
- Example
3.2.2 Filter Pick Points of occluded Target Objects
- Function description
Determine whether there are too many occluding object Point Clouds in the target detection area along the specified ROI axis or the Picking Pose axis at the Pick Point of the grasped Target Object. If so, the Target Object is considered occluded and the Pick Point is filtered out.
- Usage scenario
Applicable to depalletizing and ordered scenarios in which Target Objects are picked layer by layer, but the model recognizes lower-layer Target Objects. When picking a lower-layer Target Object, the gripper may collide with the upper-layer Target Object.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Cuboid length in X direction | Set the cuboid length in the X direction of the Picking Pose | 1500 | [1, 10000] | mm |
| Cuboid length in Y direction | Set the cuboid length in the Y direction of the Picking Pose | 1500 | [1, 10000] | mm |
| Cuboid length in Z direction | Set the cuboid length in the Z direction of the Picking Pose | 800 | [1, 10000] | mm |
| Distance Threshold between detection area and Pick Point origin | Along the ROI axis, the nearby cuboid surface area farther than this distance Threshold from the Pick Point origin is regarded as the target detection area | 50 | [1, 1000] | mm |
| Point Cloud count Threshold in detection area | If the number of occluding object Point Clouds in the target detection area exceeds this Threshold, the Pick Point is considered occluded | 1000 | [0, 100000] | / |
| Specified axis direction | Based on the pose reference specified axis direction, set the specific location of the target detection area within the cuboid space (for example, near the front/back/left/right/top/bottom surface of the cuboid) | [0,0,-1] | [1,0,0]: positive X-axis[-1,0,0]: negative X-axis[0,1,0]: positive Y-axis[0,-1,0]: negative Y-axis[0,0,1]: positive Z-axis[0,0,-1]: negative Z-axis | / |
| Use ROI 3D pose reference | If selected, adjust the collision detection area according to the ROI 3D pose reference | Unchecked | / | / |
| Save visualization data | If selected, the visualization data is stored according to the saved data path to help observe whether the generated cuboid is reasonable; if not selected, it is not saved | Unchecked | / | / |
- Example
3.2.3 Filter by Picking Pose angle range
- Function description
Determine whether the angle of the Picking Pose is within the constrained angle range, and filter out all Pick Points that do not meet the condition.
- Usage scenario
Prevent collisions caused by abnormal robot arm Picking Pose angles.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Angle filtering Threshold | Calculate the maximum angle between the specified axis of the ROI and the specified axis of the Picking Pose. Pick Points whose angle is greater than the current Threshold will be filtered out | 30 | [-360, 360] | degree° |
| Invert ROI specified axis direction | If selected, use the negative direction of the specified ROI axis for angle calculation; otherwise, use the positive direction of the specified ROI axis for angle calculation | Selected | / | / |
| Specified Picking Pose axis | Specify an axis of the Picking Pose for angle calculation | Z-axis | X/Y/Z-axis | / |
| Specified ROI axis | Specify an axis of the ROI coordinate system for angle calculation | Z-axis | X/Y/Z-axis | / |
- Example
3.2.4 Filter Pick Points outside the ROI 3D type region
- Function description
Determine whether the Pick Point is within the ROI 3D range, and remove Pick Points that are outside the ROI 3D area.
- Usage scenario
Prevent picking outside the ROI area, which may cause collisions between the robot arm and the target object.
- Parameter description
| Parameter | Description | Default |
|---|---|---|
| ROI3D type region | Usually "workspace"; "pick area" is a smaller ROI region than "workspace", which can restrict Pick Points to an ROI region smaller than the "workspace" to avoid some collision cases. | Workspace |
- Example
As shown in the figure below, when the ROI3D area and ROI2D are area a, the corresponding Pick Point is in the upper-right corner.
When the ROI3D area and ROI2D are changed to area b, the original Pick Point is outside the ROI area, so that Pick Point is removed and a new Pick Point is generated within area b.
3.2.5 [New] Filter Pick Points where the Target Object collides with the gripper (including the original function)
[New] Filter Pick Points where the Target Object collides with the gripper
- Function description
Collision detection between the gripper and the Point Cloud near the Pick Point. If the number of Point Clouds in contact with the gripper exceeds the pick collision Threshold, the Pick Point of the Target Object is considered to have a collision risk.
- Usage scenario
Used when collision detection is required between the gripper and the Point Cloud near the Target Object being picked.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Collision Threshold | Collision distance Threshold. If the distance between the scene and the gripper surface is smaller than this Threshold, it is considered a collision. The larger the Threshold, the stricter it is. Unit: mm | 7 | 1-1000 |
| Collision Point Cloud sampling | Sampling size for collision Point Clouds. The larger the value, the faster the cycle time; the smaller the value, the slower the cycle time. Effective only in "Target Object scene Point Cloud only" and "bin + Target Object scene Point Cloud" modes. Unit: mm | 5 | 1 - 1000 |
| Save visualization data for gripper collision detection | Save visualization data for collision detection between the gripper and the picked Target Object | Unchecked | Selected/Unchecked |
Filter Pick Points where the Target Object collides with the gripper
- Function description
Collision detection between the gripper and the Point Cloud near the Pick Point. If the number of Point Clouds in contact with the gripper exceeds the pick collision Threshold, the Pick Point of the Target Object is considered to have a collision risk.
- Usage scenario
Used when collision detection is required between the gripper and the Point Cloud near the Target Object being picked.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Pick collision Threshold | The maximum number of Point Clouds the gripper may contain near the Pick Point. For example, 20 means that if the number of scene Point Clouds contained by the gripper exceeds 20, it is considered a collision | 20 | 0-10000 |
| Collision Point Cloud sampling (m) | Downsampling size of the Point Cloud in the collision area. The larger the value, the faster the detection speed, but the lower the accuracy. Applicable scenario: scenarios requiring high cycle rates | 0.002 | 0.0001 - 0.5000 |
| Save visualization data for gripper collision detection | Save visualization data for collision detection between the gripper and the picked Target Object | Unchecked | Selected/Unchecked |
| Import gripper model | Select and import the gripper model used for collision detection from a folder | / | / |
**The gripper should be simplified to fewer than 500 faces**
3.2.6 [Advanced] Retain the one Pick Point with the largest/smallest pose value among instance Pick Points and filter the remaining Pick Points
- Function description
Convert the pose to the specified coordinate system, sort poses according to the value of the specified sorting axis, and retain the pose with the maximum or minimum value. This is suitable for cylindrical Target Objects when keeping the top or bottom Pick Point.
- Parameter description
| Parameter | Description | Default | Range |
|---|---|---|---|
| Specified coordinate system | Select which coordinate system the pose should be converted to for processing | ROI coordinate system | ROI coordinate system/camera coordinate system |
| Specified sorting axis | Select which axis value of the pose to sort by | Z-axis | X/Y/Z-axis |
| Take minimum value | If selected, retain the pose with the minimum value on the sorting axis; otherwise, retain the pose with the maximum value on the sorting axis | Unchecked | / |
- Example
3.2.7 [Advanced] Filter Pick Points close to the previous N Pick Points
- Function description
If the variation between the current Pick Point and any Pick Point in the cache is within the Threshold range, the Pick Point will be filtered out.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Upper limit of Pick Point change (+) | ||||
| X(mm) | Upper limit of X coordinate | 2 | [0, 10000000] | mm |
| Y(mm) | Upper limit of Y coordinate | 2 | [0, 10000000] | mm |
| Z(mm) | Upper limit of Z coordinate | 2 | [0, 10000000] | mm |
| RX(°) | Upper limit of RX rotation amount | 1 | [0, 180] | degree° |
| RY(°) | Lower limit of RY rotation amount | 1 | [0, 180] | degree° |
| RZ(°) | Lower limit of RZ rotation amount | 1 | [0, 180] | degree° |
| Lower limit of Pick Point change (-) | ||||
| X(mm) | Lower limit of X coordinate | 2 | [0, 10000000] | mm |
| Y(mm) | Lower limit of Y coordinate | 2 | [0, 10000000] | mm |
| Z(mm) | Lower limit of Z coordinate | 2 | [0, 10000000] | mm |
| RX(°) | Lower limit of RX rotation amount | 1 | [0, 180] | degree° |
| RY(°) | Lower limit of RY rotation amount | 1 | [0, 180] | degree° |
| RZ(°) | Lower limit of RZ rotation amount | 1 | [0, 180] | degree° |
| Pick Point cache count | Number of Pick Points cached. After the current Pick Point comparison is completed, it will be added to the cache in real time | 5 | [1, 100] | / |
3.2.8 [Advanced] Filter Object Poses close to the previous N Object Poses
- Function description
If the variation between the current Object Pose and any Object Pose in the cache is within the Threshold range, the Object Pose will be filtered out. When an Object Pose is determined to be similar, all Pick Points on that Target Object will be filtered out.
- Parameter description
| Parameter | Description | Default | Range | Unit |
|---|---|---|---|---|
| Upper limit of Object Pose change (+) | ||||
| X(mm) | Upper limit of X coordinate | 2 | [0, 10000000] | mm |
| Y(mm) | Upper limit of Y coordinate | 2 | [0, 10000000] | mm |
| Z(mm) | Upper limit of Z coordinate | 2 | [0, 10000000] | mm |
| RX(°) | Upper limit of RX rotation amount | 1 | [0, 180] | degree° |
| RY(°) | Lower limit of RY rotation amount | 1 | [0, 180] | degree° |
| RZ(°) | Lower limit of RZ rotation amount | 1 | [0, 180] | degree° |
| Lower limit of Object Pose change (-) | ||||
| X(mm) | Lower limit of X coordinate | 2 | [0, 10000000] | mm |
| Y(mm) | Lower limit of Y coordinate | 2 | [0, 10000000] | mm |
| Z(mm) | Lower limit of Z coordinate | 2 | [0, 10000000] | mm |
| RX(°) | Lower limit of RX rotation amount | 1 | [0, 180] | degree° |
| RY(°) | Lower limit of RY rotation amount | 1 | [0, 180] | degree° |
| RZ(°) | Lower limit of RZ rotation amount | 1 | [0, 180] | degree° |
| Object Pose cache count | Number of vision Object Poses cached. After the comparison of the current Object Pose is completed, it will be added to the cache in real time | 5 | [1, 100] | / |
3.2.9 [Advanced] Filter Pick Points outside the upper and lower limits of Pick coordinates
- Function description
Retain other Pick Points within the specified range of a reference Pick Point and filter out abnormal Pick Points.
- Usage scenario
Prevent incorrect robot picking and ensure picking accuracy.
This function is not applicable to depalletizing scenarios
- Parameter description
| Parameter | Description | Default | Unit |
|---|---|---|---|
| Reference Pick coordinates | |||
| X(mm) | X coordinate of the reference Pick Point | 0 | mm |
| Y(mm) | Y coordinate of the reference Pick Point | 0 | mm |
| Z(mm) | Z coordinate of the reference Pick Point | 0 | mm |
| RX(°) | RX rotation amount of the reference Pick Point | 0 | degree |
| RY(°) | RY rotation amount of the reference Pick Point | 0 | degree |
| RZ(°) | RZ rotation amount of the reference Pick Point | 0 | degree |
| Upper limit of Pick coordinates (+) | |||
| X(mm) | Upper limit of the X coordinate. For example, if the X coordinate of the reference Pick Point is 100 and the upper limit is set to 10, the allowed range is: [100-lower limit, 110] | 10000000, meaning no limit. | mm |
| Y(mm) | Upper limit of the Y coordinate. For example, if the Y coordinate of the reference Pick Point is 100 and the upper limit is set to 10, the allowed range is: [100-lower limit, 110] | 10000000 | mm |
| Z(mm) | Upper limit of the Z coordinate. For example, if the Z coordinate of the reference Pick Point is 100 and the upper limit is set to 10, the allowed range is: [100-lower limit, 110] | 10000000 | mm |
| RX(°) | Upper limit of the RX rotation amount. For example, if the RX rotation amount of the reference Pick Point is 180 and the upper limit is set to 10, the allowed range is (default angle wraparound applies): [[-180, -170], [180-lower limit, 180]] | 180, meaning no limit. | degree° |
| RY(°) | Upper limit of the RY rotation amount. For example, if the RY rotation amount of the reference Pick Point is 180 and the upper limit is set to 10, the allowed range is (default angle wraparound applies): [[-180, -170], [180-lower limit, 180]] | 180 | degree° |
| RZ(°) | Upper limit of the RZ rotation amount. For example, if the RZ rotation amount of the reference Pick Point is 180 and the upper limit is set to 10, the allowed range is (default angle wraparound applies): [[-180, -170], [180-lower limit, 180]] | 180 | degree° |
| Lower limit of Pick coordinates (-) | |||
| X(mm) | Lower limit of the X coordinate. For example, if the X coordinate of the reference Pick Point is 100 and the lower limit is set to 10, the allowed range is: [100-lower limit value, 110] | 10000000 | mm |
| Y(mm) | Lower limit of the Y coordinate. For example, if the Y coordinate of the reference Pick Point is 100 and the lower limit is set to 10, the allowed range is: [100-lower limit, 110] | 10000000 | mm |
| Z(mm) | Lower limit of the Z coordinate. For example, if the Z coordinate of the reference Pick Point is 100 and the lower limit is set to 10, the allowed range is: [100-lower limit, 110] | 10000000 | mm |
| RX(°) | Lower limit of the RX rotation amount. For example, if the RX rotation amount of the reference Pick Point is 180 and the lower limit is set to 10, the allowed range is (default angle wraparound applies): [[-180, -180+upper limit], [170, 180]] | 180, meaning no limit. | degree° |
| RY(°) | Lower limit of the RY rotation amount. For example, if the RY rotation amount of the reference Pick Point is 180 and the lower limit is set to 10, the allowed range is (default angle wraparound applies): [[-180, -180+upper limit], [170, 180]] | 180 | degree° |
| RZ(°) | Lower limit of the RZ rotation amount. For example, if the RZ rotation amount of the reference Pick Point is 180 and the lower limit is set to 10, the allowed range is (default angle wraparound applies): [[-180, -180+upper limit], [170, 180]] | 180 | degree° |
3.3 Pick Point Sorting
3.3.1 Reference coordinate system
- Function description
Set a unified coordinate system for all instances to group and sort instances.
- Usage scenario
Common to depalletizing scenarios, random picking scenarios, and ordered loading/unloading scenarios
Strategies related to coordinates should first set the reference coordinate system
- Parameter description
| Parameter | Description | Illustration |
|---|---|---|
| Camera coordinate system | The coordinate system origin is above the object, and the positive Z-axis direction points downward; the XYZ values are the values of the center point of the object in this coordinate system |  |
| ROI coordinate system | The coordinate system origin is approximately at the center of the pallet stack, and the positive Z-axis direction points upward; the XYZ values are the values of the center point of the object in this coordinate system |  |
| Robot arm coordinate system | The coordinate system origin is on the robot arm itself, and the positive Z-axis direction generally points upward; the XYZ values are the values of the center point of the object in this coordinate system |  |
| Pixel coordinate system | The coordinate system origin is at the top-left vertex of the RGB image and is a 2D planar coordinate system; the X and Y values are the x value of the bbox detection box and the y value of the bbox detection box, and Z is 0 |  |
3.3.2 General picking strategy
- Parameter description
| Parameter | Description |
|---|---|
| Strategy | Select which value is used for grouping and sorting and how to sort it, including Pick Point center X/Y/Z coordinate values from large to small/from small to large (mm), from the middle to the sides / from the sides to the middle along the Pick Point XY coordinate axis (mm). Multiple items can be superimposed and executed in order. |
| Grouping step size | According to the selected strategy, divide Pick Points into several groups based on the step size. The grouping step size is the interval between two groups of Pick Points |
| Number of leading groups to keep | After grouping and sorting, how many groups of instances need to be retained |
| Strategy name* | Description | Grouping step size | Number of leading groups to keep | |
|---|---|---|---|---|
| Default | Range | Default | ||
| Pick Point center X/Y/Z coordinate values from large to small / from small to large (mm) | Use the X/Y/Z coordinate values of the Pick Point center for grouping and sorting | 200.000 | [0, 10000000] | 10000 |
| From the middle to the sides / from the sides to the middle along the Pick Point XY coordinate axis (mm) | Use the X/Y coordinate values of the Pick Point center and perform grouping and sorting in the direction of "middle to sides" or "sides to middle" | 200.000 | [0, 10000000] | 10000 |
3.3.3 Carton combination strategy
To solve the problems of low efficiency and limited applicable scenarios in traditional single-pick depalletizing, PickWiz adds a carton combination strategy in depalletizing scenarios to support picking multiple Target Objects in a single operation. It supports the core scenarios of "cartons with consistent dimensions" and "rectangular suction cups", covering more real project scenarios.
3.3.3.1 Multi-pick runtime configuration
（1）In sack single depalletizing or carton single depalletizing scenarios, enable Vision computation configuration - Vision computation acceleration.
（2）Under the Pick Point sorting module, select the carton combination strategy;
（3）Strategy selection: available options are the default combination strategy or combination along a specified carton pose axis. These are two methods for finding the largest number of cartons that can be combined.
Default combination strategy: Find the largest number of cartons that can be combined along the X-axis and Y-axis directions of a carton Picking Pose.
Combine along a certain carton pose axis: Find the largest number of cartons that can be combined along the X-axis or Y-axis direction of a carton Picking Pose. This is suitable for scenarios where cartons are arranged in a straight line. When using this strategy, you need to choose the carton combination direction, namely the Picking Pose X-axis or the Picking Pose Y-axis.
Note:
The combination direction is related to the positive and negative axis directions. Cartons can be combined only when they are on the same axis and in the same direction, and after combination, the orientation of the whole stack of cartons remains consistent with the orientation of a single carton before combination. Therefore, before combining cartons, make sure all cartons to be combined are placed in the same orientation, and unify the coordinates of the cartons to be combined to the same axis direction.
（4）Combination conditions: determine which cartons can be combined and how many can be combined at most.
Maximum cartons per row: the maximum number of cartons that can be combined in one row, default is 2.
Maximum number of combination rows: the maximum number of carton rows that can be combined, default is 1.
Maximum spacing (mm): cartons to be combined cannot be too far apart in the "combination direction", In the combination direction (axis direction), when the spacing between two adjacent cartons or cartons in different rows is less than this value, they can be combined into one group. The default is 10.
- Example: when searching for the maximum number of cartons along the Picking Pose X-axis, if the spacing between two adjacent cartons in the Picking Pose X-axis direction is 8 mm (≤10), they can be combined; if the spacing is 12 mm (>10), they cannot be combined.
Maximum misalignment distance (mm): cartons to be combined cannot be too far apart in the direction "perpendicular to the combination direction" . In the direction perpendicular to the combination direction (axis direction), when the misalignment distance between two adjacent cartons or cartons in different rows is less than this value, they can be combined into one group. The default is 10.
- Example: when searching for the maximum number of cartons along the Picking Pose X-axis, if two adjacent cartons are offset in the Picking Pose Y-axis direction by 8 mm (≤10), they can be combined; if they are offset by 15 mm (>10), they are no longer aligned and cannot be picked together.
Maximum angular deviation (°): cartons to be combined should face almost the same direction. In the combination direction (axis direction), when the rotational deviation angle of the cartons is less than this value, they can be combined into one group. The default is 10.
- Example: if a carton is rotated by 5° relative to the combination direction, as long as it does not exceed 10°, it can be combined; if it is rotated by 15° (>10), the orientation differs too much, the robot will be skewed when picking, and it cannot be combined.
3.3.3.2 Robot configuration
（1）On the robot configuration page, add new placeholders in Vision computation communication message - Robot to PickWiz commands - Vision detection send command: maximum cartons per row and maximum number of combination rows, as shown below;
（2）In Vision computation communication message - PickWiz to robot commands - Pick-related information - Returned information when picking Target Objects, add Object Dimensions length, Object Dimensions width, and Target Object orientation.
After the robot configuration is completed, click the Run button.
3.3.3.3 View multi-pick runtime results
（1）In the 3D Matching window, hover the mouse over an instance to view the combined picking information of a single instance after carton combination, including 2D recognition results, Picking Pose, and instance combination information.
In the visualization window, click the Settings button in the upper right corner to set how the combined instance information is displayed.
Right-click an instance to view the combined picking information and Target Object information of the single instance.
（2）In the 2D recognition window, you can use the relevant combination buttons in the menu bar to view the combined ID, combined Mask, and combined bounding box.
