Cognitive Manipulation: Semi-supervised Visual Representation and Classroom-to-real Reinforcement Learning for Assembly in Semi-structured Environments (2024)

IV-A1 Skill Graph Based on General Knowledge and Task Specification

The abstract expert knowledge of assembly tasks in semi-structured environments can be harnessed to segment the manipulation process into distinct stages and components, as depicted in Fig. 5. The skill graph integrate symbolic and subsymbolic representations according to general versus task-specific knowledge and degrees of accessibility for a well-designed yet flexible structure.

We define general knowledge using a partial model that encapsulates spatial, temporal, and causal information. Initially, we consider the spatial information concerning the end-effector (EE) pose in the manipulation, which includes the home position ${}^{B}X_{E}^{s}$, assembly bottleneck pose ${}^{B}X_{E}^{m}$, and assembly goal pose ${}^{B}X_{E}^{g}$ within the robot base frame. Temporally, the process is segmented into four stages: reaching ${}^{B}X_{E}^{s}$ for global perception to estimate the ${}^{B}X_{E}^{g}$, planning a coarse operation from ${}^{B}X_{E}^{s}$ to ${}^{B}X_{E}^{m}$ and then to ${}^{B}X_{E}^{g}$, executing the coarse operation to reach ${}^{B}X_{E}^{m}$, and performing the fine operation for insertion at ${}^{B}X_{E}^{g}$. Causal transition conditions between these stages are defined based on the positional relationships and contact states between the peg and hole. This partial model is illustrated in Eqn. (1) and Fig. 5. Although the introduction of sequential logic and causality is crucial for operating in semi-structured environments—enhancing safety and reducing learning costs due to potential interference from other agents (robots or humans), this paper primarily focuses on the sequential logic and causal enhancement of robot learning methods, while not extensively addressing multiple exceptional states and their management strategies.

Despite the generic nature of temporal and causal information in assembly tasks, task-specific information remains essential. For the manipulation process, the bottleneck poses ${}^{B}X_{E}^{m}$ and goal pose ${}^{B}X_{E}^{g}$ of EE are determined by the task’s geometry and assembly relationships, master object pose, grasp pose for the slave object, and the tool center position (TCP) offset. Object-Embodiment-Centric (OEC) geometry representation is derived from demonstrations to enable direct prediction of key waypoints in the operational process through the master object pose estimation, as outlined in our prior work [48]. The OEC representation selects a grasping point on the task board as the coordinate origin ${}^{B}X_{O}$ and extrapolates the bottleneck pose of EE ${}^{B}X_{E}^{m}$ and goal pose ${}^{B}X_{E}^{g}$ in the robot base frame from the teaching, then transform these to ${}^{O}X_{E}^{m}$ and ${}^{O}X_{E}^{g}$ in the task frame.In the semi-structured environment, the master object is randomly placed within a workspace of range $S_{uncon}$, as specified by a human operator. The home position of the EE ${}^{B}X_{E}^{s}$ is strategically set outside this region to prevent occlusion of the eye-to-hand camera’s field of view. With partial constraints of the workspace, the pose can be determined by estimating the $x$, $y$, and $rz$ dimensions. The uncertainties introduced by the pose estimation and demonstration system, as well as contact dynamics, are also considered for precise contact-rich task execution.In conclusion, employing the general knowledge base, both planning and learning-based methods are utilized for developing a complex policy, which includes object detection for pose estimation and task-related visual information extraction, spatially dependent trajectory planning as the basic strategy, and task-specific residual strategies for handling uncertainties. A graph structure is devised to reflect the required sequence of motions and modules necessary to complete the task, as illustrated in Fig. 4.

IV-A2 Compliance Controller

Employing the virtual force-driven spring-mass-damping modal and robot kinematics, we utilize a modified Cartesian parallel position and force controller as the low-level control for robot learning, generating velocity commands. The control law for joint velocities $\dot{q}$ is expressed as:

IV-A3 Object Detection for Pose Estimation and Task Attention

Object detection is a popular algorithm used to locate objects in an image or video stream. It predicts multiple bounding boxes for objects in the image $I$, and each bounding box contains the predicted values for the object’s position $(x,y)$, size $(w,h)$, confidence $c_{con}$, and category $c_{cate}$, as shown in (3). With a pre-defined confidence level, the effective predicted bounding box for the object is selected.

To cover the entire workspace and accurately detect the object of interest, we attach an eye-to-hand RGB camera at the top of the workspace to capture the 2D image $I_{eth}$, as shown in Fig. 4 (b). An object detection-based coarse perception system generates one bounding box around the object of interest to obtain the location $(x_{0},y_{0})$ and two other bounding boxes around the predefined feature structures to obtain the location $(x_{1},y_{1})$ and $(x_{2},y_{2})$. According to the eye-to-hand transformation ${}^{r}T_{c}$, the estimated points $(x_{0}^{\prime},y_{0}^{\prime})$ are transformed to the robot frame as shown in (4). Considering the partial pose information of the object, including $rx_{con}$, $ry_{con}$, and $z_{con}$, the pose ${}^{B}X_{O}$ can be determined as in (5). Global perception is used in the first stage $n_{1}$ to determine whether the main assembly object is ready for the assembly operation on the one hand, and to provide location information for the assembly operation on the other hand.

The second model uses the image $I_{eih}$ from an eye-in-hand camera to provide local task detection as attention, enhancing the ability to differentiate tasks from the environment as indicated in Fig. 4 (b). Object detection utilizes a bounding box as a region of interest (ROI) to identify specific structures crucial for vision-based precise alignment in assembly tasks. The work uses a simple attention strategy that utilizes this bounding box $(x_{a},y_{a},w_{a},h_{a})$ to crop and resize the task-related area from the input image $I_{eih}$ and generate an attention-guided observation $I_{atten}$ for fine manipulation, enabling the residual policy to concentrate on the specific structure amidst varying environments.

IV-A4 Coarse Operation Planing

With the partial model and coarse perception system, we can plan a coarse operation as the second stage $n_{2}$. We first obtain the assembly goal point ${}^{B}X_{E}^{g}$ and bottleneck pose ${}^{B}X_{E}^{m}$ with the estimated pose ${}^{B}X_{O}$ and the OEC geometry information of ${}^{O}X_{E}^{m}$ and ${}^{O}X_{E}^{g}$, which divide the operation into contact-free and contact-rich manipulation.

The uncertainty due to pose estimation and compliance control can be ignored in the contact-free region $S_{cf}$. A fast min-jek trajectory $\tau_{cf}$ between the home point ${}^{B}X_{E}^{s}$ and the bottleneck pose ${}^{B}X_{E}^{m}$ can be generated. In addition, a high-stiffness $K_{cf}$ of compliance controller is used to ensure acceptable position tracking errors. The trajectory and stiffness provide coarse operation for the contact-free region, which can be defined as Eqn. (7).

However, the uncertainty cannot be ignored in the contact-rich region. A slow trajectory $\tau_{cr}$ and small stiffness are used in the contact-rich region $S_{cr}$. We define an exploration space $W$ to represent the offset range of a compliance robot disturbed by safe external forces $F_{max}$. It should cover the assembly depth and error range to ensure safe contact and effective error compensation, as shown in Eqn. (8). Furthermore, the small stiffness matrix of compliance control $K_{cr}$ is obtained with the estimated exploration space $W$ and maximum contact force $F_{max}$, which can be defined as,

IV-A5 Residual Policy for Fine manipulation

The manipulation is divided into two phases according to the planned coarse operation and carried out by the compliance controller in (2). In the third stage $n_{3}$, the end effector of the robot is moved from the home point ${}^{B}X_{E}^{s}$ to the bottleneck pose ${}^{B}X_{E}^{m}$ with a planned efficient policy $\pi^{cf}_{H}$.

Since the contact-rich assembly manipulation in the fourth stage $n_{4}$ requires a higher level of accuracy than conventional robot and vision systems, the planned safe policy $\pi^{cf}_{H}$ is switched and the residual policy $\pi_{\theta}$ is enabled to refine the initial policy for precise localization and complex force control. In addition to guidance from a fixed policy, the residual policy also receives attentional observation guidance $I_{atten}$ from object detection. The tactile $F$ from the force sensor mounted on the wrist and the relative pose $R_{p}={}^{B}X_{E}-{}^{B}X_{E}^{g}$ of the end-effector serve as additional observations for contact dynamics. The residual policy generates the desired force and torque $F_{d}$ in Eqn. (9), which together with $\pi^{cf}_{H}$ as input to the compliance controlle in Eqn. (2).

With the help of coarse operation planning, the learning of cognitive manipulation is carried out separately. The object detection models and hand-eye-task calibration are trained by semi-supervised visual representation learning in subsection B. The residual policy is trained by classroom-to-real residual reinforcement learning in subsection C.

IV-B1 Data Collection via Coarse Operation

The master object may appear at different points $P$ throughout the workspace $S_{w}$, requiring global localization, further global localization exists as possible relative pose error $R_{P}$ within the range $E_{r}$, requiring further local perception as visual attention for fine manipulation. To ensure data diversity for training the position and attention models, we first uniformly collected $m$ points within the workspace where the task board will appear during the real scene, denoted by $P=[x^{r},y^{r}]$. Second, we used an offset to the bottleneck pose ${}^{B}X_{E}^{m}$ and generated $n$ points to cover the uncertainty space while avoiding collisions, defined them as $R_{P}=[\Delta x^{r},\Delta y^{r},\Delta z^{r}]$. The sampling points and poses are shown in Fig. 6 (b). We collected eye-in-hand and eye-to-hand images, as well as the corresponding position and relative pose, using a hand-designed trajectory for data diversity, as shown in Eqn. (10).

IV-B2 Calibration and Automate Label

For eye-to-hand, the transformation between camera and robot, ${}^{c}T_{r}$ and ${}^{r}T_{c}$, need to be calibrated for automated label and vision-based localization in robot control. Considering only $x$ and $y$ dimensions, the transform can be formulated as Eqn. (11) and (12).

For eye-in-hand, we mainly focus on the mapping relationship of relative motion between the robot and the task in the robot coordinate system and pixel coordinate system, to realize semi-automatic annotation. Because the image Jacobian can be considered as constant in a limited space, the transform $J$ can be formulated as Eq. (13).

Using a small number of coordinates in the pixel frame $L_{eth}^{i}=[x_{i}^{c},y_{i}^{c}]$ and $L_{eih}^{i}=[{}^{l}\Delta x_{i}^{c},{}^{l}\Delta y_{i}^{c},{}^{r}\Delta x_{i}^$ provided by manual annotation and coordinates in the robot frame captured during sampling, the transformation from the robot base frame to the eye-to-hand pixel frame ${}^{c}T_{r}$, ${}^{r}T_{c}$ and the relative motion relationship between the robot base frame and the eye-to-hand pixel frame can be estimated. The calibrated transform relationship can be used to automate the labeling of the remaining images. In addition, the transform ${}^{r}T_{c}$ from the pixel frame to the robot’s base frame can be used as hand-eye-task calibration, which involves estimating the assembly pose of the robot’s end-effector for localization by the eye-to-hand camera.

IV-B3 Fine-tuning from Pre-trained Model

This work used a one-stage real-time object detection approach, YOLO (You Only Look Once), to estimate assembly goal pose and visual attention, which is famous for robustness and fast computation. In addition, the pre-trained model using the ImageNet dataset can be used as initial parameters for training in custom datasets, which require fewer samples. The image and labels are divided into training and test sets for model training and evaluation.

IV-C1 Curriculum Residual Learning

To increase the robustness of residual policy to the perceptual uncertainty of the pose estimator, a random error is injected into the trajectory. However, the error range may affect the learning efficiency due to the difficulty of exploration and the diversity of samples, as shown in Fig. 7. Therefore, this work automatically controls the task difficulty by increasing or decreasing $\varepsilon$ to the guidance error range of $E_{r0}$ to keep the success rate $s_{r}$ within the desired interval $[\alpha,\beta]$, as shown in Eqn. (15).

IV-C2 Reward Shaping

This work normalizes the Euclidean distance between the end-effector current $X$ and target pose ${}^{B}X_{E}^{g}$ to create a guide reward $R_{guid}$ that increases as it gets closer to the target pose as in Eqn. (16). The force penalty $R_{forc}$ is defined according to the interaction force $F$ for smooth operation as in Eqn. (17). Additionally, if the transition condition $c_{4}$ is satisfied, it gets a positive reward of $R_{succ}$ as in Eqn. (18). Therefore a multi-objective reward function is defined as Eqn. (19), which uses weights $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ to balance multiple sub-objectives.

IV-C3 Soft-Actor-Critic

A model-free DRL, soft actor-critic, is introduced to achieve a real-time optimal control strategy for intricate fine manipulation. Unlike pure RL, residual learning enhances the performance of the entire policy by optimizing the residual parameterized part of the policy. The state and action of the residual policy, along with the reward of the overall policy, are gathered in multiple recurring episodes and stored in a data replay buffer for off-policy learning. The training was carried out in a structured environment and then transferred to a semi-structured environment. The cognitive manipulation algorithm is depicted in Aglo.1, where Line 1-2 acquire the coarse operation $\pi^{cf}_{H}$ and $\pi^{cr}_{H}$ with the estimated localization ${}^{B}X_{E}^{g}$, Line 3 -9 obtain the desired pose, $X_{d}$, stiffness $K$ and desired force/torque $F_{d}$, Line 6 drives the robot to the assembly bottleneck pose, Line 11-18 perform fine manipulation for accurate assembly tasks.

V-A1 Hardware and Software

The experiments are conducted on a computer equipped with an Nvidia GeForce RTX 2060 GPU and an Intel i7-9700 CPU. The Robot Operating System (ROS) is utilized as the middleware, facilitating seamless communication between the learning algorithms, control modules, and the robotic system.

V-A2 Partial Model, Data Collection and Reward Design

The geometry information for data collection in semi-supervised object detection models and classroom-to-real fine manipulation policy training is obtained by demonstration. Maximum contact force $F^{max}$ is set as 10 N in x, y and z directions and 0.1 N*m in Rx, Ry and Rz directions. The hybrid policy updates the pose and force commands to the controller at a frequency of 5 Hz, while the controller outputs the target joint velocities for the robot at 120 Hz. Each experimental episode is capped at 120 steps, with policy networks undergoing 200 gradient updates per episode. The reward weights $w_{1}$, $w_{2}$, and $w_{3}$ are set to 1, 0.8, and 1, respectively, as determined by preliminary experiments to balance operation speed and smoothness. The curriculum increases or decreases by 0.5 mm to the error range from 2 mm to keep the success rate within the desired interval [0.5, 0.7].

V-A3 Baselines for Comparative Study

V-B1 Embodied Hand-Eye-Task Calibration and Semi-Automatic Annotation

The dataset is curated based on prior knowledge of uncertainty to ensure sample diversity. The initial stage aims to minimize the costs associated with manual labeling and hand-eye calibration while generating high-quality labeling data and accurately estimating the target assembly pose of the end-effector. This stage encompasses four critical processes: data acquisition, manual labeling, hand-eye-task relationship fitting, and semi-automatic labeling. We explore the dependency on the proportion of manually annotated samples and its advantages over purely manual annotation. The standard deviation of the hand-eye relationship fitting serves as a quantitative index for evaluating the calibration and annotation.

The number of manually labeled samples and the corresponding accuracy of ${}^{c}T_{r}$, $J$, and ${}^{r}T_{c}$ represented by standard deviation, as well as the standard deviation of ${}^{c}T_{r}$ and $J$ on all data sets are shown in Table III. The results demonstrate that a small number of manually labeled samples can effectively establish hand-eye calibration relations and facilitate semi-automatic labeling of the remaining images, reducing the standard deviation of data annotation. Examples of both manually and automatically annotated data are illustrated in Fig. 12. The semi-automatic annotation reduces the cost of manual annotation by 97.4%.

V-B2 Supervised Fine-tuning for Object Detection

In this phase, We examine the impact of sample diversity on model performance by designing various sampling schemes and comparing models trained on different dataset sizes. We mainly consider two factors in the sampling process: pick-and-place position and robot posture, which together affect sample size and diversity, including 10 (5*2), 30 (10*3), 60 (14*5), 120 (15*8), 180 (18*10), 384 (24*16).In addition, the combination of 10 (5*2) after data enhancement to 300 is used as a baseline to facilitate the comparison between embodied data acquisition and traditional data enhancement methods based on graph transformation. 20$\%$ of the embodied data enhanced 385 data is taken as the verification set to represent the complex state in the assembly operation process. Precision, recall, mAP@.5, and mAP@.5:.95 are employed to assess the influence of sample quantity and data enhancement methods on the performance of the object detection models.

The results indicate that data acquisition based on prior knowledge significantly enhances the environmental perception capabilities of the detection model. The mAP@.5:.95 values for the two object detection models during the training process are displayed in Fig. 12, highlighting the importance of sample diversity and the limitations of data enhancement based solely on graphical transformations. Contrasting the effect of the number of samples on the model performance, too few samples, less than 60, will cause the training process to fail to converge. With the increase in the number of samples, the trained model can obtain higher precision, recall, mAP@.5, and mAP@.5:.95 values. This is due to improved sample diversity by more background changes and robot-task relative posture. In particular, the performance of local task attention models is more sensitive to sample diversity due to unavoidable occlusion during contact-rich operations. Contrasting methods of sample generation, although data enhancement based on graph transformation can increase the number of samples to avoid overfitting, it lacks diversity. The position estimation model achieved acceptable results, while the task attention model performed poorly. The embodied data collection increases the mAP@.5:.95 by 5.5$\%$ in global perception and 27.5$\%$ in local perception compared to existing data augmentation methods.

V-B3 Residual Reinforcement Learning of Fine Manipulation

This stage involves training a residual policy, supported by a hand-designed base policy and task-focused view, on a fixed master object setting.Physical contact states are identified using LSTM networks to encode time-series data from touch and proprioception sensors, combined with visual feedback processed through a CNN for serious position error. An MLP then integrates the low-dimensional latent features from LSTM and CNN to generate residual actions. We contrast this procedure with three baselines 3, 4, and 5 to highlight the advantages of knowledge-informed learning.

Results suggest that our approach facilitates more efficient and effective learning by concentrating on task-relevant details and addressing intricate contact dynamics and positional uncertainties. The success rate and error range throughout the training of the manipulation policy are presented in Fig. 11. In baseline 3, without the base policy, pure RL struggles in precise insertion tasks due to a local optimum created by penalizing contact forces. Combining the base policy with RL-based residual policy in baseline 4 can result in success, but the curriculum only achieves an 8 mm error in force-based residual policy training due to limited observations of contact force and vision-based pose estimation, creating a Partially Observable Markov Decision Processes (POMDP). In baseline 5, utilizing raw visual information can improve observations, with the curriculum achieving around 20 mm. The ROI-based attention in our approach, allowing the policy to rely on limited features and introducing perturbations due to restricted detection accuracy, marginally influences efficiency.

V-B4 Cognitive Manipulation in Semi-structured Environment

After individual training phases, the integrated policy is evaluated for its effectiveness in terms of success rate and completion time during assembly tasks in a semi-structured environment. The manipulator grasps the slave object, a gear, while the master object, a task board, is randomly positioned within a confined workspace of 350*350 mm. We carry out 16 trials to compare our method with other baselines, excluding the non-convergent baseline 3.

The results indicate that our approach exhibits superior performance in handling challenging assembly tasks, achieving a perfect success rate and significantly reducing completion steps, as outlined in Table III. Baseline 1 can only complete 12.5 $\%$ of trials with an average of 64 steps, while baseline 2 can only complete 40 $\%$ of trials with an average of 104 steps, nearing the maximum time step. The object detection, single-camera usage, and the simple model-based method fall short of meeting the task’s accuracy and the environment’s uncertainty requirements. Although random residual actions can help to compensate for the perception errors, the semi-structured environment poses additional challenges due to movable objects. The contact force generated during the random search can displace the task board, leading to larger errors or even causing the gear to slip off the peg. Baseline 4 outperforms baseline 2 in both success rate and cost steps because force-based agents can enhance the search policy by regulating the contact force and position reference based on the estimated contact state derived from interaction forces. Although baseline 5 is more robust than baseline 4 in training, it only performs 13.95 $\%$ better in costed steps and even worse in success rate. The raw visual information enables the residual policy to compensate for larger errors in training, but its performance may degrade in different locations with different backgrounds. In comparison, the proposed method utilizes visual attention to assist the agent in focusing on the task, resulting in a success rate of 1 with an average of 54.4 costed steps. In conclusion, the success rate is increased by 13$\%$ and the number of steps is reduced by 15.4$\%$ compared to competing methods.