MIT CSAIL uses AI to teach robots to manipulate objects they've never seen before

In a few areas, the impact of artificial intelligence (AI) has been more transformative than robotics. OpenAI, a San Francisco-based start-up, has developed a model that directs mechanical hands in handling objects with cutting-edge precision. Softbank Robotics recently turned to Affectiva to integrate their emotional intelligence with their Pepper robot.

The latest advances were made by researchers at the Massachusetts Institute of Technology's Computer and Artificial Intelligence Laboratory (CSAIL), which in a paper
Descriptors and application to robotic manipulation ") has detailed a computer vision system, dubbed Dense Object Nets, which allows robots to inspect, visually understand and manipulate objects they have never seen before.

They plan to present their results at the Robot Learning Conference in Zurich, Switzerland, in October.

"Many approaches to manipulation do not identify specific parts of an object through the many directions that the object may encounter," said Lucas Manuelli, lead author of the paper, in an article published on the site. Web of MIT CSAIL. "For example, existing algorithms would be unable to grab a cup by its handle, especially if the cup could be in multiple orientations, such as the vertical or sideways."

DON is not a control system. It is rather a self-supervised deep neural network – layered algorithms that mimic the function of neurons in the brain – driven to generate descriptions of objects in the form of precise coordinates. After training, he is able to select reference frames independently and, when presenting a new object, combine them to visualize their three-dimensional shape.

According to the researchers, object descriptors take only 20 minutes on average, and they are task independent – that is, they apply to both rigid objects (eg hats) and non-rigid objects (plush toys). ). (In a training cycle, the system learned a cap descriptor after seeing only six different types.)

In addition, the descriptors remain consistent despite differences in color, texture, and shape of objects, giving DON a head start on models that use RGB or depth data. Since the latter does not have a coherent representation of the objects and effectively searches for the "enterable" characteristics, they can not find such points on objects with even slight deformations.

"In factories, robots often need complex part loaders," Manuelli said. "But a system like this, able to understand the directions of the objects, could just take a picture and be able to grasp and adjust the object accordingly."

During the tests, the team selected a pixel in a reference image so that the system could identify it autonomously. They then used a Kuka arm to grab isolated objects (a caterpillar toy), objects in a given class (different types of sneakers) and objects in a clutter (a shoe in a spread of others). shoes).

During a demonstration, the robotic arm managed to pull a hat on a pile of similar hats, although he had never seen any pictures of the hats in the training data. In another, he grabbed the right ear of a caterpillar toy from different configurations, demonstrating that he could distinguish the left from the right on symmetrical objects.

"We observe that for a wide variety of objects, we can acquire dense descriptors that are consistent between viewpoints and configurations," the researchers wrote. "The variety of objects includes moderately deformable objects such as soft plush toys, shoes, mugs and hats, and may include objects with very low texture. Many of these objects came just from the lab (including the shoes and hats of authors and co-workers), and we were impressed by the variety of objects for which dense and consistent visual patterns can be learned with the same network architecture and training. "

The team thinks that DON could be useful in industrial environments (think of warehouse robots sorting objects), but he hopes to develop a more efficient version capable of performing tasks with a "deeper understanding" of the corresponding objects.

"We think that dense object arrays are a new representation of objects that can enable many new approaches to robotic manipulation," the researchers wrote. "We are interested in exploring new approaches to solving manipulation problems that exploit the dense visual information provided by dense descriptors and how these dense descriptors can benefit other types of robotic learning, eg. learn to grab, manipulate and place a set of objects of interest. "