Connect an industrial robot to python code and build a bridge to a world of possibilities

Our company owns an 11 year old Mitsubishi 6 axis robot running Melfa Basic used mainly for testing and for feasibility studies involving automatic measurement projects. A little over a year ago I concluded that AI was going to be the next best thing since sliced bread and our company had to set off on the long and stony path from "just learning" to machine learning.

Now armed with TensorFlow and a machine learning pipeline, simulated part image datasets of various objects from lego bricks to bushings, we searched for a real world test. It was clear that our test robot had to follow the same path and also get involved in project AI.

We reactivated a two-way socket communication between the robot control unit and a PC used in previous projects. On the robot side one process handled the continious position and status feedback from the robot, a second process received string messages and separated the tokens into commands and position data. A third Melfa Basic task orchestrated the tasks by starting, restarting and halting the communication as required.

Below is a sample string message to send the robot to a new position:

msg = "GOPO +0550.00 +0100.00 +0220.00 -0180.00 +0000.00 +0045.00"

The first 4 letter code GOPO, short for the command "goto position", is followed by the position data itself. The interface is quite simple, language agnostic however a little slow updating the status / positions with a refresh cycle time of 20ms. The UDP version which has been used in other applications can update the same data every 7ms. Speed of communication was, this time, not a high priority - a simple python integration was top of the list.

Searching for a host for our industrial machine learning and vision applications the list was rapidly narrowed down to a Nvidia Jetson AGX Xavier. The Xavier is the fastest unit available from Nvidia for "on the edge" processing with the possibility of connecting up to 6 cameras.

The target application - a bin picking application for screws first using on board OpenCV functionality later enhanced with machine learning.

OpenCV Functionality


The OpenCV functionality was divided into 3 phases.

  • identify the target screw
  • calculate its position in x,y and z
  • calculate the corresponding gripper position

The first phase is an eight stage process to identify the most accessable screw:

  • capture the image
  • convert to black and white
  • filter using a threshold
  • blur to blend details into objects
  • erode to separate the objects
  • remove noise
  • detect contours
  • interpret the contours
  • repeat with a new threshold until only the most accessable screw is detected