Rotobot OpenFX Plugin

Use of Rotobot to integrate computer graphics elements behind a machine learning based mask

Rotobot is a tool that does what is known as “semantic instance segmentation” or “semantic segmentation”

That sure is a mouthful, but what does it mean?

It means you can choose a category such as person and all of the pixels that belong to the class of “person” will be isolated to that class. But instead of lumping all the “person” pixels together into a single layer, individual people will be in their own layer.

Rotobot is now available in the shop, for free download with a watermark and reasonably priced for annual licensing, for MacOS, Windows and Linux.

As you can see, “person” is a class and an instance of a person will pick one person out from a crowd. We do this by choosing the pixel coordinate that lies within a bounding box from a class, which means that if you use a tracking marker to track a person’s eye for instance, you can feed that pixel coordinate into Rotobot to isolate that person from the crowd.

We have also added just simple “segmentation”, which adds all of the “people” into one mask. This adds greater detail and stability, but we are packaging both models to use as your needs suit you. The detail comes at greater computational cost, but should still be faster than doing it by hand.

That is great for person, but what else can you isolate?

Screen Shot 2018-09-08 at 3.57.25 pm
“Person”, “Bicycle”, “Car”, “Motorcycle”, “Airplane”, “Bus”, “Train”, “Truck”, “Boat”, “Traffic Light”, “Fire Hydrant”, “Stop Sign”, “Parking meter”, “Bench”, “Bird”,”Cat” ,”Dog”,”Horse” ,”Sheep” ,”Cow” ,”Elephant” ,”Bear” ,”Zebra” ,”Giraffe” ,”Backpack” ,”Umbrella” ,”Handbag” ,”Tie” ,”Suitcase” ,”Frisbee” ,”Skis” ,”Snowboard” ,”Sportsball” ,”Kite” ,”Baseball bat” ,”Baseball glove” ,”Skateboard” ,”Surfboard” ,”Tennis Racquet” ,”Bottle” ,”Wine Glass” ,”Cup” ,”Fork” ,”Knife” ,”Spoon” ,”Bowl” ,”Banana” ,”Apple” ,”Sandwich” ,”Orange” ,”Broccoli” ,”Carrot” ,”Hotdog” ,”Pizza” ,”Donut” ,”Cake” ,”Chair” ,”Couch” ,”Potted Plant” ,”Bed” ,”Dining Table” ,”Toilet” ,”TV” ,”Laptop” ,”Mouse” ,”Remote” ,”Keyboard” ,”Cell Phone” ,”Microwave” ,”Oven” ,”Toaster” ,”Sink” ,”Refrigerator” ,”Book” ,”Clock” ,”Vase” ,”Scissors” ,”Teddy Bear” ,”Hairdryer” ,”Toothbrush” ,”Banner” ,”Blanket”

The list for segmentation is a little more limited to 19 classes:

“aeroplane”, “bicycle”, “bird”, “boat”, “bottle”, “bus”,
“car”, “cat”, “chair”, “cow”, “dining table”, “dog”, “horse”, “motorbike”,
“person”, “potted plant”, “sheep”, “sofa”, “train”, “tv”

So how will I manage all that and keep the graphical user interface simple for making masks?

The current approach is that there are three panels for red, green and blue.

Simply choose the categories and pixel coordinates that you want in red, green, or blue.

So you could choose, truck, car, train and bus in red, and person in green and then bird in red. Then shuffling the channels out you can use this to mask and effect like desaturating the background that is not person or the vehicle categories.

Typically we would spend hours creating masks by hand tracing around footage at 24, 29.97 or 30 frames per second. With Rotobot, we can process a frame of footage in as little as 5-20 seconds depending on your hardware (64 bit processors and 2GB of RAM are required. No GPU is required.), but may be as high as two to three minutes for complete accuracy. There is also a setup cost of decrypting the model for the first frame of processing. But over a number of frames this will amortise away to nothing.

The accuracy of the masks is limited to the quality of the deep learning model behind Rotobot. The developers at Kognat are working hard to improve the accuracy and resolution.

Rotobot has been tested at most of the major visual effects studios around the world and is available for 64 bit operating systems as follows: Linux, Windows and macOS.

The screen shot shows the Foundry Nuke interface, but OpenFX is a plugin standard that works across a number of platforms it has been confirmed to work in the Free Open Source Software called Natron also. OpenFX plugins only run in licensed versions of Nuke and will not work with unpaid versions.

One more thing to let you know about Rotobot, DOES NOT use the internet to process the images, the only network traffic would be searching for the license server. I know people working on feature films are concerned about security and are often fire walled from rest of the internet, that is no problem for the Rotobot OpenFX plugin. At Kognat we prefer that the software is simply installed and runs on your hardware on your network.