Rotobot was trained in sRGB, the two models Segmentation and Instance Segmentation give different results

It is a little known fact that Rotobot was trained in the colour space of the Internet sRGB or a Gamma of 1.8-2.2 with respect to light. This means that if your footage is not in sRGB it is not going to detect as well as if it were in sRGB. So how do you do that?

Using a OCIO Colorspace Transform node, you can come from an “in” color space of linear to an “out” colour space of sRGB

Showing how to isolate a good edge using both Segmentation and Instance Segmenation together previewing beta008

Batch Processing with Natron

Schematic of How to setup batch processing with Natron

Kognat has had feedback that the processing times of a Rotobot are long, compared to the typical compositing or colour grade process which is a lot more interactive. To this end we have put up a video tutorial of processing footage with Rotobot, in an OpenSource OpenFX host called Natron, which is available for download from the internet, to run the software you will need to allocate trust to the developers, as it not certified by the operating system. The demonstration is on macOS, but if needed I can repeat the demonstration on other operating systems. It makes use of the command line interface called the “Terminal” on OSX, but more generally known as the command prompt or the shell.

This is a simple process, in large visual effect facilities this batch process would be divided up among many machines where each machine will process on frame of footage and respond that the frame is complete and ask for another frame or batch of frames to process.

Using an Free OpenSource OpenFX host means you can free up license costs. While calculating long computational cycles.

Batch processing with Natron

Getting Started in Nuke


Using Nuke is a little overwhelming at first, but in this video, I cover getting a mask to disk, using the Rotobot_InstanceSegmentation node, followed by Shuffle and then Write. This has been calculated for a sequence in advance of this video, but one frame is calculated live. The mask is then brought back in via a Read node, the alpha channel is then inverted using an Invert Node, to allow a Multiply Node to change the colour of the background.

Getting Started with Nuke and Rotobot.

Using Resolve Fusion (free version) to save masks

The video is lengthy, allow 2 minutes with fast forwarding, or 6 minutes with patience listening to my dialog about how it works and what to expect. This is updated, we bring in the footage, switch to Fusion, create the Robotbot_Segmentation node, followed by a ChannelBoolean to copy the red channel to the alpha, create a Saver.. then preview the result on the ChannelBoolean, before saving the sequence to disk.

Using Fusion within Resolve to create a Saver node to write Rotobot’s Mask to disk, in an RGBA OpenEXR.

Demonstrating the same thing is possible on Windows 10, using the Fusion Module for Resolve.

Same again, but using a “Saver” node which is designed for the job, Slower Windows 10 machine.

DaVinci Resolve Support

A few colour grading folk on social media were curious if Rotobot OpenFX Plugin works out of the box on DaVinci Resolve. I did the same thing under Windows 10 earlier in the day, but here is a recording of me putting Rotobot through its paces on a MacBook Pro under macOS 10.12 using Resolve

DaVinci Resolve slowly calculating Deep Learning Masks (note lack of watermark)

AI Collaborative Network #2

Full house at Lot 14 on North Terrace in Adelaide. A great opportunity to collaborate with other AI researchers, developers and business people. Looking forward to hear the presentations. Happy Launch Day Kognat.

Gamurs AI using computer vision and AI to analyse game play footage to improve the performance of e sports teams

Detecting subterranean vapour and liquid on Mars using ML on remote imaging and how it relates to wind patterns.

Great demo on containers and queues for non HPC based training by Adam from IBM

Frontier microscopy using “Marvin” rotobotic microscope and AI to detect asbestos in microscope samples of air filters to determine health risks. Well worth automating.

Great panel discussion

Higher Quality Masks

We have been training with segmentation rather than instance segmentation. Meaning that all the “people ” appear in one layer rather than each person split into their own layer.

The upside it is near pixel accurate on a 1080p canvas.

The downside is that it takes a couple of minutes a frame to calculate , it can be GPU accelerated given enough GPU memory and we know the requirement is more than a 4Gb card can provide.

We hope to make this available to the public within a fortnight.

Above is a screenshot of the script in Foundry’s Nuke . The deep purple node is carrying the neural network’s load the other nodes are coloring the background green.


I hope you enjoy some holiday footage from 2012.

EDIT: 16th November 2018

Here are some more samples