Cuebric: The Breakthrough Generative A.I. Tool For Virtual Production
Real-time asset production, ideation, previs, and location scouting tool for volume stages
Today, creative A.I. production company Seyhan Lee launched Cuebric, an all-in-one tool for filmmakers that lets them generate, segment, inpaint, edit, and upscale any landscape they wish to see on set. I know it sounds near insanity, but creators on set will now be able to wave their hands, tell the Cuebric prompt operator: “Bring me to Iceland, but on Mars, with a bunch of pyramids,” and be ready to film the actors on a fully prepared, 2.5-dimensional background.
What is virtual production?
For readers new to the term “virtual production”, let me give a quick introduction. The method of producing and recording movie and television visual effects in a room with a curved LED stage and real-time technology is known in the film industry as “virtual production.” We can also refer to it as “the new green screen” in simple laymen’s terms.
In pre-virtual production times, when the director wanted to see the actor in an outlandish land, the actor had to be filmed in front of a green screen, where the CGI department’s work would be integrated after the filming. Whereas now, the worlds that make up the movie are created before the filming date, ready to be placed on the LED volume stage. In this way, instead of acting in front of a green screen and having to imagine Mars, the actors can see and feel the environment. Meanwhile, directors who want dusk/dawn light can hold the set for a full day, and producers can change the location from Alaska to Hawaii during a coffee break.
Dune (2021), Batman (2022), Loki (2021), and Black Adam (2022) are among the well-known films and television series that have used virtual production. However, the technology was popularized mainly by the Mandalorian in 2019. Please watch below as creator Jon Favreau, Industrial Light and Magic, and their partner Epic Games (Unreal Engine) explain how they filmed Mandalorian Season 1 with virtual production.
2.5 Dimension versus 3 Dimension on volume stages
For virtual production to work, the content on the volume stage needs to be dimensionalized. For the scene to be credible, the background must also move left with the actor when the actor moves left. The Unreal Engine is used on set by CGI environment designers to render the assets they’ve already made in real-time. These backgrounds are three-dimensional, just like in a game. Creating realistic, three-dimensional Hollywood-quality backgrounds is not for the faint-hearted. Each scene typically requires an army of CGI artists to work for months.
If 3D is the option that can be fully controlled, then the volume stages also let you film in what is called “two and a half dimensions,” which offer less control over movement. Simply put, a 2.5-D image is a two-dimensional image with a parallax effect. Cuebric was made so that these 2-D landscapes could be built on set in real-time. It is a quick, cost-effective alternative to labor-intensive 3-D environments and works well with them.
How does Cuebric work?
Cuebric includes five tools in one for a director to use in real-time to manifest their immediate idea or vision on a volume stage. What are these required functionalities for an image to be generated and end up as a layered, ready, be-filmed 2.5-D asset in Unreal Engine? Let’s discover.
Step #1 ━ Image generation and image upload
Cuebric uses Stable Diffusion as its image-generation engine. The tool enables artists to generate images from any prompt at 4K, 8K, and even 16K resolution. To turn the director’s idea into a film-like setting, a prompt artist with experience making film environments is needed. Seyhan Lee will provide prompt artists in the early phases of the tool. The user can also upload an already-made 2D asset to the tool.
Step #2 ━ Segmentation
A 2-D image must be segmented into parts to become 2.5-D. Team Seyhan Lee has used open-source datasets of landscapes, cities, objects, people, and other things to teach Cuebric how to segment the generated image into layers. The tool suggests different parts of an image to cut out, which the user will then put back together into layers.
Step #3 ━ Inpainting
When an image is segmented (cut out) into different parts, it naturally creates holes (white space) when the layers are segmented on set. The artist can either fill these gaps by reconstructing the missing parts of the image or generate completely new pieces with the help of Stable Diffusion inpainting.
Step #4 ━ Editing
In the first version of Cuebric, a compact image editing tool lets users get rid of unwanted artifacts on the segmented layers or fill in the gaps with creative content made by the program. In the next version of Cuebric, there will be a web-based image editor that can do all the essential things that Photoshop can do, like layering, masking, inpainting, and green screen choking. These features allow the artist to make Cuebric the first and only step in generating 2.5D scenes.
Step #5 ━ Image generation and image upload
For volume stages, Cuebric lets the generated plates and uploaded images be scaled up to a resolution of up to 16K.
Step #6 ━ Export to Unreal Engine
Once the plates are done, users can use built-in tools to import their work into the Unreal Engine immediately. All it takes is a click of a button…
…ready to be filmed on the virtual production stage!
Cuebric can be used as a concept, a location scouting, and a previs tool. On the other hand, it will be helpful to creators and productions with a more flexible workflow. It’s not necessary to work on a volume stage, but directors may use it with concept and previs artists to come up with ideas and get plates ready for the shoot day.
Cuebric is designed to be a partner for 3D CGI assets. During a production, not each scene requires a CGI army; for these 15–20 scenes in between shots, behind dialogue, Cuebric will save the production from excessive work.
I have read in this article by Elad Gil that for a revolutionary A.I. tool to penetrate the market, it needs to be ten times better, cheaper, or faster. I agree with this. When company owners, producers, and filmmakers get out of bed, they do not fetch their assistants with the desire to experiment with novel products so that they can destroy their workflow and take time to learn a new tool. 😊 However, when the tool saves teams and production ten times the cost and effort, the industry adapts to a new workflow. Film production is not an exception. In the following years, all creative industries will be transformed by generative A.I. Cuebric is the first wave of this new creative revolution.
Cuebric is the first step into a truly real-time and A.I.-driven on-set production solution. It combines the power of generating images with clever 2.5-D projections in real-time. I cannot wait to use it more.
— Johannes Saam, Academy Award winning VFX artist.
XR Studios, Cuebric’s launch partner
Cuebric launched in partnership with XR Studios, a production company that focuses on technology and is known as a leader in making immersive tech workflow solutions. The studio is known for making solutions for how to work with Extended Reality (XR) and Augmented Reality (AR). Billie Eilish, Katy Perry, Riot Games, Post Malone, Kid Cudi, the MTV Music Awards, American Idol, Amazon, and others have performed on the XR Studios Hollywood stage.
Cuebric’s availability
In January, only a small number of production stages and creative content developers will be able to use Cuebric. Since the tool needs to be "prompted" to make an image, it needs to be used by people who have around 50,000 iterations of experience making environments so that the director's vision can be carried out correctly on set. While Seyhan Lee's team was making Cuebric, they were also training people to be prompt experts so that they could offer prompting services along with the tool.
If you would like to test Cuebric on your production or volume stage, please get in touch with team Seyhan Lee directly: cuebric@seyhanlee.com
Dear reader, thank you for taking the time to read this article. These industry-changing novel ways of motion picture creation spread to others when you engage with my thoughts socially, so thank you for subscribing to my Medium channel and sharing this article with your following.
With love, Pinar.
