Sketch to Screen

Technical Research & Implementation Strategy for OpenUSD Challenge • Spring 2026

Project Description

AI Voxel Studio is a creative pipeline that transforms sketches and reference images into immersive, stylized voxel worlds.

Role: Pipeline Ideation, Tool Development, Unreal Integration

Video: City Sequencer

The Pipeline

01 - AI Stylization: Sketch + Reference → Stylized Image | SDXL, IP-Adapter, ControlNet

02 - 3DGS Generation: Stylized Image → Gaussian Splat (PLY) | Marble AI Lab, Apple SharpML

03 - USD Conversion: PLY Point Cloud → Voxel USD | Custom Python Tools

04 - Real-time Deploy: USD Assets → Immersive Environment | Unreal Engine 5

Process

Sketch

Reference

StyleOutput

Source image + reference style ︎︎︎ Stylized image

Tools Used

- Stable-diffusion-xl

- H94/IP-Adapter XL

- ControlNet

Process

Image → Gaussian Splat (PLY)

Marble AI Lab
Apple SharpML

World Generator in Marble

PLY Result from Sharp-ml

Gaussian Splats Cleanup

Super Splats

Inport in SuperSplats

Rect Select

Manually Cleanup

GSOPs in Houdini - More effecient, more flexible

PLY-to-USD Voxelization Tool

Research:

How 3DGS PLYs Differ from Regular PLYs: PlayCanvas Documentation

Unlike regular PLY files that contain vertices and faces defining mesh topology, 3DGS PLY files contain only point data with no connectivity information. Each "vertex" represents an independent Gaussian splat.

Tool Development

Core

GUI

Result

A Python-based tool that converts Gaussian Splat point clouds (PLY format) into voxelized USD assets optimized for real-time ren

Key Features

- Multiple Export Modes: Support for different USD structures based on use case

- Configurable Voxel Shapes: Cube, Sphere, or Cross primitives

- Color Quantization: Reduce unique colors to optimize material count

- GPU Instancing: Leverage USD PointInstancer for millions of voxels

- Hidden Face Culling: Remove internal faces to reduce polygon count

Transform for Unreal

The USD includes a -90 degree rotation and 100x scale to match Unreal's coordinate system.

Unreal Engine 5 Integration

1. Vertex Color Usage

2. LiDAR Point Cloud Integration

Key Discoveries

• GSOPs - Optimize 3DGS

• Voxel and Point Cloud

• Collision Capabilities

• USDC instead of USDA

• Potential to modulate (PCG)

Final Results

Still

Video

TBD

Meet the Team

Danci Shen | Pipeline Ideation, USD Tool, Unreal Integration | https://omolism.cargo.site/

Itim Kongsakulvatanasook | AI Transfer Tool, Houdini GSOPs | https://itimkongs.com/

Ben Jones | Real-time Rendering, Post-process Tool, PCG | https://www.benjvisuals.com/

Future Directions

TBD

References

- Pixar USD Documentation

- Unreal Engine USD Plugin

- 3D Gaussian Splatting

- Apple SharpML