AI Voxel Studio
Overview
AI Voxel Studio is a creative pipeline that transforms sketches and reference images into immersive, stylized voxel worlds. Built during a 48-hour AI JAM challenge, the project bridges AI image generation, 3D Gaussian Splatting, custom USD voxelization tools, and real-time rendering in Unreal Engine 5.
The Pipeline
AI Stylization
Sketch + Reference → Stylized Image via SDXL, IP-Adapter, ControlNet
3DGS Generation
Stylized Image → Gaussian Splat (PLY) via Marble AI Lab, Apple SharpML
USD Conversion
PLY Point Cloud → Voxel USD via custom Python voxelization tools
Real-time Deploy
USD Assets → Immersive Environment in Unreal Engine 5
AI Stylization
Source image combined with a reference style produces a stylized output. The AI transfer step uses Stable Diffusion XL with H94/IP-Adapter XL for style conditioning and ControlNet for structural guidance from the original sketch.
3DGS Generation
Stylized images are converted to Gaussian Splats (PLY) using Marble AI Lab for world generation and Apple SharpML for single-object reconstruction.
Gaussian Splats Cleanup
Raw splats require cleanup before conversion. Two approaches were used — SuperSplats for manual selection and removal, and GSOPs in Houdini for a more efficient procedural workflow.
PLY-to-USD Voxelization Tool
A custom Python tool that converts Gaussian Splat point clouds (PLY format) into voxelized USD assets optimized for real-time rendering. Unlike regular PLY files with mesh topology, 3DGS PLY files contain only independent point data — each vertex represents a Gaussian splat with no connectivity.
PlayCanvas: How 3DGS PLYs Differ from Regular PLYs
Key Features
- Multiple Export Modes — Different USD structures based on use case
- Configurable Voxel Shapes — Cube, Sphere, or Cross primitives
- Color Quantization — Reduce unique colors to optimize material count
- GPU Instancing — USD PointInstancer for millions of voxels
- Hidden Face Culling — Remove internal faces to reduce polygon count
- Unreal Transform — -90 degree rotation and 100x scale for UE coordinate system
Unreal Engine Integration
Vertex Color Usage
LiDAR Point Cloud Integration
Technical Breakthroughs
- GSOPs — Procedural 3DGS optimization in Houdini
- Voxel & Point Cloud — Bridging splat data to renderable geometry
- Collision Capabilities — Voxel meshes support physics interaction
- USDC over USDA — Binary format for faster loading at scale
- PCG Potential — Procedural Content Generation with voxel building blocks
Results
