• AI Previs Generation — Generate AI previs for Shot 3 & Shot 3B
• Team Direction — Discuss project direction and creative goals with team
• 4DGS Plugin Test — Test 4DGS plugins in Unreal & compare performance
• Interaction R&D — Research interaction mechanics for the experience
• MediaPipe Prototype — WIP: Using MediaPipe to drive plant growth (Shot 4B Interaction)

Generated AI-driven previs to help visualize composition and timing for Shot 3 and Shot 3B before committing to full production.

• Created multiple AI previs iterations for Shot 3
• Generated Shot 3B variations exploring different camera angles
• Shared previs with team for feedback and selection

Tested 4DGS (4D Gaussian Splatting) plugins in Unreal Engine and compared their performance, visual quality, and workflow integration.

• Installed and configured multiple 4DGS plugins in UE
• Compared rendering quality, frame rate, and memory usage
• Tested temporal playback stability across plugins
• Documented comparison results for team reference

Explored different approaches for real-time interaction within the experience, focusing on gesture-based and body-tracking methods.

• Surveyed existing interaction frameworks (MediaPipe, Kinect, Leap Motion)
• Evaluated gesture recognition accuracy and latency
• Identified MediaPipe as primary candidate for hand/body tracking

• AI previs for Shot 3 & Shot 3B
• 4DGS plugin comparison report
• Team direction alignment notes
• Interaction research summary
• MediaPipe plant growth prototype (WIP)

• Interaction Bug Fixes — Fix bugs in the MediaPipe interaction system
• MediaPipe Script Improvements — Add lerp smoothing & fix previous state persistence
• Daffodil Shader — Help with shader work in Unreal for the daffodil asset
• Niagara Particles — Build particle system for Shot 5 interaction

Improved the MediaPipe-to-Unreal pipeline with interpolation for smoother transitions and fixed the previous state issue that was resetting plant growth on each frame.

• Added lerp (linear interpolation) to smooth gesture-driven values
• Fixed previous state not being stored correctly between MediaPipe callbacks
• Plant growth now retains its state when hands leave the frame
• Tuned interpolation speed for natural-feeling response

Building a Niagara particle system in Unreal Engine for the Shot 5 interaction sequence.

• Integrated particle system with collision solver
• image sample from a texture to particles.

• Fixed MediaPipe interaction system
• Improved lerp smoothing & state persistence in script
• Daffodil shader progress in Unreal
• Niagara particle system for Shot 5

• Post-Process Volume Stylization — Test stylized rendering with post-process volume
• Niagara Particle — Add Y-axis movement, layered noise and wind forces
• Shot 4B Landscape — Help with landscape asset editing for Shot 4B
• Interaction Logic— Fix interaction to work across different maps in editor

Enhanced the Niagara particle system with more dynamic movement and natural behavior by adding vertical motion, multi-layer noise, and wind simulation.

• Added Y-axis velocity component for vertical particle movement
• Integrated wind force module for natural particle drift.

Optimized the interaction system architecture so it now works across different maps within the editor, improving workflow flexibility.

• Interaction now functions correctly in editor mode across all project maps
• Improved modularity for easier integration with different scenes

• Add daffodil leaves to the particle/interaction system
• Configure particle system with MediaPipe hand tracking interaction

• Post-process volume stylization tests
• Enhanced Niagara system with Y-axis motion, noise layers, and wind
• Shot 4B landscape asset edits
• Cross-map interaction system

• Shot 4B Blockout — Build layout and interaction blockout for Shot 4B
  
• SSS Material Work — Refine subsurface scattering material for grape hyacinth and leaves
• 3DGS Training Prep — Render out camera sequence for 3DGS training in Postshot

Built the layout and interaction blockout for Shot 4B, establishing camera framing, staging, and the interaction beats before committing to final assets.

• Blocked out the scene composition  for Shot 4B
• Mapped interaction beats against the timing of the shot
• Proved out the previous landscape setup in engine to confirm it holds up under the new shot requirements before building on top of it.

Continued refining Shot 3B, lighting and material changements based on the previous round of previs feedback.

Refined the subsurface scattering material for the grape hyacinth flowers and their leaves, focusing on translucency and light transport that reads naturally in motion.

• Tuned SSS color, and falloff for grape hyacinth petals
• Tested response under the current Shot 4B lighting setup

Rendered out the camera sequence needed to train a 3D Gaussian Splatting model in Postshot, feeding into the downstream 3DGS pipeline.

• Set up the capture camera sequence covering the target coverage
• Rendered the image set for Postshot training input
• Verified frame coverage and consistency for 3DGS reconstruction

• Shot 4B layout and interaction blockout
• Refined Shot 3B iteration
• SSS material pass for grape hyacinth and leaves
• Camera sequence rendered for Postshot 3DGS training

• Interaction Setup — Set up the interaction part for the new layout
• Setdress Support — Help with setdress across the scene
• Texture & Shader Troubleshooting — Diagnose and fix texture and shader issues

Set up the interaction part on top of the re-directed layout, wiring it into the updated scene so the existing interaction logic carries over cleanly.

Worked on a vine generator to procedurally create growth patterns that wrap around scene geometry, giving the team a flexible way to drape vines on architecture and props.

• Re-directed layout and landscape
• Interaction setup wired into the new layout
• Setdress contributions across the scene
• Working vine generator prototype
• Texture and shader fixes

• 3DGS Pointcloud Overlay — Test 3DGS with pointcloud overlay using the XYZ tool
• Statue & Gazebo Placement — Finalize the placement of the statue and gazebo in the scene
• Interaction Camera — Set up the camera for the interaction

Tested 3DGS with a pointcloud overlay using the XYZ tool to evaluate how the splat reconstruction lines up against the source pointcloud and to spot drift, holes, or alignment issues before relying on the splat downstream.

• Loaded the trained 3DGS asset alongside its pointcloud
• Used XYZ tool to overlay both representations and compare
  

Locked in the placement of the statue and gazebo, treating them as anchor props that drive both composition and interaction staging in the scene.

• Iterated on statue position and orientation against camera framing
• Placed gazebo to support sightlines and interaction flow
  

Set up the camera for the interaction so the user’s gesture-driven moments land in frame as intended, with the framing tuned around the now-locked statue and gazebo.

Deliverables

• 3DGS pointcloud overlay test using XYZ tool
• Finalized statue and gazebo placement
• Interaction camera setup

• Vine Growth Animation — Help with the vine growth animation and research interaction potential
• Statue VAT Interaction — Add statue interaction using VAT driven by MediaPipe
• 3DGS Capture & Cleanup — Capture and clean up 3DGS from Unreal Engine; pointcloud overlay and UDS voxel
• VP Shoot Tech Doc — Prepare tech documentation for the VP shoot, including MediaPipe and OSC spec

Supported the vine growth animation work and explored how the growth could become an interaction beat rather than a purely scripted moment.

• Iterated on growth timing and curve shaping with the team
• Researched interaction hooks that could drive the growth via gesture input

Added a new statue interaction using Vertex Animation Textures driven by MediaPipe input, so hand-tracked gestures can trigger and scrub the statue animation in engine.

• Wired MediaPipe gesture data into the VAT playback parameters
• Tuned response so the interaction reads the statue motion

Captured 3DGS from Unreal Engine and ran a cleanup pass, using pointcloud overlay and UDS voxel to validate the result and trim noise from the splat.

• Captured the source data out of Unreal for 3DGS training
• Performed cleanup pass on the trained splat
• Used pointcloud overlay to verify alignment against the source geometry
• Applied USD voxel to further isolate clean regions of the splat

Prepared the technical documentation for the upcoming VP shoot, covering the MediaPipe pipeline and the OSC message spec so the on-set team can wire everything up without guesswork.

• Documented the MediaPipe capture and routing setup
• Wrote the OSC message spec: addresses, payload format, expected ranges
• Shared the doc with the team ahead of the shoot

• Vine growth animation support and interaction research notes
• Statue VAT interaction driven by MediaPipe
• Cleaned 3DGS capture with pointcloud overlay and UDS voxel validation
• VP shoot tech doc covering MediaPipe and OSC spec

• VP Load Test — Run VP load test and convert MP4 to MOV HAPQ for disguise
• 3DGS in Unreal — Integrate 3DGS in Unreal using Volinga and SplatRenderer, add FX
• LED Wall Streaming Build — Prepare a clean version with LED Wall setup for streaming
• Shoot Day — Finalize shoot list and shoot on Tue 5/12

Ran a VP load test to validate playback on the disguise side, including converting MP4 deliverables into MOV HAPQ so the media plays back cleanly through the disguise pipeline.

• Converted MP4 sources to MOV HAPQ for disguise ingest
• Loaded media into disguise and verified playback stability
• Logged any frame or color issues to address before the shoot

Integrated 3DGS assets into Unreal using both the Volinga plugin and SplatRenderer, then layered additional FX on top of the splats in Unreal so the splat content sits inside the scene rather than reading as a flat insert.

• Brought 3DGS assets into Unreal via Volinga
• Cross-checked rendering via SplatRenderer for comparison
• Added FX passes over the splats in Unreal to tie them into the scene

Finalized the shoot list and shot on Tuesday, May 12, executing the planned coverage against the VP / LED Wall setup prepared earlier in the week.

• Finalized shoot list with the team
• Shot on 5/12 against the prepared LED Wall.

• VP load test completed with MP4 → MOV HAPQ conversions for disguise
• 3DGS integrated in Unreal via Volinga and SplatRenderer with added FX
• Clean LED Wall streaming build
• Finalized shoot list and completed 5/12 shoot

Continued building the browser-based WebGL viewer for the 3D Gaussian Splatting content, and added mouse click interaction so users can engage with the splat directly in the browser.

• Added mouse click interaction on the WebGL splat
• Wired the interaction into the existing viewer state

Added a visualization of the captured camera data inside the viewer, and layered a post-process pass over the WebGL render to lift the overall image quality.

• Visualized the camera data (capture positions / frustums) in the viewer
• Added a post-process pass to the web viewer

Authored a tech breakdown for each of the main assets, and added a realtime opening sequence rendered directly in WebGL.

• Added a tech breakdown for each main asset
• Built a realtime opening sequence in WebGL

• WebGL 3DGS web viewer with mouse click interaction
• Camera data visualization in the viewer
• Post-process pass added to the web viewer
• Tech breakdown for each main asset
• Realtime opening sequence in WebGL