Overview

OpenProbe reads a photograph the way a gaffer and a look dev artist read a plate, and it does the reading with physics rather than vibes. Drop in an image and its exposure metadata becomes a calibrated light meter: colour temperature, illuminance in lux, per source luminance in nits, and OpenPBR material estimates, all exportable as a ready to light USD rig. Everything runs client side in the browser, so the photographs never leave the machine.

The Photograph as Instrument

A camera meters a scene, decides that the average should land on middle grey, and encodes that decision in three numbers. Run the reflected light meter equation backwards and those numbers give back the luminance that was actually in front of the lens:

• Luminance, L = (Y / 0.18) times K times N squared, divided by ISO times shutter time, in candela per square metre
• Illuminance, E = pi times L divided by reflectance, the Lambertian bridge from what a surface emits to what falls on it
• Color, a bright band white point through CIE xy to McCamy CCT, with Duv from the Krystek locus

Two hardening passes were needed before the colour read could be trusted at all:

• Sky rejection, bright blue pixels in the upper frame are excluded. The sky's own chromaticity sits between 10,000 and 20,000 K while everything below it is lit by roughly 5,500 K sun, and a midday shot read 9,336 K until the sky stopped voting
• Halo sampling, a blown source sits at the sensor white point and reads as fake daylight, so the colour is taken from the glow ring just outside it. A rattan pendant that reported 6,276 K reports 2,147 K once measured from its halo

Perception That Stays Out of the Way

Four models run entirely on device, lazily loaded and cached, on WebGPU where it exists and WASM where it does not. The design rule throughout: models are infrastructure the user never configures. There is no mode to pick, no toggle to find. One gesture on the photograph, and the right path is chosen silently.

• RT-DETRv2 at 21 MB for object detection, chosen over YOLO because Apache 2.0 lets the tool ship closed, and because its accuracy roughly doubles the previous detector
• SegFormer on ADE20K for plane knowledge, walls, floors, tables, mirrors, running invisibly on every photo
• Depth Anything for relative depth, feeding the perspective solver and the depth view
• CLIP ViT-B/32 for zero shot material class across a 32 class taxonomy built from MINC crossed with production categories

Plane Knowledge, Invisible

Segmentation was originally a visible mode with hover highlights and click to sample. It got deleted, because a wall sized bounding box crosses lighting gradients, window openings and furniture, and a crop like that has no business becoming a basecolor map. The useful half survives, silently:

• A drag box landing mostly on one large plane picks up that plane's mask automatically
• The material names itself wall_01 from the plane it was read on
• Everything standing in front of the wall is excluded from the statistics and from the exported texture
• No mode, no toggle, no announcement. The user drew a box

Detector Labels Are Not Truth

COCO has eighty classes and none of them is "product packaging", so a box of oatmeal gets called a book. Rather than pretend otherwise, known catch all labels get a second opinion: the crop is re judged by CLIP against honest rivals, packaging against book against carton against print, and the winner renames the box. Detections show for a beat, then tuck into a chip that reads AI suggestions, for reference, a hand drawn box reads truest.

From Bright Pixels to a Lighting Rig

Finding the lights took two corrections, and the second was caused by the first:

• Brightness is the wrong question. A warm paper lantern in a graded interior peaks at 0.4 linear and is unmistakably a lamp; a white studio wall hits 0.9 everywhere and emits nothing. The gate became local contrast, a source defined by outshining its own surroundings against a large radius background estimate
• Loosening the bar let every glint through. A desk photo came back with eight "lights", of which two were lamps and six were highlights on glass bottles
• The separator is physical. A real source spills a halo, decaying smoothly across several rings into the air and the lens; a specular highlight is a mirror image sitting on a surface and ends in a cliff. Comparing an inner ring against an outer one across the source boundary drops the reflections and keeps the lamps

Typed Lights, Not a Pile of Spheres

A lighting artist does not create "detected bright thing", they create a rect light for a window, a sphere for a bulb, a disk for a downlight, a dome for the ambient wrap. OpenProbe classifies each source into that vocabulary from its shape and exports the matching UsdLux prim, so the rig arrives usable in Omniverse and Blender instead of arriving as geometry homework.

• RectLight, windows, softboxes, light panels, carrying measured width and height
• SphereLight and DiskLight, bulbs and downlights, radius from measured extent
• DistantLight, the sun, inferred from a sunlit patch rather than from the sun itself, with time of day and altitude bands read from atmospheric colour
• DomeLight, the ambient wrap, the piece a relight cannot do without
• Off frame key and fill, read from shadow weighted luminance gradients per frame half, exported aimed at the measured bearing

Intensity semantics matter more than they sound. Omniverse RTX reads a DistantLight's intensity as illuminance in lux, but an area light's as luminance in nits. Writing a lumen figure into that field, which the tool did until a real Omniverse session caught it, lands an order of magnitude dark. The measured nits value goes in the intensity field, approximate flux rides alongside as custom attributes, and every source row shows the paste ready number.

Materials

Drag a box on any surface and it is estimated as an OpenPBR material:

• Albedo, de lit against a local illumination anchor rather than a global average. A face in shadow inside a high key frame read as dark purple until the neighbourhood got a vote
• Roughness, from the specular highlight's half width. An honest proxy that degrades on textured surfaces and says so
• Metalness, from highlight tint, since metals colour their reflections and dielectrics do not, with a separate path for polished metal reflecting its environment
• Class priors, the reflexes a look dev artist would apply anyway: clearcoat on varnished wood, anisotropy on brushed metal, fuzz on fabric, subsurface on skin
• Scattering radii in millimetres, because a subsurface weight without a radius is just grey haze. Red travels furthest through flesh, which is why ears glow; marble scatters shallow and evenly under a coat; snow scatters deep and neutral. Those three numbers are the difference between skin and wax

Perspective, Solved Three Ways

A floor photographed at an angle keystones its texture, and tiling that crop produces garbage. Three sources contribute, ranked by how much they actually know:

• Geometry first, line segments clustered into orientation families, each family's vanishing point solved by exhaustive enumeration. Deterministic by construction, after an earlier randomised build could answer differently for identical input
• Depth second, as the planarity gate and axis source
• Texture last, since transverse texture period shrinks linearly to zero at the vanishing line
• None of the three, and the crop ships flat with a note rather than a guessed warp

The same restraint applies to resolution. Once a plane is segmented the tool can widen a sample past the box that was drawn, and for a long time it always did, on the reasoning that more pixels is more material. That reasoning is wrong on a receding surface: a plane in a photograph is one label, not one viewpoint, so the largest clean square on a wall may sit at a completely different scale than the box the artist pointed at, and the extra resolution gets paid for with a jog at every seam. Now both windows are measured and the better seam agreement wins, which on the specimen tile means the tool declines its own upgrade.

Every photometric number downstream of a light source rides on one question: how far away was it. Emitting area is solid angle times distance squared, so distance enters the flux squared, and the three-to-five times band the tool used to print was never really about light. It was the cost of guessing a lamp's distance from a prior about the lamp's own size, which is a guess about the very thing being measured.

A metric depth network is the obvious move and the wrong one twice over: it would answer every pixel whether or not it knew, and it would be a second heavyweight download on a tool whose whole promise is that the photograph never leaves the machine. The better answer was already resident. The detector runs on every photo anyway, and some of its classes carry a genuinely tight real world size. A toilet is 78 cm tall in every bathroom on earth, a stop sign is 75 cm across by regulation, a standing adult is 1.70 m give or take seven percent. One of those in frame, plus the focal length, turns a pinhole projection into a ruler.

• A class qualifies only if its spread is tighter than the answer it would inform, which rules out televisions, beds and potted plants
• A box clipped by the frame edge is a lower bound, not a measurement, so it is rejected, but only on the axis being measured. A fridge running off the left of the frame is still exactly as tall as a fridge
• A person counts only standing, because a seated or mid stride bounding box measures something other than height
• References combine in log space, weighted by their own spread, and when independent references disagree the interval inflates instead of pretending
• Two references at different depths pin the whole depth map to metres, and every source in the frame gets its own distance rather than sharing one

On the garage specimen two bicycles put the scene at 15.65 m inside a 13.83 to 17.70 m band, and the brightest unclipped source moves from a nine times spread, 303 to 2726 lumens, to a spread of 1.64, 4860 to 7953. The midpoint moved too, from 909 lumens to 6217: the old fixture size prior was not merely uncertain, it was biased, placing an exterior lamp at 5.98 m when the driveway furniture says 15.65. A blown out core keeps its wide band regardless, because no distance measurement recovers a luminance the sensor never recorded. When nothing usable is in frame the tool says so and the assumption stands, labelled as an assumption. The interval got smaller because the evidence got better, which is the only honest reason for an interval to move.

Handoff

The output is designed for the first ten minutes of a real shot, not for a screenshot. USD arrives with typed UsdLux prims and OpenPBR materials under a Looks scope following the Kit convention, bound to preview meshes so materials are visible the moment the file opens. glTF carries punctual lights in physical candela and lux. JSON includes per engine hint blocks: watts for Blender, lumen and Kelvin for Unreal, physical units for Unity HDRP.

• USD, UsdLux rig plus UsdPreviewSurface materials, custom openprobe attributes preserving the full OpenPBR set
• MaterialX 1.39, open_pbr_surface with the texture network wired, including displacement
• Material package, basecolor, normal, height and ORM maps that tile where the surface has a lattice to lock onto, USD, MaterialX, and a README stating conventions, texel density and provenance
• Blender importer, builds the ambient as world lighting, lamps as their matching types, and sets blend modes so transparent materials actually read as transparent
 

Stack and Credits

• Application, React 19, TypeScript, Vite, Zustand, GSAP
• On device inference, transformers.js v3 with WebGPU and WASM fallback, RT-DETRv2, SegFormer, Depth Anything, CLIP, Swin2SR
• Standards, OpenPBR Surface, OpenUSD and UsdLux, MaterialX 1.39, glTF with KHR extensions
• Photometry, ISO 2720 reflected light metering, McCamy CCT, Krystek Planckian locus
• Delivery, static site on the Cloudflare Pages global edge, no server, no upload

Built solo, from the physics outward. The accuracy programme, a ground truth error matrix against calibrated meters and a colour card test set, is the honest next milestone and is named as such inside the tool's own method page.