Cyborg Collective · Scans in 3-D

How a scan is made

From a headset's pictures to a Gaussian splat, a solid and a print file: the chain, what has been checked and what has not, and every tool as a plain file.

A thing or a place is photographed with a Meta Quest 3S headset. A laptop turns the photographs into a Gaussian splat and, for an object, into a closed solid sliced for a 3-D printer, and the result goes onto this site. Everything it is made with is on this page.

.\start_quest_scan.ps1scan, splat, solid, print file .\start_quest_scan.ps1 -Live -Publishthe same, live on this site while scanning and published here afterwards

One command on the laptop the headset is plugged into. Without -Live or -Publish everything stays on that laptop; nothing is ever sent to a printer.

6ense™ the chain · from what a headset sees to something that can be held

Blue is what a laptop makes after the scan; green goes out while the headset is still scanning; gold is this site.

The chain from a headset to a print file and to this site A headset, worn, keeps pictures with its position and depth rays. A laptop trains a Gaussian splat from them, makes a closed solid from the splat and slices it into a print file. While scanning, live Gaussians, one per depth reading, go to this site. Afterwards the trained splat, compressed, goes to this site too. headset a Quest 3S, worn pictures with position + depth rays splat trained by Brush, ~2 min solid closed, flat base print file sliced, Bambu H2C live Gaussians one per depth reading compressed Open3D, .spz, a tenth this site live · scans

The chain

Six steps. The two with a dashed edge happen only when asked for, because they are public.

  1. The headset keeps picturesIts app keeps a picture from the passthrough camera every 5 cm or 6 degrees of head movement, with the headset's own position and direction for it and an 80 × 60 grid of depth rays behind it. Pulling the trigger on a thing names it as the object. The headset has no lidar: its depth comes from its two tracking cameras.
  2. Live, if asked forWhile it scans, the app also sends one depth reading per ray to the laptop over the USB cable. The laptop keeps one per 2 cm of surface and passes them through the relay; the live page draws each as a Gaussian the moment it arrives. These are placed, not trained.
  3. The laptop trains the splatThe headset's positions are used as they are, with no structure from motion. The pictures are cropped round the object and Brush trains the Gaussians for 8,000 steps, about two minutes on a laptop's graphics card. A tenth of the pictures are kept back and compared with what the splat draws from their positions.
  4. From splat to solidThe splat's depth is drawn from every camera position and voted into a grid: outside where enough cameras see past a cell, surface where they see it, solid where none could see, so the body is closed. It is cut flat on the surface it stood on.
  5. SlicedThe solid is scaled and sliced for a Bambu H2C with the slicer's command line, which reports the time and the grams. Nothing is sent to a printer.
  6. Published, if asked forOpen3D reads the trained splat and writes it compressed, about a tenth of the size; that file, the solid, a render and the print files go into this site's repository. Open3D does not make the splat: it has no training. The camera pictures are not published.

What has been checked, and what has not

0.47 mmT. rex: half the surface within this of the true shape
0.37 mmTriceratops, the same measure
0.3 %T. rex: its sizes, against the true ones
0.5 %T. rex: its volume
0.02 pxreal depth rays against the camera model, sideways
26pictures in the one sitting on the headset so far
Against known shapesTwo made-up scans of known models, drawn at the headset camera's geometry with noise in the depth, went through the same steps as a real scan. The solids are measured against the models they were drawn from. Both are on the list, labelled as tests.
On the headsetOne sitting so far, from one seat in a room: a view to look round rather than an object. Its depth rays land within 0.02 px sideways and 4 px vertically of where they were cast, so the camera model holds on the device.
These pagesEach has a browser test that travels with it (test_scans_page.py, test_live_page.py). The live page's test sends a made-up room through a relay and requires the page to draw it and let it go.
Not yetA walk all the way round one object with the headset; the live view from the headset itself (only made-up rooms have been sent so far); AR and VR inside the headset from this site; a print.

The tools

Plain files, as they are used; nothing here is built. Python (numpy, scipy, numba, Open3D, scikit-image, Pillow, playwright), PowerShell, and one C# script for Unity 6 with Meta's XR packages. Their defaults name folders in the user's home folder. The viewer is view.html and the live page live.html: their source is the page itself, with scans.css.

On the headset

PrintWhatISee.csThe headset app (Unity, Meta's passthrough camera and depth): keeps a picture, the camera's position and a grid of depth rays every 5 cm of head movement; the trigger names the object; the splat sensor sends one reading per depth ray for the live view.
PrintSeeBuild.csBuilds the app's scene and its Android package.
compile_check.pyCompiles the app's scripts in seconds, without a build.
printsee_build.ps1Runs the Unity build without a window, watching the disk.

One sitting

start_quest_scan.ps1The one command: scan, splat, solid, print file; -Live and -Publish send to this site.
scan_session.pyWaits for the headset to be worn, starts the app, follows its log, then runs the steps below.
quest_scan.pyThe laptop's part: checks the scan, writes the training set, trains the splat (Brush), cuts out the object, makes the solid, slices it, draws the pictures.

From a phone

phone_scan.pyA phone's recording (a ROS 2 bag from MuseLog's Scan in 3-D) to a splat: takes the pictures out, works out where each was taken from (structure from motion), sets up, middle and a guessed size, trains, draws the pictures.
phone_scan_worker.pyRuns on a computer with a graphics card: asks the relay for scans that phones sent, makes the splat of each, publishes it here, and takes down what its owner removed.

The parts of the laptop's work

scan_lib.pyReads a scan folder; the headset camera's model; writes the training set in COLMAP's layout; reads and writes splat files.
splat_render.pyDraws a Gaussian splat on the processor: colour, depth and coverage from any camera position.
scan_solid.pyFrom splat to solid: the surface the object stands on, the object's own Gaussians, depth from every side voted into a closed body with a flat base.
slice_h2c.pySlices an STL for a Bambu H2C with the slicer's own command line; reports time and grams.
scan_view.pyServes the viewer (view.html on this site) on the laptop and to the headset through the USB cable.
fetch_vendor.pyFetches the two libraries the viewer runs on, checked against recorded hashes.

A room seen from one spot

scan_view_folder.pyGathers an early sitting's pictures into one folder.
scan_mono_depth.pyDepth for pictures that have none: a depth network's map per picture, pinned to the headset's own measurements.
scan_room_model.pyFuses those depth maps into a coloured shell of the room and a diorama to print.

Checking the method

scan_synth.pyDraws a made-up scan of a known model, at the headset camera's geometry and with noisy depth.
scan_gate.pyMeasures the result against the known model and says pass or fail.

To this site

scan_publish.pyPuts a finished scan here (the splat compressed by Open3D), takes one off, and writes these pages.
scan_pages.pyThe words and the markup of the list of scans and of this page.
quest_splat.pyThe live view's sender: takes the app's depth readings over the USB cable, keeps one per 2 cm, records them, passes them to the relay.
start_quest_splat.ps1The live sender on its own, without a scan.
splat_live_check.pySends a labelled test pattern through the public relay and checks that the home page draws it and lets it go.
SCAN_README.mdThe notes kept with the tools: how to run them, every file, the method, what was checked and what was not.

What they stand on

Brushtrains the splat (Apache 2.0)
Open3Dcompresses it to .spz (MIT)
Sparkdraws Gaussian splats in the browser (MIT; kept in vendor/)
three.jsthe 3-D underneath (MIT; kept in vendor/)
Meta's XR interfacesthe headset's passthrough camera and depth
Bambu Studioits command line slices the solid