The building as it is, objects as boxes.
The reconstruction pipeline now builds the building as it is and only the things in it as boxes. Columns, stairs, ramps and curved walls take their rough real shape and the colour of the wall, floor or ceiling they belong to; furniture, vehicles and everything else that exists apart from the construction keeps one box. The host then fits the building to the predicted depth, so that the proxy space connects the way the photographs show.
Scenes used during development
Rotunda museum
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Mission church
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Palace of Fine Arts
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Meeting room (ground-truth boxes)
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Held-out scenes
The same pipeline, unchanged, on ten scenes that were not used while developing it: a courtroom, a curved auditorium, an outdoor bandshell, an apartment in Berlin, a street seen from a car, an aerial view of Cologne, a classroom, a small bedroom, a ballroom and a courthouse. Top: input photographs. Bottom: proxy from the same cameras.
Courtroom
Auditorium
Bandshell
Apartment
Street
Cologne from the air
Classroom
Small bedroom
Ballroom
Courthouse
Accuracy and time
Both versions were run in the same batches on the same machines: two to five cold reconstructions per scene each (48 frames, Pi3X, one editing session); times are means. Over the six scenes the new pipeline takes 0.92 of the time (geometric mean, 95% bootstrap interval 0.79 to 1.04). Ground-truth scores of the final code: previous release seven runs per scene, this release two (meeting room) and five (bedroom). Held-out video: twelve stretches of input photographs are taken out, the camera follows them, and the generated video is compared with the photographs (same seed, one chunk). Depth agreement is the share of input pixels where the proxy depth agrees with the predicted depth.
| measure | previous release | this release | note |
|---|---|---|---|
| Time, meeting room | 413 s | 303 s | |
| Time, office | 433 s | 369 s | |
| Time, Palace of Fine Arts | 637 s | 546 s | |
| Time, bedroom | 369 s | 351 s | |
| Time, Mont-Saint-Michel | 502 s | 528 s | |
| Time, lighthouse | 498 s | 558 s | one run of 675 s |
| Meeting room: mIoU | 0.610 | 0.639 | difference +0.029 (95% interval +0.012 to +0.046) |
| Meeting room: recall at IoU 0.5 | 0.82 | 0.97 | |
| Bedroom: mIoU | 0.672 | 0.677 | difference +0.005 (95% interval -0.009 to +0.020) |
| Bedroom: recall at IoU 0.5 / 0.25 | 0.92 / 1.00 | 0.95 / 1.00 | |
| Held-out video: PSNR | 15.60 dB | 14.92 dB | difference -0.68 dB (95% interval -1.22 to -0.06); the cause is being measured |
| Held-out video: SSIM / LPIPS | 0.556 / 0.349 | 0.539 / 0.377 | lower is better for LPIPS |
| Object boxes, mission church | 180 | 70 to 84 | seats as blocks of rows |
| Object boxes, rotunda museum | 111 | 36 to 49 | columns, stairs and balustrades are building parts |
| Seat boxes overlapping a seat box, church | 35 | 0 | |
| Depth agreement: meeting room / bedroom / office | 0.71 / 0.76 / 0.88 | 0.71 / 0.78 / 0.87 | |
| Depth agreement: Mont-Saint-Michel / lighthouse / Palace | 0.85 / 0.79 / 0.53 | 0.86 / 0.78 / 0.51 |
User requests
A request in plain words still decides the boxes for what it names; everything else follows the defaults.
- One box per row of seats (mission church): 39 row boxes instead of 12 seat blocks.
- No boxes for the pictures on the walls (apartment): 17 framed pictures left out, everything else as before.
- One box for every lamp (courtroom): each of the four lamps the planner found has its own box (without the request: none); the seats stay in 15 boxes.
Video from the new proxies
Proxy along an automatic camera path (left) and the video Mundo generates from it (right), two chunks each. The video model was not retrained for the new building parts.
Rotunda museum
Mission church
Palace of Fine Arts
Courtroom (held out)
What changed
- The proxy has two layers: the building (walls, floors, ceilings and what is built as one with them) and the objects in it. Columns, piers, beams and platforms are level building parts given by their footprint and height; stairs, ramps, sloping ground and vaults are sloping planes. They are drawn in the colours of walls, floors and ceilings, and a curved wall is drawn without lines between its facets.
- Which objects get a box is one general judgement instead of a list of categories. What someone describing the place would name keeps its box. Other things that fill less than about 2% of their best view are merged into groups or left out, so a larger place gets a coarser proxy. Open or thin bodies that a box cannot describe are left out.
- A box for a group takes the direction of the outline of all its parts, and boxes of one kind do not overlap.
- Bodies that are left out stay in the scene as invisible obstacles that camera paths keep out of.
- Before and after the agent edits the scene, the host fits the building to the predicted depth: floors and ceilings move to the observed levels and walls onto the observed surfaces, walls, floors and ceilings open where the views see through them, partly seen spaces extend as far as their floor is observed, and parts that hold an input camera or block a view that sees past them are cut back.
- A user request decides the boxes for what it names, as before.