The building as it is, objects as boxes.
The reconstruction pipeline now builds the building as it is and only the things in it as boxes. Columns, stairs, ramps and curved walls take their rough real shape and the colour of the wall, floor or ceiling they belong to; furniture, vehicles and everything else that exists apart from the construction keeps one box. The host then fits the building to the predicted depth, so that the proxy space connects the way the photographs show.
Scenes used during development
Rotunda museum
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Mission church
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Palace of Fine Arts
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Meeting room (ground-truth boxes)
Top: input photographs. Middle: proxy of the previous release. Bottom: proxy of this release, from the same cameras.
Held-out scenes
The same pipeline, unchanged, on ten scenes that were not used while developing it: a courtroom, a curved auditorium, an outdoor bandshell, an apartment in Berlin, a street seen from a car, an aerial view of Cologne, a classroom, a small bedroom, a ballroom and a courthouse. Top: input photographs. Bottom: proxy from the same cameras.
Courtroom
Auditorium
Bandshell
Apartment
Street
Cologne from the air
Classroom
Small bedroom
Ballroom
Courthouse
Accuracy and time
Wall clock of a cold reconstruction (48 frames, Pi3X, one editing session), mean over the runs made. Ground-truth scores on the two scenes that have ground-truth boxes, mean over three runs (two for the previous release on the bedroom). Depth agreement is the share of input pixels where the proxy depth agrees with the predicted depth.
| measure | previous release | this release | note |
|---|---|---|---|
| Time, meeting room | 407 s | 386 s | |
| Time, bedroom | 368 s | 424 s | slower: longer editing sessions |
| Time, office | 448 s | 367 s | |
| Time, rotunda museum | 925 s | 720 s | |
| Time, mission church | 1068 s | 942 s | |
| Time, Mont-Saint-Michel | 511 s | 532 s | |
| Time, Palace of Fine Arts | 613 s | 513 s | |
| Time, lighthouse | 509 s | 486 s | |
| Meeting room: mIoU / recall at IoU 0.5 | 0.618 / 0.824 | 0.649 / 0.882 | |
| Bedroom: mIoU / recall at IoU 0.5 | 0.675 / 0.923 | 0.669 / 0.949 | |
| Object boxes, mission church | 180 | 70 to 84 | seats as blocks of rows |
| Object boxes, rotunda museum | 111 | 36 to 49 | columns, stairs and balustrades are building parts |
| Seat boxes overlapping a seat box, church | 35 | 0 | |
| Depth agreement, rotunda museum | 0.83 | 0.81 to 0.84 | |
| Depth agreement, mission church | 0.85 | 0.87 to 0.88 | |
| Depth agreement, Palace of Fine Arts | 0.51 | 0.49 | outdoors more is left out |
User requests
A request in plain words still decides the boxes for what it names; everything else follows the defaults.
- One box per row of seats (mission church): 39 row boxes instead of 12 seat blocks.
- No boxes for the pictures on the walls (apartment): 17 framed pictures left out, everything else as before.
- One box for every lamp (courtroom): each of the four lamps the planner found has its own box (without the request: none); the seats stay in 15 boxes.
Video from the new proxies
Proxy along an automatic camera path (left) and the video Mundo generates from it (right), two chunks each.
Rotunda museum
Mission church
What changed
- The proxy has two layers: the building (walls, floors, ceilings and what is built as one with them) and the objects in it. Columns, piers, beams and platforms are level building parts given by their footprint and height; stairs, ramps, sloping ground and vaults are sloping planes. They are drawn in the colours of walls, floors and ceilings, and a curved wall is drawn without lines between its facets.
- Which objects get a box is one general judgement instead of a list of categories. What someone describing the place would name keeps its box. Other things that fill less than about 2% of their best view are merged into groups or left out, so a larger place gets a coarser proxy. Open or thin bodies that a box cannot describe are left out.
- A box for a group takes the direction of the outline of all its parts, and boxes of one kind do not overlap.
- Bodies that are left out stay in the scene as invisible obstacles that camera paths keep out of.
- Before and after the agent edits the scene, the host fits the building to the predicted depth: walls move onto the observed surfaces, walls, floors and ceilings open where the views see through them, partly seen spaces extend as far as their floor is observed, and parts that hold an input camera or block a view that sees past them are cut back.
- A user request decides the boxes for what it names, as before.