A Digital Sense Scientific Insight Series article on three landmark computer vision papers, VGGT (best paper CVPR 2025), VGGT-Ω (best paper finalist CVPR 2026), and D4RT (best paper CVPR 2026), and what the shift toward feed-forward neural reconstruction means for applied AI.
For most of its history, 3D computer vision has been a discipline of careful geometry. Recovering the shape of a scene from photographs meant matching features across images, triangulating points, and refining everything through iterative optimization. Those methods work, but they are slow, fragile, and demand expert tuning. Over the last two years a different idea has taken hold: train a large transformer on enough 3D-annotated data, and it predicts scene geometry directly in a single forward pass, often more accurately and faster than the pipelines it replaces.
Three papers chart this transition. VGGT (Visual Geometry Grounded Transformer), from the Visual Geometry Group at the University of Oxford and Meta AI, won the Best Paper Award at CVPR 2025. The same group then introduced VGGT-Ω, a best paper finalist at CVPR 2026, which scales feed-forward reconstruction in data and model size and extends it into dynamic scenes. In parallel, D4RT (Dynamic 4D Reconstruction and Tracking), from Google DeepMind with collaborators at Oxford and University College London, carried the feed-forward paradigm into the temporal dimension and won a Best Paper Award at CVPR 2026.
(This post includes some figures directly taken from the original papers as they were published, without altering their numbering nor captions.)
What problem do these methods solve?
The classical route to 3D structure from images rests on multi-view geometry, principally Structure-from-Motion (SfM) and Multi-View Stereo (MVS). The canonical pipeline, embodied by tools such as COLMAP, runs in stages: detect and match keypoints, estimate camera positions, triangulate a sparse point cloud, then refine through Bundle Adjustment (BA), a non-linear optimization that jointly adjusts cameras and points to minimize reprojection error. The result is mathematically interpretable and geometrically consistent, but computationally intensive, sensitive to textureless or repetitive surfaces, and brittle when images do not overlap cleanly, with failures at one stage compounding downstream.
Machine learning first entered as components bolted onto that scaffold: better keypoint detectors, learned matchers, monocular depth estimators. Methods such as DUSt3R and its successor MASt3R then began predicting aligned point clouds directly from an image pair without known camera parameters, though they still work two views at a time and rely on post-processing to fuse pairwise results into a full scene.
Some more recent feed-forward methods will be used later in this post as comparison points. MegaSaM combines feed-forward depth prediction with optimization-based refinement to recover structure and motion from casual dynamic video, and remains a strong dynamic baseline despite that test-time optimization. π³ is a VGGT-style feed-forward model that removes the fixed reference-frame assumption through a permutation-equivariant design. Depth Anything 3 is another recent feed-forward reconstructor, built on a plain DINO backbone, that also targets dynamic scenes.
VGGT solves static reconstruction in one step. Given one image up to several hundred images, it predicts camera intrinsics and extrinsics, depth maps, point maps, and 3D point tracks at once, in under a second, with no optimization loop. VGGT-Ω keeps that single-pass design but shows that quality scales predictably with model and data size, and extends it to dynamic content while making training far more efficient.
D4RT addresses the harder, four-dimensional version of the reconstruction problem. As the real world moves: a video is a sequence of flat 2D projections in which objects translate, deform, and occlude one another while the camera also moves. Recovering geometry here means locating where every point sits in 3D at every moment, and separating object motion from camera motion.
Why does this matter?
The practical consequence is a step change in speed, simplicity, and accessibility.
Speed reframes what is possible. A reconstruction that once took minutes now runs in a fraction of a second, moving 3D and 4D perception from offline batch processing into real-time settings. D4RT estimates camera pose at more than 200 frames per second on an NVIDIA A100, roughly 9 times faster than VGGT and about 100 times faster than the optimization-based MegaSaM, all timed on the same A100 and with D4RT still more accurate. VGGT-Ω, for its part, runs about 50 times faster than MegaSaM on Sintel while substantially exceeding its accuracy.
Simplicity lowers the barrier to adoption. Classical pipelines demand domain expertise to configure and debug; a single network that ingests images or video and returns geometry is far easier to integrate and to use correctly.
For the industries where Digital Sense operates, the implications are direct. In renewable energy site investigation, the kind of offshore work we have supported for Ørsted, structure recovery and asset inspection benefit from reconstruction that tolerates difficult, low-texture imagery. In smart-city monitoring, such as our work with the NYC Department of Transportation, reconstructing and tracking urban scenes from vehicle or fixed-camera footage becomes tractable at scale. The same capabilities feed robotics, industrial inspection, digital twins, and agricultural canopy analysis, wherever understanding 3D structure quickly and reliably is the bottleneck.
VGGT: reconstructing static scenes in a single pass
VGGT's central design choice is restraint. Rather than engineer an architecture full of geometric priors, the authors use a fairly standard large transformer and let it learn geometry from a large body of 3D-annotated data, the philosophy behind general-purpose backbones such as GPT-style models, CLIP, and DINO.
A DINO encoder splits each image into patch tokens. The combined set passes through 24 layers alternating frame-wise self-attention (within a single image) and global self-attention (across all images). This Alternating-Attention design balances per-image consistency against cross-image integration, and beats both global-only and cross-attention variants. Dedicated heads then read out results: a camera head for intrinsics and extrinsics, and a Dense Prediction Transformer (DPT) head for depth and point maps. A tracking module built on the CoTracker2 architecture consumes dense tracking features from the transformer and is jointly trained to output point tracks. All quantities live in the coordinate frame of the first image (as popularized by DUSt3R), the world reference.
A subtle finding concerns over-complete prediction. Several outputs are mathematically redundant: camera parameters can be derived from point maps, and depth from point maps plus cameras. Training the network to predict all of them anyway improves accuracy, yet at inference the most accurate 3D points come not from the point-map head but from combining the separately predicted depth and camera outputs.
In pure feed-forward mode VGGT surpasses methods that rely on slow optimization for camera pose estimation (see the table below), and applying an optional Bundle Adjustment (BA) refinement further raises accuracy.
Table 1. Camera pose estimation (AUC@30, higher is better; runtime on one H100 GPU). Selected results from the VGGT paper, evaluated on 10 random frames per scene.
VGGT is also more accurate and faster across other tasks. On dense point-map estimation VGGT cuts the overall Chamfer error of DUSt3R from 1.005 to 0.677 while running about 35 times faster, and on two-view matching it beats the specialized state of the art despite not being trained for it. Within about a year VGGT became one of the most built-upon papers from its conference, with more than 1,500 citations and variants for acceleration, long sequences, LiDAR fusion, and semantic mapping.
VGGT-Ω: scaling feed-forward reconstruction and reaching into dynamic scenes
VGGT proved the paradigm. VGGT-Ω scales it in training data and parameters, and extends it to dynamic scenes, through three architectural changes:
- Register attention. VGGT already appended per-frame register tokens; VGGT-Ω turns these into "scene" tokens and, in a quarter of the global attention layers, restricts cross-frame exchange to the registers alone. Because global attention maps are very sparse, this acts as an efficient substitute for full global attention, saving around 23% of backbone FLOPs and 16% of memory during training with no measurable accuracy drop.
- A lighter dense head. The most memory-intensive high-resolution convolutional blocks are replaced with a lightweight Multi-Layer Perceptron and pixel-shuffle upsampler.
- Fewer heads. Multi-task supervision is kept, but only a single dense head for depth and a single sparse head for cameras remain; point maps and tracks are supervised through losses rather than dedicated heads.
Together these cut GPU memory during training by about 70%.
VGGT was trained almost entirely on still photos of static scenes, so it never learned what motion looks like and tended to fail when objects in a video moved. VGGT-Ω fixes this without any dedicated motion detector. Cutting training memory by about 70% made it affordable to train on roughly 15 times more data, including hundreds of thousands of real videos that naturally contain moving people and vehicles. And rather than predicting one rigid 3D point map for the whole scene, which breaks when parts of it move, it predicts only per-image depth maps plus camera positions and lets the network infer motion implicitly. Because depth is estimated per frame, a moving object is placed correctly within that frame without the model committing to a single position for it across time. Given enough dynamic video, the model seems to learn an internal sense of what moves versus what is static.
The payoff of these changes is large (see the table below), especially on the dynamic benchmarks where the original VGGT was weakest.
Table 2. VGGT-Ω on Sintel (a highly dynamic benchmark). Camera estimation as AUC (higher is better); depth as δ1.25, the percentage of pixels within a factor of ground truth (higher is better).
On Sintel camera estimation, VGGT-Ω lifts AUC@3° from 22.5 to 40.0, a 77% relative gain, while remaining about 50 times faster than MegaSaM. The figure below shows that the gains scale cleanly: as capacity grows from 0.2 to 10 billion parameters and data from a few thousand to two million sequences, 3D point error falls monotonically, from 0.107 to 0.046 with model size and from 0.275 to 0.073 with data size. Both curves suggest power-law behavior, the signature of foundation models in language and vision.
One further result hints at a broader role. The scene-token registers encode information well beyond raw geometry: fed frozen into a robotics vision-language-action model, they raised its average LIBERO success rate from 97.1 to 98.5%, and could be matched to natural-language descriptions of a scene. A model trained only to reconstruct geometry ends up learning a general-purpose spatial representation, akin to the reusable backbones that have proven so valuable in language and 2D vision.
D4RT: reconstructing dynamic scenes one query at a time
D4RT takes a different route, built around an idea from the Scene Representation Transformer. The encoder of D4RT is a Vision Transformer that reuses the interleaved frame-wise and global self-attention introduced by VGGT, which compresses an entire video into a single Global Scene Representation, computed once and held fixed.
The innovation is how it is read out (see the figure below). Instead of decoding everything everywhere, a lightweight decoder poses one flexible question: where is a given pixel in 3D, at a chosen target time, seen from a chosen camera viewpoint? Each query specifies five values, the pixel coordinates (u, v), the source frame, the target timestep, and the camera-reference frame, and the decoder returns the 3D position. Adding a small embedding of the local 9 by 9 RGB patch around the query point sharpens fine detail considerably.
Three properties make this powerful:
- Target time and camera reference are independent, allowing to fully disentangle motion in time from motion of the viewpoint.
- Every query decodes independently, so queries parallelize trivially and the model trains on a sampled subset.
- A single interface produces every output of interest simply by varying the query.
Table 3. One interface, many tasks. How different 4D outputs follow from the query parameters in D4RT.
Camera poses need no learned pose head: the model decodes the same grid of points from two reference frames and finds the rigid transformation between them with Umeyama's algorithm, which computes in closed form the single rotation, translation, and scale that best aligns one set of 3D points onto another in a least-squares sense,so no iterative solver is required. Because the decoder is sparse and lightweight, D4RT can also track every pixel in a video, using an occupancy-grid algorithm that avoids redundant work for a 5 to 15 times adaptive speedup depending on motion complexity.
D4RT runs 18 to 300 times faster than prior methods on tracking workloads, and sets new marks on dynamic reconstruction.
Table 4. Accuracy with motion. Selected D4RT results against recent baselines (lower is better for point-cloud L1; higher is better for Pose AUC).
VGGT-Ω and D4RT: two routes into dynamic reconstruction
The two CVPR 2026 methods share a lineage. Both are feed-forward transformers that eliminate heavy test-time optimization, both reuse VGGT's alternating frame-wise and global attention in their encoders, both recover camera pose through closed-form Umeyama alignment rather than a learned pose head, and both minimize hand-built priors in favor of learning from large data. Both push well beyond the original VGGT into dynamic scenes.
They differ in what they treat as the core problem. VGGT-Ω is a reconstruction backbone, betting that scaling data and parameters plus a reusable representation is the path forward. It outputs per-frame depth and camera parameters, handles a single image to hundreds of unordered views, but it does not correspond moving points; it reconstructs geometry rather than tracking it. VGGT-Ω is a motion-aware reconstruction model, but not a tracker. Indeed, it reconstructs each frame's geometry but does not put in correspondence moving points across time, which is precisely the gap D4RT closes.
D4RT is a unified 4D engine, betting that reconstruction should become an efficient querying problem: encode the video once, then ask the decoder for the 3D position of any pixel, at any time, from any viewpoint. That single interface yields point clouds, depth, cameras, and, crucially, dense correspondence for every pixel including moving ones, exactly what VGGT-Ω lacks.
So is one simply better? On dynamic correspondence and all-pixel 4D tracking, D4RT is ahead, with state-of-the-art results and very high tracking throughput. But other things favor VGGT-Ω, and they are not minor. It accepts unordered image collections rather than requiring a video, which matters for photogrammetry-style inputs and sparse capture; it demonstrates clean power-law scaling to 10 billion parameters. The two are currently best read as complementary bets and have not been benchmarked against each other.
On the downside, both depend on large, carefully curated training data and heavy accelerator budgets, VGGT-Ω on GPU clusters with a 10 billion parameter top variant, D4RT on TPUs with a billion-parameter encoder, so neither is lightweight to reproduce.
Conclusions
VGGT, VGGT-Ω, and D4RT are three points on the same curve. VGGT showed that a general transformer, given enough 3D-annotated data, can absorb the work that geometry pipelines once did by hand, faster and often more accurately. VGGT-Ω showed this scales predictably with data and model size, extends cleanly into dynamic scenes, and turns reconstruction into a reusable spatial representation. D4RT reframed reconstruction as an efficient querying problem and carried it fully into time. The direction is unmistakable: from static to dynamic, from many specialized models to one interface, from offline optimization to real-time inference, and from task-specific networks to scalable spatial foundation models.
For organizations whose products depend on understanding the physical world, from satellites observing the Earth to robots moving through it, this is a shift worth tracking closely. The hard part is no longer whether neural reconstruction works. It is choosing and adjusting the right method for a given constraint, adapting it to domain-specific imagery, and engineering it into a reliable production system. That is precisely the applied-research work Digital Sense has done for more than a decade, across remote sensing, computer vision, and machine learning, for partners including Satellogic, Ørsted, and ULTA Beauty.
If your team is evaluating how feed-forward 3D or 4D reconstruction could fit into your products, we would be glad to talk. Explore more in our Scientific Insight Series on the Digital Sense blog, review our success stories, or start a conversation with our team to scope what is possible.
References
- Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., & Novotny, D. (2025). VGGT: Visual Geometry Grounded Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5294–5306. CVPR 2025 Best Paper Award.
- Wang, J., Chen, M., Zhang, S., Karaev, N., Schönberger, J., et al. (2026). VGGT-Ω. arXiv preprint arXiv:2605.15195. CVPR 2026 Best Paper Finalist.
- Zhang, C., Le Moing, G., Koppula, S., Rocco, I., et al. (2026). Efficiently Reconstructing Dynamic Scenes One D4RT at a Time. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7382–7392. CVPR 2026 Best Paper Award. (Preprint: arXiv:2512.08924.)
.jpg)