Repurposing Image Generation Models for Depth Estimation ── What Marigold V2 reveals about diffusion transformers beyond image synthesis

1. The Problem: Why Single-Image Depth Is Still Hard

Monocular depth estimation asks a deceptively simple question: given one photograph, how far away is each pixel? The answers feed into 3D scene reconstruction, portrait-mode blur, autonomous navigation, and robotic manipulation. A single camera is cheaper and lighter than a stereo rig or a LiDAR scanner, which is why so much effort goes into squeezing depth out of a flat image.

Two stubborn problems remain even in mature models. First, a model trained on indoor scenes tends to lose accuracy when shown an outdoor landscape it has never encountered—the out-of-distribution problem. Second, fine geometric details get lost. Fur, foliage, and hair-thin edges blur into flat regions, producing depth maps that look correct from a distance but fall apart at the boundaries where precision matters most.

2. The Proposal: Borrowing the Eyes of a Generative Model

Marigold V2 starts from a premise: a diffusion transformer (DiT) trained on massive image datasets has already learned to represent spatial structure. If the model can generate a photorealistic room, it must, in some sense, understand which objects are in front of which. The original Marigold (accepted at CVPR 2024) introduced this idea using an earlier generation of diffusion architectures. V2 updates the approach for the DiT era and adds several technical ingredients.

First, the multi-step generation process is collapsed into a single inference step by building on flow-matching models. Quantization is applied where needed to preserve the model’s capacity while reducing computational cost. The goal is a model that runs cheaply without losing what it learned during pretraining.

Second, the authors analyze artifacts that appear with naive fine-tuning and propose two remedies. One aligns the model’s internal representations with semantic features extracted from ground-truth depth maps. The other introduces a two-stage fine-tuning protocol organized around a Sinkhorn-based loss function. Together, these steps are meant to produce cleaner, crisper depth outputs.

3. What the Abstract Reports

The authors report a 16–26% improvement in AbsRel (absolute relative error) over the previous best results on the KITTI and ETH3D benchmarks. These are standard evaluation datasets in the depth-estimation community: KITTI is captured from a car driving through urban streets; ETH3D contains indoor and outdoor scenes with high-resolution ground-truth.

Qualitatively, they claim the model resolves fur, foliage, and hair-thin edges that prior models could not capture. Beyond depth, Marigold V2 is reported to achieve state-of-the-art results on related dense regression tasks, including surface normals estimation and intrinsic image decomposition.

This is a preprint. The reported numbers reflect the authors’ own experiments. Independent replication has not yet been published, and the claims have not undergone peer review.

4. A Concrete Scenario: A Robot Reading the Depth of a Shelf

5. What Is Not New, and Where the Limits Are

The idea of repurposing generative models for depth estimation was already demonstrated by the original Marigold. V2 extends it to the DiT architecture and adds single-step inference—an evolutionary refinement, not a conceptual leap. The move from U-Net to DiT follows the same trajectory that the rest of the generative-model community has taken.

Several limits are worth noting. The 16–26% improvement is measured on KITTI and ETH3D specifically; the abstract does not report performance on other domains such as medical imaging or satellite imagery. The size and inference cost of the underlying DiT model are not detailed, leaving practical compute requirements unclear. The trade-off between quantization and depth accuracy is not spelled out. And the abstract provides no theoretical explanation for why the Sinkhorn-based loss is effective—only empirical evidence within the reported experimental scope. Whether the same protocol transfers to other dense regression tasks without re-tuning remains an open question.

6. Why This Topic Is Drawing Attention Now

The paper has collected 52 upvotes on Hugging Face and 330 stars on GitHub. Keywords from the selection include “diffusion transformer,” “flow-matching,” “monocular depth estimation,” and “quantization.”

A broader trend is at work. Diffusion models originally built for image synthesis are being repurposed for perceptual tasks—depth estimation, semantic segmentation, optical flow. The representations learned during generation appear to encode enough scene understanding to be useful for downstream analysis. Marigold V2 sits at the intersection of this repurposing trend and the architectural shift from U-Net–based diffusion to transformer–based diffusion.

Upvotes and stars reflect community interest, not correctness. They indicate that the topic resonates with researchers and engineers, not that the claims have been independently validated.

7. Connections to Pharma and Regulatory Practice

Monocular depth estimation is distant from pharmaceutical work. But the underlying methodology—taking a model trained for one generative task and repurposing it for a different analytical task—carries a direct parallel in drug discovery.

Generative models trained on chemical structures are increasingly being adapted for activity prediction and toxicity estimation. The hypothesis that representations learned for generation can serve discrimination is shared across domains: images and molecules alike. Marigold V2 is one concrete instance of this hypothesis, tested in the vision domain rather than the molecular one, but the architectural pattern is the same.

A second point of contact is quantization for cost reduction. Pharmaceutical companies evaluating in-house deployment of large models face direct trade-offs between prediction quality and compute cost. Techniques that compress a model without degrading its output affect build-or-buy decisions and infrastructure planning. That said, the quantization–accuracy trade-off is highly task-specific. A compression level that works for depth maps may not preserve the fine distinctions needed in molecular property prediction, so transferring conclusions across domains requires careful validation.