While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.

Physically Based Rendering (PBR) is the physical simulation of light transport within a scene, i.e., how light participates in events such as reflections, refractions and scattering when interacting with objects. Recent advances have made the diffentiation of PBR possible and computationally tractable, allowing the reconstruction of physical scene properties (such as lighting, materials, shapes, etc.) from two-dimensional images using gradient-based optimisation. This enables us to optimise physically-interpretable parameters instead of reasoning about pixel-based images. Here we demonstrate the potential of differentiable PBR in two settings: (a) to find physically based model metamers to explain model sensitivities and invariances to physical object properties and (b) to create controversial scenes where two models disagree on the interpretation of a physical scene, using gloss appearance as an example. These examples display the promise of differentiable PBR for understanding perceptual inferences in humans and machines.

I will give a remote talk about our latest preprint at Vision Science Society 2026 about our latest preprint MRD.