Foundations of Deep Generative Models

Modern diffusion and flow-based generative models have become powerful tools for scientific discovery, yet many scientific problems go beyond probability paths defined in fixed-dimensional Euclidean spaces. Biological trajectories may lie on unknown manifolds, physical systems may obey Lie group symmetries, and protein or molecular design may require variable-length generation. This talk presents recent work from our group on generative modeling methods that respect these structures.

First, I will present a flow-matching method for trajectory inference and generation on unknown manifolds. We learn a geodesically convex latent space that enables meaningful interpolation, and then pull the learned transport dynamics back to the data space. This allows the model to discover the geometry of a general scientific data manifold and use it for more effective transport and generation. Second, I will describe Trivialized Generative Models, a new family of generative models for Lie groups. TGM learns endpoint-constrained paths in a fixed Lie algebra and lifts them to the Lie group, providing a simple and powerful way to define valid group-valued generative dynamics. This endpoint formulation enables flexible path design beyond standard geodesic or exponential interpolation, avoids key complications of general Riemannian generative models, naturally supports non-compact groups, and extends to few-step consistency models and momentum-based higher-order dynamics. Finally, I will introduce Generalized Poisson Flow for variable-length protein design, where the model jointly learns the evolution of length and the generation of length-dependent multimodal features. Our method achieves state-of-the-art empirical performance across unconditional structure and sequence generation, motif scaffolding, and peptide co-design, validating a new paradigm for flexible-length protein design.

Together, these works suggest a broader foundation for scientific generative modeling: designing probability paths that respect the geometry, symmetry, and variable-dimensional structure of scientific data.

This talk explores recent theoretical and methodological advancements in diffusion models. On the theoretical side, we demonstrate that diffusion models can circumvent the curse of dimensionality by discovering intrinsic low-dimensional structures within data distributions. We discuss this capability for both continuous and discrete variables and establish their theoretical optimality, showing that these advantages are driven by their feature learning abilities.

Methodologically, we introduce novel inference-time-adaptation/post-training techniques for discrete diffusion models. Leveraging Doob's h-transform as our primary technical tool, we demonstrate that integrating this transform with the efficient sampling capabilities of diffusion models facilitates effective inference time adaptation without explicit parameter updates. Specifically, our approach enables reinforcement learning on the unmasking order distribution without requiring model updates, yielding substantial performance improvements, and we show the unmasking order adaptation has a significant impact on the performance improvement.

In this talk, we will try to clarify different levels and mechanisms of intelligence from historical, scientific, mathematical, and computational perspective. From the evolution of intelligence in nature, from phylogenetic, to ontogenetic, societal, and to scientific intelligence, we will try to shed light on how to understand the true nature of the seemingly dramatic advancements in the technologies of machine intelligence in the past decade. We achieve this goal by developing a principled theoretical framework to explain deductively the practices of deep representation learning from the first principle of pursuing low-dimensional structures in data distributions. This framework not only reveals true nature hence both capabilities and limitations of the current deep architectures, and but also provides principled guidelines to develop more complete and more efficient learning architectures and systems. Eventually, we will clarify the difference and relationship between Knowledge and Intelligence, which may guide us to pursue the goal of developing systems with true intelligence, at least at the level for a predictive and generative memory. If time permits, we will also showcase some of the ongoing new technological developments towards realizing intelligence within an open real physical world.

How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data. A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation using the Bayesian posterior. We show that spatially local BIRD models closely approximate trained diffusion models early in training, across different architectures such as UNets and DiTs. We identify an exact information-theoretic phase transition boundary between memorization and generalization in the joint space of amount of training data, time in the reverse generative process, and amount of information restriction: a BIRD model memorizes when the mutual information between its restricted noisy observations and the training data exceeds the log number of training points, and it generalizes otherwise. Experiments across a range of datasets confirm our theoretically predicted location for the transition. We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-trained diffusion models track the memorization-generalization phase boundary by increasingly restricting information over time. Overall, our results reveal a fundamental role for information restriction in generative AI to circumvent the curse of dimensionality. This is joint work with Henry Hunt and Mason Kamb.

Modern generative models increasingly spend compute at inference time. When does this extra compute improve performance, when does it saturate, and when can it backfire? This talk introduces a family of exactly solvable high-dimensional models, analyzed using tools from random matrix theory, that make these questions analytically tractable and offer a quantitative theory of test-time scaling.

To understand why generative models such as flow matching and diffusion models succeed on high-dimensional data, it is essential to analyze their behavior under the manifold hypothesis, which posits that the data distribution is supported on a low-dimensional submanifold. In this talk, we focus on Optimal Transport Conditional Flow Matching and show an exact proximal form via an extended Brenier potential under the manifold hypothesis. More precisely, the mapping to recover the target point, or the "decoder", is expressed by a proximal operator, which yields an explicit expression of the vector field. Using mathematical tools from convex analysis, we analyze the local behavior of the vector field around the manifold and show the stability of the manifold structure against perturbation to the dynamics: the dynamics does not expand or shrink exponentially along manifold directions.