ITC Colloquium: Miles Cranmer - The Era of Scientific Foundation Models

In person for CfA members and streaming on YouTube.

The Era of Scientific Foundation Models

Why can't machine learning models generalize? Physical theories certainly can! General relativity predicted black holes and gravitational waves decades before any observational evidence. How can physics extrapolate so far beyond its data?

I think the answer is induction. The physicist assumes a new model ought to accommodate old models. Yet the machine learner does not do this; instead, they ask their models to exist in isolation, and to bootstrap all knowledge from scratch. However, the past five years have seen this machine learning practice shift with the notion of generalist "foundation models" coming into vogue: models pretrained on massive diverse datasets which learn general representations (such as large language models).

In this talk I will focus on “PolymathicAI,” a research collaboration attempting to push this practice for science, building large-scale multi-disciplinary scientific foundation models. I will present two of our recent flagship models: Walrus, a 1.3B-parameter physics foundation model pretrained on 19 different PDEs in 2D and 3D, and AION, a family of multimodal foundation models for astronomy, pretrained on over 200 million objects across 39 modalities from five major surveys. I will discuss using these models to achieve stronger performance per datum in physics and astronomy. I will also consider whether their representations encode physical structures, and discuss model discovery via PySR and SymTorch.