NAFEMS, the International Association for Engineering Modelling, Analysis and Simulation Community, introduced AIPhysics.org, an open, independent benchmarking initiative for AI methods in engineering simulation. The initiative will be initially launched in partnership with Rescale, the digital engineering platform for the AI era, with the aim of engaging in further partnerships with vendors and engineering teams throughout the simulation community. The announcement came at the NAFEMS “Artificial Intelligence and Machine Learning in CAE Applications” Conference in Munich, Germany.
The initiative aims to bring together industry, academia, engineering practitioners and software providers to develop shared benchmarks and evaluation methods for AI-based simulation. As founding members, NAFEMS will ensure independent governance, neutral evaluation, and the development of the benchmarking methodology through an AIPhysics.org Steering Committee. Rescale, as a founding technical partner, will provide AI frameworks, inference and cloud infrastructure to support submissions, evaluations, data security and technical support.
AI surrogate models for CFD, FEA and other modeling and simulation areas are improving fast. But engineers lack a trusted, open way to compare them on the problems they actually solve. General machine-learning leaderboards don't answer the questions simulation teams ask: Does the model hold up across real physics? Does it get the engineering quantities right? How does it compare with trusted reference data?
AIPhysics.org aims to answer those questions with three things:
- Public training data for real engineering problems, from stressed plates to full-vehicle aerodynamics and crash.
- Blind evaluation on held-out test cases that are never released, so no model can be tuned to the test.
- Published results and methods, so anyone can see how every number was produced.
The initial benchmark set covers five problems across structural, fluid and crash physics: Plate with Hole, 2D Airfoil, DrivAerML (road-car external aerodynamics), HiLiftAeroML (high-lift aircraft) and an OpenRadioss frontal crash model.
The initiative is independent by design. Every submission will go through the same process and scoring pipeline, including the reference runs. Every dataset will credit its source, and methodology changes will be recorded in a public decision log.
"For forty years NAFEMS has helped the simulation community decide what 'good' looks like, from the original FE benchmarks to today's work on simulation credibility. AI changes the tools, not the need. Engineers deserve to know how an AI model performs on real physics before they trust it with a design decision. AIPhysics.org will give the whole community - vendors, researchers and users alike - the same yardstick, under independent governance."
- Tim Morris, CEO, NAFEMS
Rescale will co-manage the website, submission system and evaluation pipeline, the secure infrastructure that keeps test data private, and the resources for every evaluation.
"AI physics will only reach production engineering if the people using it can trust the results. That trust has to be earned in the open, on neutral ground. We're proud to provide the AI infrastructure, an initial set of test cases and model benchmark data, and we'll continue to submit our own models under exactly the same rules as everyone else to contribute to this valuable industry data set."
- Joris Poort, founder and CEO, Rescale
AIPhysics.org will be open to everyone. It will welcome model developers, AI framework providers, software vendors (including Rescale's competitors), academic and research groups, national laboratories, engineering teams in industry and standards bodies. Participants can:
- Submit a model: train on public data with any tools, then submit for blind evaluation. Code and weights can stay private.
- Contribute a benchmark: bring simulation data for a new problem; the team will help build public training and private test sets.
- Help shape the methods: join working groups forming on metrics and evaluation for each field.
- Become a partner: join the initiative on transparent terms set by the Steering Committee.
Additional founding members of the Steering Committee will be announced shortly.
