MATS Documentation

Welcome. This site centralizes verification guides, methodological explanations, and references used by the Model Verification Team.

What is MATS?¶

MATS - the “Model Analysis Tool Suite” - is a quick & versatile tool for viewing model verification statistics in GSL.

It was created to allow model developers to perform hypothesis-driven testing. It enables them to quickly quantify the impact of model changes by comparing experimental model output against a massive array of real-world observations, as well as current operational model output. Turner et al. (2020)

Key capabilities of MATS¶

• One of the most critical technical features of MATS is its use of a database that stores partial sums, contingency table counts, and predefined statistics.

  • Benefit: Instead of recalculating statistics from raw data every time a user makes a request, the system retrieves pre-calculated statistics. This drastically reduces response time, allowing developers to generate complex plots in seconds rather than hours.

• To access the data in the database, MATS provides a web-based GUI which allows users to interactively filter and compare data.

• To add data to the database, MATS has a number of scripts that can be utilized from HPC to extract, transform and load the data.

• MATS is not limited to a single type of data - it contains specialized applications for a variety of meterological phenomena.

• MATS automatically calculates 95% confidence intervals (error bars) for its metrics. This helps developers determine if an improvement in a model is “statistically significant” or just due to random chance.

• Support for the MET Framework. MATS intergrates with the Model Evaluation Tools (MET), the industry standard for verification.

Who is this documentation for?¶

• Forecasters and scientists in GSL running verification for research, primarily in model development.

• Developers and analysts maintaining verification pipelines and storage

• Contributors documenting processes, conventions, and shared knowledge

Documentation contents¶

• Real-time/operational (“rt”) and retrospective/experimental (“retro”) verification workflows supporting both grid-to-grid, grid-to-observation comparissons, as well as other techniques

• Practical, task-focused guides for common scenarios and data sources

• Concept explanations for statistical methods and metrics

• Reference material for tools, configs, and operational practices

Start here¶

• Verification Guiding Principles → Guiding Principles

• Tutorials: step-by-step introductions and end-to-end walkthroughs → Tutorials

• How-To Guides: focused answers to specific tasks → How-To Guides

• Explanations: concepts, methods, and reasoning → Explanation

• References: commands, configuration, and facts → Reference

• Contributing: structure, style, and repo setup → Contributor Docs

Contact¶

For questions on MATS or any issues encountered while following with the instructions, email mats.gsl@noaa.gov.

Contributing to these docs¶

We welcome contributions to and questions on the documentation here! Feel free to open an issue or discussion in Github, or send us an email, if there’s anything that could be updated.