METAR Surface Retrospective verification

This document is a guide for verifying retro cases against METAR data in the MATS MySQL database. These METARs come from the MADIS netCDF files in their archive, and do NOT interact with NCO prepBUFR files directly.

Updated: August 28, 2025
POC: MATS Service account (mats.gsl@noaa.gov)

Reload METAR data¶

NOTE: This step is necessary only if your retro time period hasn’t already been loaded. If you know your retro period has been reprocessed inside the database, please skip this section. If you are unsure, proceed to step 1.

1. Ensure that observations for your period are within the MySQL database. If you have access to role.amb-verif on either Jet or Hera, simply type “mysql” and you can get into the terminal. The table you’ll need to check is the madis3.obs_retro table. To see what days have already been processed, run this query from the database terminal:

   select from_unixtime(floor(time/86400)*86400) as day
      from madis3.obs_retro
      group by day
      order by day;

2. If your retrospective days don’t exist in the database, log onto model-verif-prod.gsd.esrl.noaa.gov as the “amb-verif” role account (if you do not have access, please ask a MATS team member to either reload METARS for you, or work with ITS to add to you to the role account)

3. Run the following command:

   ~amb-verif@model-verif-prod:~/madis51/retro> load_retro.pl {start_epoch} {end_epoch}

   This may take a few hours depending on the length of your retro period. Additionally, the MADIS ftp server occasionally hangs when first attempts to connect are made. Usually trying a second time works, but let us know if problems continue

Process Retro grids¶

1. Log onto the HPC (Jet or Hera) where your output grids are stored as the “role.amb-verif” role account. If you cannot become role.amb-verif, please let us know and we will have you added.

2. Go to the directory ~role.amb-verif/ruc_madis_surface/RR_retro. If your retro is not already linked, create the soft links to the cycle-level directory of your output with a case name of your choosing to highlight the experiment, with the available conventions:

   ln -s {case_cycle_directory} {experiment_name}

3. Go to the ~role.amb-verif/ruc_madis_surface directory and run the following command:

   retro2.pl {experiment_name} {start_epoch} {end_epoch} {reprocess? 0 - no, 1 -
   yes}

4. Results should appear on MATS the following morning, but a metadata scan can be run to expedite the process. As role.amb-verif on Hera/Jet, run:

   ~role.amb-verif/mats_metadata/update_metadata_surface.py {case_name}

   If you see a >0 number of records (‘numrecs’) found in the output, your retro is stored in the database with metadata enabled.

5. The final step is to refresh the metadata inside the MATS client by refreshing the Surface app page a few times in your browser.

If you encounter any issues during any of the steps outlined here, either on Jet or Hera, please contact the MATS team (mats.gsl@noaa.gov)