Aircraft (AMDAR Traditional) Retrospective Verification

This document is a guide for verifying retro cases against AMDAR data in the MATS MariaDB database. These aircraft observations come from the MADIS real time netCDF feed with additional QC applied within GSL, and does not directly involve preBUFR processing. This document details how to load a retro run into the acars_RR2 database from Jet or Hera as the role.amb-verif user. This has replaced the old acars_RR database as of October 2020. If you need an old retro run moved to the new database, please contact a member of the MATS team.

Updated: May 15, 2025
POC: MATS Service account (mats.gsl@noaa.gov)

Reload AMDAR data¶

1. Log in as role.amb-verif on either the Jet or Hera HPC system and check to see if retrospective ACARS files are available in the following directory for your time period: (if you do not have access, please ask a MATS team member for assistance in gaining access to the role account):

   ~/role.amb-verif/acars_RR/ACARS_RETRO_DIR

2. If they are not, you will need to modify and run the restore_acars_files.ksh script in the same directory in order to pull them back off of DSG space on the HPSS system. This may take some time if you are running over a long period. The DSG archive does have some missing days due to HPSS maintenance periods. If you encounter one, please contact us and we can pull back data from the MADIS archive

Process Retro grids¶

1. Log onto the HPC (Jet or Hera) where your output grids are stored as the role.amb-verif role account.

2. Go to the directory "~role.amb-verif/acars_RR/retro_runs". If your retro is not already linked, create the soft links to the cycle-level directory of your output with a case name of your choosing to highlight the experiment, with the available conventions (isobaric output only):

   ln -s {case_cycle_directory} {experiment_name}

3. In ~role.amb-verif/acars_RR, run the following command to create a reference forecast lengths file and modify as desired:

   cp fcst_lens_template {experiment_name}_fcst_lens

4. Again, in the ~role.amb-verif/acars_RR directory, you can now process the model-obs pairs with the following command line option:

   ./gen_retro_acars_stats3.pl {experiment_name} {start_epoch} {end_epoch} {number_vertical/isobaric_levels} {reprocess? 0 - no, 1 - yes}

   Or using a Slurm role.amb-verif cron entry (for longer periods):

   40 * * * * $HOME/utilities/sb.sh $HOME/acars_RR/retro.sh {experiment_name} {start_epoch} {end_epoch} {number_vertical/isobaric_levels} {reprocess? 0 - no, 1 - yes}

Populate the MariaDB database¶

1. Once the model-obs pair files are created in the temporary directory (~role.amb-verif/acars_RR/tmp2), you may either run the following command, or ensure that it is included in the role.amb-verif crontab to load the pairs into the database and generate the partial sums used by MATS:

   ~role.amb-verif/acars_RR/load_acars_model_data_retro2.pl

2. Results should appear on MATS the following morning, but a metadata scan can be run to expedite the process. As role.amb-verif on Hera, run ~/mats_metadata/update_metadata_aircraft.py{experiment_name}. If you see a >0 number of records (‘numrecs’) found in the output, your retro is stored in the database with metadata enabled.

3. The final step is to refresh the metadata inside the MATS client using the following URL: https://gsl.noaa.gov/mats/upperair/refreshMetadata. Once complete, return to the upperair app home page and check the data source drop down for your retro (be sure to have the truth selector set to AMDAR). If present, you are free to create plots.

If you encounter any issues during any of the steps outlined here, either on Jet or Hera, please contact the MATS team (mats.gsl@noaa.gov)