sensible way to flag model instability?

[GanZhang-GFD]

Does the team have a sensible way to detect and flag unstable simulations?

Context: I finished some AMIP runs forced by modified boundary conditions. For each start date, I have twenty ensemble members generated with different random seeding. While simulations are stable for most start dates, a few combinations of initial conditions and boundary forcings are so unstable that most runs fail.

The plot compares the standard deviations along the longitude dimension at the start and the end dates. The ratios are in the [latitude, ensemble member] space. Any ratios beyond 5.0 are likely beyond the normal range of seasonality and indicate some model instability. Conventional physical models usually flag instability and terminate simulations early.

The plot is generated using the following code:

    ds_test = xr.open_mfdataset(filelist, preprocess=add_ens_dim)
    (ds_test['geopotential'].isel(time=-1).sel(level=500).std('longitude').compute()/ \
     ds_test['geopotential'].isel(time= 0).sel(level=500).std('longitude').compute()).T \
        .plot(cmap='PiYG_r', levels=[0.5, 0.75, 1, 1.25, 1.5, 2.5, 5, 10, 20])
    plt.title('Ratio of Std.Dev. by Latitude \n' + \
              pd.to_datetime(ds_test.time[-1].values).strftime('%Y-%m-%d') + ' vs ' + \
              pd.to_datetime(ds_test.time[0].values).strftime('%Y-%m-%d'))

For reference, the same analysis of a good date looks like this:

[shoyer]

This is a great topic for discussion! CC @yaniyuval who is looking into this.

The short answer is that we are still trying to nail down stability for NeuralGCM models, so we can fix this in future versions. We have also noticed that some initial conditions can be much more unstable than others. In particular, it seems that instability is often evident first from a drift in mean surface pressure.

[yaniyuval]

Like Stephan wrote, we are still working on instability issues.
In some simulations I found that there is an early drift of the global mean pressure surface (which happens a lot before the actual instability). This drift might be related to the sharp topography we are using (as we have not smoothed the orography). We are currently working on a fix to this issue, but even after this fix it won't solve all instabilities.

[GanZhang-GFD]

Interesting. Thanks for your quick replies. I looked at your paper figures and some of my cases. I agree that the instability often appears to originate from the near-surface.

The most common mode of failure appears to be:

1. Near-surface (gravity?) waves develop, propagate, and amplify in the tropics.
2. Upward propagation deposits momentum in the upper troposphere and stratosphere.
3. The waves extend poleward and gradually contaminate the whole atmosphere.

The good side of the story is that such propagating behaviors are consistent with physics. It also indicates the model may capture processes related to the stratosphere-troposphere coupling, which is a weakness for many physical models and is useful for long-range predictions. Whether NeuralGCM does better remains to be determined.

For the instability, my guesses are: 1) smoothing the topography and tweaking the settings of lower boundaries probably can help; 2) if the dynamical core model has hyperdiffusion parameters, tuning it may help dampen some initial fine-scale noises.

Edit: the stratosphere (0-150 hPa) also has early signs of instability.

Looking forward to the fix that you two mentioned.