Fenix_Init hangs after an error when app contains (multiple) derived robust communicators

[rfvander]

The AMR PRK derives two robust communicators from the original robust communicator. In this implementation we use MPI_COMM_WORLD as the input communicator for Fenix_Init, and provide NULL for the output communicator address.

[rfvander]

I could determine that the code hangs in the preinit part.
I take that back. The code hangs in Fenix_Init, but I don't know where yet.

[rfvander]

In case of the successful run with the stencil PRK, when I kill one rank, the Fenix library reports that some processes failed, and also that the communicator was revoked. In case of the hanging AMR PRK no communicator is reported revoked.

[rfvander]

More debug info. In the successful Stencil PRK, if I use newcomm as the resilient output communicator and replace all subsequent references to MPI_COMM_WORLD with newcomm, the code runs successfully after an error. However, if I use NULL for the resilient output communicator and use MPI_Comm_dup to create newcomm out of MPI_COMM_WORLD after Fenix_Init, the code hangs after an error.

[rfvander]

We need to maintain a list of communicators derived from the robust output communicator, and revoke all of them when an error occurs.