x/crisis does NOT cause chain halt

Impact

If an invariant check fails on a Cosmos SDK network and a transaction is sent to the x/crisis module to halt the chain, the chain does not halt. All versions of the x/crisis module is affected on all versions of the Cosmos SDK.

Details

The x/crisis module is supposed to allow anyone to halt a chain in the event of a violated invariant by sending a MsgVerifyInvariant with the name of the invariant. Processing this message is supposed to cause the nodes to panic. However, because the panic is within a transaction, it is caught by the SDK’s built-in panic-recovery machinery and just treated as a normal “invalid” transaction (ie. it returns a non-zero abci Code). Thus the x/crisis transactions don’t actually cause chains to halt. If there is an invariant violation, it can be confirmed with an x/crisis transaction, but it won’t cause any nodes to halt, they will just continue processing blocks.

That said, any node running with start --inv-check-period X will actually panic when it runs the periodic check (though it will still not panic just by processing an x/crisis transaction). Since this panic is located in EndBlock, it is not caught by the panic-recovery machinery and does actually crash the node. Presumably few if any nodes actually run with this in production because of how long the invariant checks take, and this runs all of them every X blocks.

Patches

No patches will be released.

The x/crisis module was originally intended to allow chains to halt rather than continue with some unknown behaviour in the case of an invariant violation (safety over liveness). However, as chains mature, and especially as the potential cost of halting increases, chains should consider carefully what invariants they really want to halt for, and what invariants are just sort of helpful sanity checks, but may not be worth halting for.

In some cases, chains have already broken the invariant calculations but have dealt with the consequences off-chain or during development. Halting these chains would be counter-productive.

The SDK team is working on new modules that allow chain developers to fine-tune the chain invariants and the necessary actions.

Hence, the decision was made that the x/crisis module will not be patched for chain halts. The module will be deprecated when new modules take over its responsibilities.

Workarounds

In case of a valid invariant check failure that requires a chain halt, the network validators are encouraged to coordinate off-chain for network halts. This has been an already established process for security patches.

References

SDK developer epic about invariant checking: https://github.com/cosmos/cosmos-sdk/issues/15706 Public report: https://github.com/cosmos/cosmos-sdk/issues/15325