Liveness Vulnerability Found: A Patched Mumbai Proposal Is Available

TL;DR: We are proposing a patched Mumbai protocol upgrade to address a liveness vulnerability.

We have discovered a vulnerability which could affect the Tezos network’s liveness but not its safety. In other words, the vulnerability could slow or halt the network, but not put funds at risk.

We have investigated various ways of mitigating the problem via an Octez shell update (everything but the protocol), but none were found to be satisfactory, and a patch for the protocol itself is required.

We have taken the step of producing a patched version of the Mumbai protocol upgrade proposal that addresses this vulnerability. The hash of Mumbai 2, the patched version of Mumbai, is: PtMumbai2TmsJHNGRkD8v8YDbtao7BLUC3wjASn1inAKLFCjaH1.

As the Mumbai proposal is currently in the Promotion period of the governance cycle, there are two scenarios going forward:

  • User activated protocol override: The community votes Yay to Mumbai in the current voting period. All nodes running v16.0 of Octez will activate the patched version, PtMumbai2, instead of PtMumbaii. This is what happened with the patched Babylon, Edo, and Hangzhou proposals.

  • Restarting the governance cycle: The community votes Nay to Mumbai in the current voting period, and the governance process “reverts” to a new proposal period. A patched Mumbai protocol will then be proposed and must undergo a full governance cycle. This should last about 2.5 months assuming approval in every voting period.

A new release candidate, Octez v16.0~rc3, has just been published, and v16.0 is expected shortly after. In addition to the Mumbai 2 protocol, this release candidate also includes performance improvements in the baker that we consider necessary for the block time reduction coming with Mumbai.

Important: Octez v16.0 will be required for Mumbai

As stated in a previous blogpost, upgrading to v16.0 (or later) is necessary to participate in consensus once Mumbai is activated, independent from the discussion above.

We remind the community that bakers representing at least 2/3 of the total stake must be running the same protocol version for the chain to stay live, due to the requirements of the Tenderbake consensus algorithm.

The Mumbainet test network will be restarted shortly to instead use the Mumbai 2 protocol. Joining the new rebooted test network will require Octez v16.0~rc3, or the upcoming stable release. Please check this page for further detail.

We acknowledge that this is coming at a late stage in the current voting process, and we ask for the community’s understanding and cooperation in addressing the situation. A full write-up on the vulnerability will be published when it is safe to do so.

We are confident that the measures outlined in this post will sufficiently address the present issue and look forward to unleashing the potential of the ground-breaking features contained in Mumbai!


Incident Report available:

and also on our blog:

@NomadicLabs thank you for identifying this issue. I would love to hear your thoughts on how the community could bring the patch process on-chain, instead of relying on user activated protocol overrides in scenarios like this.

If you don’t think it’s worth it that’s fine too, I’d love to understand why.

I started a thread about this quite a while back,l: How can we incorporate the patching of issues found during protocol upgrade testing into the on-chain governance process?.