Bug discovered in Qena42: Our recommendations

Research and Development
1

Nomadic Labs would like to share a few observations to assist bakers in making an informed decision and to support maintaining high standards for the Tezos governance process.

The protocol proposal Qena42 (formally, hash PsQEna427TJJ2dYatjWS12qKihty5huvBDmNHNAgCeTzWCpF673) was selected by bakers during Proposal period #133. Advertised as a variant of Quebec, it differs in two ways:

  • It omits the “Adaptive Maximum” feature.
  • It attempts to change the staking target from 50% to 42%.

While we continue to view an adaptive maximum curve as beneficial, and believe changing the staking target as done in Qena42 is not a practical or effective way to address issues noted since June, our evaluation of Qena42 has identified some additional areas worth considering.

  • Implementation Consistency: In a preliminary code review, Nomadic Labs found a discrepancy between the intended 42% staking target adjustment and its current implementation in the code (the target is still set to 50%). Further review would help clarify and align the implementation with the proposal’s specifications.

  • Quality Standards: The code behind the submitted hash originates from a branch that was not validated against the full test suite prior to submission. This may require additional validation work to minimize potential functional regressions or security risks. We invite developers of proposals to follow accepted software engineering principles to ensure code quality standards.

As part of the governance process, we strongly encourage bakers to ensure that they’re satisfied with the quality of the implementation of any proposal they vote on, paying particular attention to aspects like code quality and validation status (such as ensuring the Continuous Integration (CI) tests are green). Making an informed decision, especially in regard to code robustness and adherence to quality standards, is essential for the long-term security and reliability of the Tezos blockchain.

In light of these observations, Nomadic Labs suggests two realistic options:

  1. Vote “No” on Qena42: Restarting the governance process would provide an opportunity to ensure alignment with the rigorous quality standards important for blockchain protocol development.

  2. Prepare a User Activated Protocol Override: This has been used in the past to address implementation bugs. It would require bakers to install a client that overrides the protocol time of activation. This would give the proposal authors time to:

  • Fully align the implementation with the advertised specifications and quality standards.
  • Provide assurances to the community regarding any necessary adjustments to support the security and integrity of Tezos in alignment with community preferences.
8 Likes
2

Hi,

Let’s put proposal ideas aside and start with the basics:

I take full responsibility for the bug in Qena42. It wasn’t intentional and should be fixed if possible.

Quality Standard:

The code behind the submitted hash originates from a branch that wasn’t validated against the full test suite before submission.

This is false. The code originates from commit 93b781c06d844645c778245ff46a12f58bc7c0fc in the Tezos master branch, which was tested by the CI pipeline.

Yes, a few modifications we made weren’t directly tested by the pipeline, but the rest of the code was. I think you’ll agree that modifications can’t be tested in the pipeline if we don’t have access to it…

Now, let’s break down the full sequence of events.

Storyline Breakdown:

  • We submitted the Qena proposal in response to Quebec.
  • It was accepted into proposals and entered the exploration phase.
  • The core team requested an MR to integrate Qena. We were asked to:
    • Rebase on Quebec to simplify the review.
    • Remove comments (which we added for easier rollback during proposal development).
    • Since removing comments altered the hash, we were informed it needed to be UAPO for adoption.
  • Qena was rejected in the exploration phase.

Note: We participate in regular developer meetings and expected to discuss how to proceed during these.

  • Immediately following the rejection of Qena, a new version of Quebec was announced without notifying us that it was in preparation. This update did not address the concerns of Qena voters, despite it’s success in proposals and large support in exploration.
  • The new Quebec proposal coincidentally increased the stake by 9x (incentive for smaller bakers?).
  • As the Quebec proposal changed, we were forced to rebase Qena quickly to compete in proposals and mitigate the disadvantage of late injection.
    • We adjusted the target downward, intending to meet the community halfway (we thought we did, in our defense - this was noted in review comments as needing documentation).
    • Applied patches from the original Qena.
    • Kept the rest of the code unchanged and announced the proposal.
  • We entered the second proposal phase without actively promoting Qena; we only responded to the official articles, as I felt they didn’t do justice to Qena.
  • Qena42 won the proposals once more.

Qena42 Integration:

  • This time, we were proactive and rebased on the master branch to expedite the process. (I guess this is why there is a claim it is based on untested branch.)
  • The bug was discovered.
  • Core devs suggested UAPO.
  • A prominent Tezos community member (name not disclosed for privacy) raised concerns about using UAPO in this instance.
  • Suddenly, UAPO became a contentious issue, and we couldn’t get a straightforward answer from NL on its acceptability, despite waiting over 24 hours (and to clarify, we didn’t request UAPO, it was suggested to us).

And now, we’re accused of using code from an unverified branch, suggesting bakers must install a client patching a security-threatening bug that supposedly endangers Tezos’ existence as if they wouldn’t be installing a client with the new protocol version anyway. (I apologize for exaggerating.)

Current Situation Recap:

  • There is no standardized documentation for raising alternative proposals.
  • To test our changes properly, we would need to run pipelines on GitLab, within the Tezos core repository. How a competing proposal is expected to be reviewed, integrated, and tested in the core repo within the required timeframe is beyond me—especially since I doubt the core team will work on Sunday. (note: Saturday announced and Sunday injected Quebec)
  • We cannot easily spin up testnets. Testnets require Alpine containers built from Tezos CI (it’s challenging to build them without CI).
    • Note: We attempted to provide Ubuntu-based containers, but testnets require Alpine.
  • Now it is stated that the proposals must be flawless in the proposal phase, and testing phase seems merely symbolic.

I don’t see how anyone could launch a competing proposal if there’s no infrastructure to support it. Maybe it’s not supposed to be possible. Maybe the expectation is to choose only from pre-selected solutions. Please, let me know.
It seems that only core teams can submit viable proposals, based on their version of “secure, quality-standard code,” while core changes are developed and announced without prior discussion. :person_shrugging:

In closing, I apologize again for our oversight. It’s disheartening that we’ve reached this point and must share the entire history. I mistakenly assumed that Tezos governance was about ideas.

If this setup is acceptable for Tezos governance, I’m prepared to step back from proposing altogether. If you believe something needs improvement in the process, please let our friends at NL know.

By the way, we provided feedback on Slack about what’s needed to streamline the community proposal process.

As always, with love
V

P.S.: Regarding the Qena proposal: We will assist in the preparation of a UAPO if bakers decide to accept it as is (provided we’re permitted to do so). If it’s denied, it will be reinjected either as is or, if Quebec changes, with the additional improvements Quebec provides.

9 Likes
3

Hate to say it but this is another instance of the classic TF/TT/NL political playbook. Rather than help elevate and support the community’s wishes, they obfuscate, accuse, and withhold information to try and paint people who care about tezos in a bad light.

Community rejected Quebec twice. Community voted in Qena twice.

Did NL listen? Nope, but they found an angle, put out a hit piece, and shilled Quebec again in it.

This tells you all you need to know about what happens in the offices of the TF/TT/NL cabal.

This is the true tezos “governance”, most of our top talent have already left over this type of thing happening for the past decade. The usual suspects are TF/TT so I’m surprised to see NL falling in line… but we know who takes care of their new offices, right?

NL: you are great at coding, stick to it. Don’t become TF Cabal or it’s really all over.

3 Likes
4

UAPO or no UAPO, the path forward should be to clarify how community proposals (i.e. core proposals /w minor changes) need to be handled or whether they should not be handled at all.

We’re talking about baker signaling via more popular community proposals vs. baker signaling via continued rejection of core proposals. Which one? To me it seems like the former is easier, more open and less burden long term on the core dev process.

I’d like to cut through all the muck and propose to all of you a vision of the future 15-30 years from now. There is only proposal inflation funding available to bakers in order the fund the operations of the core ecosystem services. Who do we want to propose this funding? Do we want community proposals or do we want to have the core developers decide? If the proposal system doesn’t change in some way (like separating funding/governance & tech proposals), I don’t see how we can logistically take care of the ecosystem long term. Dealing with funding questions seems like a burden to core dev and that it ideally should come directly from bakers.

:heart_hands:

6 Likes
5

Governance is working as intended. If bakers don’t want to accept quebec then they vote for Qena. Just like primate and V can go lobby bakers, that doesn’t mean NL/TT/“TF” cannot…

Or wait, are you trying the the same play you did on twitter about your last grant which was cry about TF being bad and then it was revealed you tried threatening them because you made a bad deal for yourself and want to blame someone for it? We get it arri, you’re butt hurt because you have the business acumen of a field mouse. Be better.

3 Likes
6

Please stay on topic and discuss the merits without personal attacks. Thanks.

2 Likes
7

Keep analyzing this as a process issue. When was the last time the active protocol on mainnet passed on-chain governance? It’s been a while and that should be an indication something is wrong that is not just a bit of isolated sloppiness.

If community proposals need more time then lengthen the proposal period, split the time into submission are taken and voted on so there’s no benefit to rushing.

No one will be happy about this, but reset and run it again. Something to learn from each time that will prevent more problems later.

1 Like
9

In fact, for ordinary Tez holders, they also have their own ideas and the desire to put forward proposals. Due to network censorship, language barriers, and lack of technical knowledge involved in submitting proposals, they are unable to publish valuable proposals that they have in mind.

In addition, some proposers with ideas lack the ability to materialize their proposals. Therefore, the Tezos community needs to collect information to help these groups publish their own proposals more easily and make the publication of proposals more free and convenient.

The community governance of Tezos needs reform. Low-quality proposals are eroding the efficiency of protocol upgrades.

1 Like
10

It would require bakers to install a client that overrides the protocol time of activation.

Just a note on this: early UAPO does not require any action from bakers, unlike UAU or late UAPO.

  • Early UAPO – UAPO integrated into the only client released.
  • Late UAPO – UAPO released after the initial protocol release (not applicable to Qena at this point).
  • UAU – Upgrades outside the standard governance schedule, such as the recent Paris upgrade.
3 Likes
11

What I want to express is that Tezos lacks a process to turn ideas into high-quality and safe proposals. This is the reason for the generation of low-quality proposals.

The community needs to build a proposal process to make proposals as democratic, liberal and safe as possible.

1 Like
12

In other channels, we noticed some confusion about the extent of the Qena42 proposal.

Qena42 differs from Quebec in only one line effectively. The change of target to 42% would be another line, but since it isn’t in the protocol code - it is in the default chain parameters (the bug), the effective difference remains just that one line. Everything else is the same as Quebec.

2 Likes
13

I would like to start by saying thank you to Nomadic Labs for testing Qena42 and finding and reporting this bug.

That said, the tone of this announcement comes across as the petty chiding coming from someone with a bruised ego because they didn’t get what they wanted, directed toward the proposal creators and the bakers.

I lost track along the way, and I think perhaps it has recently gone down somewhat, but until just a year or two ago it used to be that pretty close to 50% of the proposals from Nomadic Labs have had significant bugs discovered in them during or just after the governance cycle, requiring urgent user activated overrides/patches. And how many times did Nomadic Labs recommend people vote against the proposal and restart the governance process “to ensure alignment with the rigorous quality standards important for blockchain protocol development”? That number is pretty close to zero.

How are we supposed to take Nomadic Labs seriously when they take a high and mighty stance and shame the proposal creators for making mistakes and not following “accepted software engineering principles” as if Nomadic Labs has never submitted buggy proposals? Or when Nomadic Labs chides bakers for not “paying particular attention to aspects like code quality” when bakers have repeatedly learned from Nomadic Labs themselves that buggy proposals are to be expected?

An idiom comes to mind: “People who live in glass houses shouldn’t throw stones.”

While I think the point could be argued, if we grant that Nomadic Labs does indeed have “rigorous quality standards” after all, and that they apply these standards to their own work as strongly as they do to the work of others, then let us not forget that it took them literally years of creating buggy proposals for them to develop those standards.

So let’s cut some slack to the team that created Qena42, which is basically their first proposal, shall we?

3 Likes
14

I don’t mean to bring more heat to the debate, but we definitely need to zoom out before making such announcements, even more when the difference between the two proposals is just 1 LINE OF CODE.

I hope we can find commom ground and push in the same direction again which is ultimately to improve the Tezos Protocol.

We just need to improve the process for non-core developers to make proposals as well, so, we can truly claim we have an open system in which truly everyone can participate. I guess it is hard but we have done harder things before.

Cheers.

3 Likes
15

I’ve often questioned the application of User Activated Protocol Overrides (UAPOs) by Nomadic Labs, especially where they don’t directly relate to technical bugs. However, in this instance, it’s essential to acknowledge the diligence Nomadic Labs has demonstrated.

Nomadic Labs has taken responsible, timely action by identifying a clear inconsistency between Qena42’s stated specifications and its current implementation and then transparently notifying the community. This step ensures that bakers can make informed decisions with a clear understanding of the proposal’s implications. Furthermore, Nomadic Labs has rightly highlighted the established procedure in cases such as these: proceeding with on-chain governance and, in exceptional circumstances considerin a UAPO (is this occasion so exceptional? that’s the whole point Imho).

To those advocating for a more collaborative approach from Nomadic Labs in crafting a consensus proposal, I wholeheartedly agree with the need for adaptation. Yet, I believe it’s also crucial to recognize that Nomadic Labs took active steps to foster community input earlier this year. In August, they opened a forum thread to gather feedback on the Quebec proposal, thoughtfully considering well-defined amendments, including adjustments to the maximum curve and expanded staking capacity for bakers. While consensus requires compromise, I would contend that Nomadic Labs made significant accommodations to refine their initial proposal. Nonetheless, it appears these adjustments did not fully satisfy the proponents of Qena, who elected to submit an alternative proposal with seemingly minor code changes but with substantial functional implications (a totally legit option on the other hand).

1 Like
16

To those advocating for a more collaborative approach from Nomadic Labs in crafting a consensus proposal, I wholeheartedly agree with the need for adaptation. Yet, I believe it’s also crucial to recognize that Nomadic Labs took active steps to foster community input earlier this year. In August, they opened a forum thread to gather feedback on the Quebec proposal, thoughtfully considering well-defined amendments, including adjustments to the maximum curve and expanded staking capacity for bakers. While consensus requires compromise, I would contend that Nomadic Labs made significant accommodations to refine their initial proposal.

Does this mean that a party, which is meant to be apolitical and does not participate in voting, is deciding how the proposal should look, and the voting parties are expected to compromise to meet them halfway?

Shouldn’t they listen to the parties participating in the vote and offer the options they seek, rather than deciding on their own?

2 Likes
18

I am not affiliated with Nomadic Labs, but as a Tezos Foundation subsidized entity I would assume it proposes what is believed the best for the protocol (not just for some big bakers)—as any team, including yours, is entitled to do. The decision to incorporate third-party feedback is, as I understand it, at the discretion of the proposer, and in this instance, I note that Nomadic Labs has indeed made adjustments based on external input.

2 Likes
19

I am very excited by all of this! :sweat_smile:

Governance is a conversation, and we are having it right here and now. And by using our vote. I love it :heart: Whomever said governance was supposed to be easy or straightforward misunderstood the point of having it :wink:

Great work by Nomadic for uncovering the bug and posting here! Thanks :pray:

My two cents is that bakers voted for the intention behind Qena42, and it is totally fine with a UAPO to fix the bug.

On another note I believe it is unrealistic for bakers to verify CI tests or read through proposal code. We, the community, depend on Nomadic and other professionals to do this, and let us know if there are any problems. This post is a great example of how this can (should?) be done :+1:

Let’s keep the vote going and prepare a UAPO in the meantime :rocket:

2 Likes
20

Oh, one more thing!

The fact that it is a high bar for someone to create a proposal (read the code, understand it, learn the language, run tests etc.) I think is a very good thing!

It prevents cybil attacks on our governance process !! :ok_hand:

But it is still very important that it is possible. Qena42 proves this!

If the “official” proposals start straying far away from what the community needs and wants, I will dig into the code myself :wink:

21

It prevents cybil attacks on our governance process !! :ok_hand:

High bar does not prevent Sybil attacks (see recent XZ sabotage). Reviews do.

Security through obscurity is generally not a good idea.

1 Like
22

A proper vote can not be made until bakers know if Qena42 will be deployed with or without UAPO.