Etherlink 7.2 — undisclosed security upgrade via Fast governance on Wednesday, September 16th

Research and Development
1

This is a joint post from Nomadic Labs, TriliTech, and Functori

Hi everyone,

What is coming

As part of our continuous security review and hardening efforts on the Etherlink 7.1 (Ganesha) kernel, we will submit a new kernel, Etherlink 7.2, through Etherlink’s Fast governance mechanism on Wednesday, September 16th. It contains security patches that mitigate potential liveness attacks on Etherlink. No funds are at risk. It carries no new features: it is the Ganesha kernel currently running on Mainnet, plus these patches.

As we have done before with security issues found on Etherlink Mainnet, we cannot share the technical details of these patches ahead of the vote. This means bakers will not be able to reproduce the kernel’s root hash from public source before voting. Once the upgrade is activated and Mainnet is protected, we commit to publishing the source of the patches, the full changelog, and the reproduction instructions for the root hash, so that anyone can check that the kernel bakers voted on contains exactly what we said it contains.

Why fast-track

These issues are best fixed in the kernel rather than mitigated at the sequencer level, and there is no reason to leave them open for a full ten-day Slow governance cycle. Fast governance lets us close them in a single day, with full disclosure right after activation. We are posting now, ahead of time, so bakers have as much notice as possible, since fast governance voting windows are short.

:date: Voting periods

  • Proposal period: fast governance period 693, from L1 level #14,971,489 to #14,976,288. Expected to open on Wednesday, September 16 at ~08:31 CEST and close at ~16:31 CEST.

  • Promotion period: fast governance period 694, from L1 level #14,976,289 to #14,981,088. Expected to open at ~16:31 CEST on Wednesday, September 16 and close at ~00.31 CEST on Thursday, September 17.

The levels are fixed; the times are estimates and tend to drift later as blocks are produced slightly slower than the nominal schedule.

:backhand_index_pointing_right: Fast governance periods last ~8 hours. The Proposal period covers most of Monday’s working day and requires a 5% quorum to move to Promotion. The Promotion period runs into Monday evening and requires a 15% quorum and an 80% “yea” supermajority. If you can cast your upvote during the Proposal period and your “yea” vote in the evening, it will make a real difference.

:key: Kernel root hash: 009e07373751a43acbf0cd747b404f45130e0ad561c4b7da937e02eb812bd0f7e8

You can upvote the Etherlink 7.2 kernel during the Proposal period using the following Octez CLI command:

octez-client call KT19oUVQPnVLuUBYXrBVd46WJnNAMpqkKSwo from <baking_key or voting_key> \

  --entrypoint "upvote_proposal" \

  --arg "0x009e07373751a43acbf0cd747b404f45130e0ad561c4b7da937e02eb812bd0f7e8" \

  --burn-cap 0.11

If the Proposal period is successful, you will be able to vote for the kernel during the Promotion period:

octez-client call KT19oUVQPnVLuUBYXrBVd46WJnNAMpqkKSwo from <baking_key or voting_key> \

  --entrypoint "vote" --arg '"yea"' \

  --burn-cap 0.11

See How is Etherlink governed? | Etherlink documentation for further instructions on how to participate in Etherlink’s governance.

:folded_hands: Thank you for your continued support of Etherlink, and for helping strengthen its security posture.

4 Likes
2

`Once the upgrade is activated and Mainnet is protected`

This is the thirst security upgrade using fast track in only two months - which never happened before in L1. I`m once more telling my baker my vote is a NO on these things. Don`t get me wrong I have a big stake on Tezos since ICO, but on a moral side I prefer to see things getting destroyed than keep going with bad behaviour.

`We are posting now, ahead of time, so bakers have as much notice as possible, since fast governance voting windows are short.`

One day before notice now is `ahead of time.` :joy: Poor bakers cannot even go on vacation.

3

I, for one, am fully sick of the ‘trust me broh’ release cycle we’re on. I won’t participate in it any further.

5

You know @skullzarmy, it’s not possible to disclose security vulnerabilities before the fixes are in production. Otherwise, you’re simply telling malicious actors what to exploit.

2 Likes
6

fix the processes that are pushing vulnerabilities into production. this is the formal verification chain is it not?

if it is then even with a blind hash you can prove certain properties about the code are going to hold right? so do that

7

Why fix the process when they can just ignore and destroy the goodwill they have with bakers with their vibe coding?

8

I recognize what’s happening is unfortunate but necessary.

Let me just say, this is why we need more psychology and perhaps a little less technology focus. The posture that’s been assumed by the devs and by the funding entities invites uncertainty, speculation, and distrust. That has consequences when we need people to come together in situations like this.

I’ve said before that asking bakers to come together and do things by the book breeds affinity and teamwork. The relationship we build between these moments matters when security requires us to ask people to trust what they cannot yet verify.

We can keep the details of a vulnerability confidential while still being more open and engaged with each other. That goodwill needs tending long before we need to draw on it.

Do you want people to celebrate your work or push back against every single thing that you do?

5 Likes
9

The core devs and those that are being trusted to write competent code has done the exact opposite in the past half year. It has been one emergency security upgrade after another. Maybe if the core devs did the bare minimum of talking to the people who they’re asking for help in pushing their fixes explain a bit about what went wrong with I don’t know, maybe a post-mortem for starters, would go a long way. Call me old fashioned, but the LEAST they can do is TELL US what went wrong with ANY of these emergency upgrades.

10

It is equally impossible to put a stamp of governance and approval on something I cannot see. I appreciate your position and I hope you can appreciate mine. There’s got to be a better way forward.

3 Likes
11

@skullzarmy Of course I value your opinion. I acknowledge that it is frustrating to vote for something without knowing what it is. When you are being asked to vote, you naturally feel entitled to know what you are voting for, that is fundamental.

When it comes to security, though, we cannot take the risk of disclosing a vulnerability that would almost certainly be exploited, and that is just as frustrating for the team responsible for handling it.

There is, however, a way to verify after the fact that the announcement made when calling for the vote was truthful: reverse-engineering the resulting kernel with Claude and reviewing the source code makes it possible to verify that what was voted on does indeed match what was stated. And if one day that turned out not to be the case, those who made such a misleading proposal would permanently lose their credibility: no one would trust them again. While this is not a guarantee, it does provide a degree of assurance that the proposal is being made in good faith.

Post-mortems afterwards serve that purpose as well.

It is imperfect, I agree, but it is the only workable solution identified so far.

In any case, I obviously understand your point. I would just ask you not to see this as disregard for bakers and users in general. It isn’t. We would all prefer things to be different.

3 Likes
12

That`s a good line of thought. But this all could be avoided if all this upgrades were going trhought testnet before just releasing them using fast track into mainnet.

Again I will say here that it may be harmless now but in the future a developer with bad intentions can easy exploit these mechanism of voting into something that no one knows what it is.

13

This goes along with the overall closing off of core dev (and now lots of tooling also) from the rest of the ecosystem over time.

In a more finished form, everything would behave more like a tech bazaar where all layers intermingle. What we have right now is a strict corporate top-down structure. In this arrangement, it pays no benefit to interact with the community.

The thing that gets me is, even in the corporate structure arrangement, the move nowadays is to empower multiple employees to discuss the product all the time on X, to mimic grassroots energy and participation.

I’ll save my further elaborations on this for a different time.

Let’s get 7.2 passed and go from there, hopefully to bigger and better things.

14

If we vote NO in just one of these releases it will put their job into risk and then it will change everything.

15

That’s not how things work. If we blockade legitimate (albeit unfortunate) progress that they’re trying to make, that’ll only lead to further distance from them to us.

Cultural change here is only possible from the inside as things stand. Devs answer to management, management answers to their execs and that’s where change needs to come from.

Ladies and gentlemen, one day Tezos L1 will have to vote to mint the funding for everything (that can’t self-sustain itself). That’s the final state of things. Everyone will be judged by the same measure. We all ought to all start heading in this direction sooner rather than later.

16

You are in AI era now. Have you seen recent patch tuesday from Microsoft with 1000 undisclosed security bugfixes?

1 Like
17

A testnet instance with an undiscovered exploit has 2 scenarios:

  • spotted by a bad actor but not not a whitehat or internal audit => exploited if the code gets promoted to mainnet
  • spotted by a whitehat or internal audit and a bad actor before mainnet => patched, maybe a bug bounty paid

If you look at the recent Cosmos staking hack, it’s a salutary tale in how not to do SecOps & responsible disclosure, and it had this same scenario as you suggest - a public fix, some networks upgraded before others & limited/missing disclosure comms via secure channels to the downstream chains…

It’s a tough choice.

2 Likes
18

Are you seriously comparing tezos core devs to microsoft devs? If you’re going there, we are in BIG trouble.

19

How about releasing a post mortem which they have YET to do? Let’s start there. Anything else is just noise.

20

Just giving my two cents considering the latest tension rise.

What about a Tezos Community Security Council DAO, a group of individuals/entities/bakers that are real-life identified and check extremely strict requirements, but do not belong to the core dev teams but to the community itself. Essentially community members which are trusted with classified information.

Some basic requirement examples:

  • Hold X amount of XTZ;
  • Hols XTZ for X amount of time;
  • Knowledgeable in a relevant area to be able to understand security issues (software, security, etc);
  • Some form of security clearance;
  • etc

Members are voted in and out by the security council itself, anyone can try to join the council.

Core dev teams would have to schedule one/two meetings (cover enough members to pass the approval) with the council members, showcase the PoC via video call (not share the exploit code), explain with additional detail the issue being adressed, how it was patched, etc.

Once this is done, security council would vote on the approval (this needs to be carefully thought on the percentages for approval/reject) of the upgrade and stamp their own ‘seal of approval’. Security Council members would also share the burden of the community interaction, they need to be somewhat active on this forum.

So the “Tezos Community Security Council” would hold the burden of building trustworthiness with the community, this could be done by simply being active in foruns and social media, orchestrating some work on the community education side via virtual meetings, social media posts talking on typical scam vectors, etc.

And if the Tezos Core Dev teams want the seal of approval of the council, they would have to somehow ‘convince’ it. This ‘seal of approval’ would be naturally on-chain, and core dev teams would be able to add it to their upgrade post here in the forum.

At the end of the day, it’s a matter of trust between two sides over a sensitive subject that needs to be treated carefully. But the current way it’s being handled isn’t sustainable in the long run, something needs to change.

An important detail in my opinion, the founding members of the council would need to be validated and picked by the core dev teams themselves (subject to the community reaction to this decision of course), the starting members are extremely important and need to be vouched by both sides.

Just a lunch thought, take the idea as an abstract :sweat_smile: I do think its much more productive to currently trust and vote to approve though (at the end of the day we’re talking here about the core dev teams, it’s a security upgrade on something they built themselves), and hold a solution discussion for this on a different thread, you can then add a post here pointing to it, I’m not an active/known enough member to lead that. Once a solution is heavily backed by the community than the core dev teams would have to put effort into endorsing to whatever procedures were decided there or hold the consequences of dragging the increase of community distrust. But I also think we need to start building some autonomy and bring community solutions to the discussion table.

21

As a former whitehat hacker, I’m just telling that you are likely to see even more of these, before it gets better regardless of “quality of code”. And I strongly advise against sharing even tiny details with public before it is safe to disclose, because I certainly know what I would do if I was a baddie

P.S. it is always fine, if issue got identified, patched and not exploited. We can complain all we want after, but I’m personally very forgiving with such cases as I know how little choice devs have

1 Like