A Heads up! to Tezos bakers: getting ready for 10s block time in 2024

Research and Development
1

Earlier this year, the Mumbai protocol upgrade had successfully halved block time from 30s to 15s. The Nairobi protocol, the one currently governing Tezos Mainnet, introduced further optimizations to reach consensus faster. For Tezos to remain a cutting-edge blockchain, we plan to reduce latency further, taking minimal block time from 15s to 10s in an upcoming protocol P upgrade proposal.

This post aims to give the community a heads-up on the upcoming changes in order to make sure we are all ready.

Over the last few months, we developed a testing and profiling framework which simulates the behavior of Tezos Mainnet under stress – i.e. under a full load. We had two goals: the first was to analyze Octez and improve its overall performance. The second was to measure the impact of these changes in a realistic lower-latency environment.

These experiments enabled us to identify major bottlenecks in Octez. We then fixed them one by one incrementally, and validated their impact again using this custom-built framework. The results give us enough confidence to propose reducing block time in the next protocol proposal.

It has always been our core belief that bakers should be able to participate in Tezos consensus with affordable, lower-end infrastructure. This is a core strength of Tezos which inherently impacts decentralization, so we want to preserve it as Tezos evolves.

Therefore, our experiments included setups with the following specification:

  • 4-core Intel CPUs @ 2.80GHz (we tested both arm64 and amd64/x86-64 architectures).
  • 8GB of RAM
  • SSD-like I/O performances
  • A 100Mbps network bandwidth

As for signing blocks and operations, our experiments simulated a signing delay of 1s, a ballpark figure comparable to a Ledger Nano S’ performance.

With the implemented optimizations and the configuration above, we managed to deterministically achieve successful experiments that determined that a full-load Tezos Mainnet would be safe with a minimal block time of 10s – where safe means that more than 99% of blocks are produced at round 0.

We plan to publish a blog post detailing the full methodology and the results obtained during this project. In particular, we will show that we could go for more ambitious latency reductions assuming high-end configurations.

In the meantime, we kindly ask you to use this thread to share with us details about your baking setups, ask further questions, and raise any concerns you may have.

17 Likes
2

100Mbps for both up and down? I have 1Gbps down but only 30Mbps up. Is this a scenario that was tested?

2 Likes
3

We currently dedicate 2 cores and 12 GB RAM to the octez suite, and so far we rarely miss attestations. We can dedicate more CPUs if needed later, and 12GB RAM appears to be much more than actually needed, 8 would largely suffice.

Pretty “often” though, we miss a few (2~4) of the firsts attestations of the cycle, usually within 3 or 4 minutes. We don’t know whether this could be tied to the bottlenecks you identified. We suspect the data compacting performed by octez-node in the background to be the cause (extra I/O consumption, CPU resources are not involved as we tried with 8 cores), but we are really not sure.

3 Likes
4

How was the testing conducted? I am interested in observing a setup with two bakers on the mainnet: one located in Texas, USA, and the other in Singapore. Both should be configured behind firewalls, disallowing incoming connections. I would like to see analysis of the time taken for the baker in the USA to produce a block and for it to be accepted by the Singaporean baker, and vice versa. This process should be repeated with multiple blocks to gather significant data. Once we have this information, we can proceed to discuss performance metrics.

By the way, @Boulange , feel free to disregard the concern about the 4-core configuration. Since Octez operates on a single thread, it shouldn’t be an issue.

2 Likes
5

But octez-node spawns a child process. These two consume about 110~150% CPU by the beginning of the cycles (the first hour maybe). This must be the data compacting I was referring to above.
But the rest of the time it’s true their sum stay far below 100%.

2 Likes
6
  1. I live in one of the top 5 largest cities in USA. My ‘business class’ internet is only 25Mbps/6Mbps. (Yes, I know USA sucks but this is what I have). Are you testing with real-world / “normal people” configurations? Or will this change enforce hardware minimums on everyone?
  2. 10s block times. Why? There’s no explanation here on why this matters. 15s is already damn fast. Why do we need it to be faster?
1 Like
7

Tezos has one of the highest latency among L1s and it’s routinely cited as an obstacle to adoption, particularly in defi. Generally speaking, low latency has worked very well as a proposition for chains like Solana and Avalanche.

I think the best of both world is to have most activity in L2 with decentralized sequencers, because the latency / decentralization tradeoff is different for sequencers than for L1 nodes (sequencers cannot cause the type of faults that L1 block creators can and therefore it’s safer to impose very low latency requirement on them). So the good news is we can have < 500ms latency in a future L2 without imposing this requirement on the L1. However, this won’t come for a while, and much of the activity is still on the L1 where latency is uncompetitive.

The connection between latency and bandwith is not obvious by the way, it comes from the number of attestations that need to be transmitted. Moving to BLS signatures with aggregation could help here, but won’t be supported well by hardware wallets and HSMs.

3 Likes
8

Thank you for sharing your concern!
The mention of “100Mbps network bandwidth” was indeed misleading. What we intended to convey is a high-speed network connection with low latency. Currently, the Octez node requires approximately ~2MBps, and with our ongoing improvements, this figure is expected to decrease. If you are running a node and baker with a low-latency connection, there should be no impact on your end.
We hope this clarification is reassuring.

3 Likes
9

About your first concern, please see our answer to RichAyotte (A Heads up! to Tezos bakers: getting ready for 10s block time in 2024 - #8 by zaynah)

1 Like
10

You are right about mentioning physical network delays. In fact, our results demonstrate a safe block time of less than 10 seconds. To account for physical network delays, we incorporate a margin of 3 seconds.
In our ongoing efforts to enhance confidence in the 10-second block time, we are conducting experiments involving artificial network delays. These experiments involve the introduction of randomly selected delays to simulate more realistic distances between nodes. We plan to share the outcomes of these experiments in the upcoming blog post.

2 Likes
11

The physical network delays constitute only a portion of the issue. The other aspect involves the distribution of the network and the number of hops between nodes. This is why I insisted on positioning both test nodes behind a firewall. By doing so, they cannot directly interconnect, allowing us to obtain a sample of the typical delays caused by hopping.

Introducing random artificial delays would not accurately account for these factors.

1 Like
12

We have been monitoring these types of situations at the beginning of the cycles.

Could you please clarify what you mean by ‘missing attestations’? Does your baker generate attestations, only to find that they do not appear in the final blocks? Alternatively, is your baker unable to attest these blocks?

1 Like
13

By “missing attestations”, we mean not injecting - at all or on time, we’re not sure - the attestations that our baker is eligible to produce. It’s difficult to be more precise because the octez-baker logs don’t tell much: there are no error messages, just a lack of injection.
For example, this happened yesterday at the very beginning of the cycle 685:

image
image418×516 45 KB

1 Like
14

Hello! Thanks for sharing. We use:

CPU Intel 1.80GHz 4 core
RAM 64 Gb
ssd Adaptec hardware raid
net bandwidth 100

Pretty “often” though, we miss a few (~1-2) of the firsts attestations of the cycle, usually within 3 or 4 minutes.
Could you please tell, are there any upcoming plans to further modify the specs used to benchmark the underlying engineering effort?

2 Likes
15

I have looked at the block 4775941, and it is a “big” block which can take a lot of time to be applied on some machines. This is a recurring situation that should be mitigated with our recent improvements. Your node, should be able to attest earlier on this type of blocks.

2 Likes
16

Strange that some bakers encounter the missed attestations at the beginning of a cycle. I typically don’t, and i have hardware that’s just above the minimum required. My bandwidth ranges between 150MB - 300MB, and I’m in the middle of the USA, don’t know if that helps. I took a look at level 4775941 that @Boulange posted, and I see that there only were 6,020 endorsements for the whole block, and that’s lower than the usual 6,970+. Same for the first three positions of 685. Curious what causes these lower numbers. https://nomadic-labs.gitlab.io/teztale-dataviz/DelayToConsensus#server=https%3A%2F%2Fteztale-mainnet.obs.nomadic-labs.cloud%2F&rangeLowerBound=-20&rangeUpperBound=4775945

2 Likes
17

Very good to know, thank you for investigating!

18

Thanks, @ACoquereau Your post landed at the same time as mine and addresses the question I had about these missed attestations. Hope you’re right that the proposed changes will improve things. Slightly off-topic question: what makes a block “large” ? Do you tell by the amount of storage, or by gas used or reserved?

1 Like
19

Your observation is very interesting. The same thing happened at the very beginning of the 686, earlier today (CET):

image
image1538×819 69.6 KB

20

4 cores, really. Last i checked opam used is not multi cpu, it runs on one. 2 cores should suffice, especially in a virtual cpu. Or has this changed.

2 Likes