Network Scaling Issues

Mods - not sure if I’m posting this in the right section, but please free to move if need be.

I have tried to discover (through various channels - Reddit, Discord, Twitter, etc.) what has been clogging the network this past week or so. Haven’t seen any updates from developer channels either (my tezos-dev Slack access was revoked for whatever reason - maybe inactivity, I don’t know).

Can anyone comment on the real issues around the recent network slowdown. There seemed to be rumors around mempool issues, RPC calls overwhelming the network, bakers not updating universally due to a bug, etc. Not sure if any of this is true, but would be great if someone with the understanding of the situation can comment/clarify.

If it’s a scaling issue, would adding more nodes help? How does something like TezEdge help with scaling in situations like these? Is this indicative of Infura type of situation on Ethereum (I hope we can avoid this, so enterprise-scale dApps don’t have to pay-to-play)?

I try to be helpful to technical folks, so if adding more indie nodes is a thing, would be great to know that as well.

If it’s a baker-specific issue, it would be surprising to discover that bakers not updating their nodes can actually bring the network to a halt.

If it’s simply a programming issue/bug, I guess nothing to discuss (but still please confirm)!

Thanks!

3 Likes

It doesn’t. TezEdge is just a tezos node written in Rust instead of OCAML.

I’m assuming you’re aware network usage is up a lot (my guess is > 5x) because of SalsaDAO?

Of course, but not sure whether you’re justifying the slowdown? It’s just one dapp.

Edit: By the way, this is not coming out of the left field - I, like many others, have experienced missed transactions. Keefer from Kolibri even tried to assist me and confirmed an operation on a borrow was simply “missed” due to the network being bogged down. If hicetnunc and salsadao alone can bring the network to a crawl, how do we get get ready for the Uniswap’s and the Compound’s (millions/billions of transactions) of the world. I hope you understand the seriousness of the issue @emchristiansen .

1 Like

Thanks for posting. I’d like to understand this better myself.

If you scroll down, you’ll find your answer.

Is it confirmed that 8.3 actually fixes though @AlexL? Seems like there are still a lot of reports of missed/backtracked/failed transactions. That’s why I raised the potential probability of bakers not fully updated… In which case, would still be great to confirm that the network’s now waiting on everyone to update. If this is the case, then I reckon we face similar issues in the future, no? Unless there’s active pursuit of a solution to prevent this in the future?

8.3 has fixed many of the missed/failed transactions, however, a lot of bakers including some large exchange bakers are still on 8.2 so we will naturally still have those missed/failed transactions. Issues in the future like bakers not upgrading from 8.2 to 8.3, doubtful. Currently, if a baker is on 8.2, they can still bake/endorse blocks and the chain still recognizes it. If we have protocol upgrade and you don’t upgrade to 9.0 etc, then you won’t be able to bake/endorse.

2 Likes

So you are saying that afayk 8.3 completely fixes the problem but since many bakers have not upgraded yet, there still are some missed/failed transactions.

Is there any place where you can see what version bakers are running?

We show a 24h summary of baker versions seen in our Baker List and the version number for recent blocks baked at Activity/Blocks if that helps.

1 Like

Thanks a lot. That is great info.

It seems that bootstrapping is taking a little longer than normal.