Turning Ithacanet into a permanent testnet: Ghostnet

Research and Development
1

As a prerequisite for a permanent testnet on Tezos, we need a mechanism to centrally upgrade its protocol to closely follow mainnet.

The governance dictator key is going to be merged into octez soon, which means such a mechanism will be available.

See previous agora discussion as well.

Therefore, we are proposing to turn Ithacanet into a permanent testnet, using the following method:

  • shortly after Jakarta activation, we upgrade Ithacanet to Jakarta using a user-activated upgrade: this requires most testnet operators to change their node’s configuration to hard-code the upgrade. This needs to happen soon, in less than 2 weeks.
  • after Jakarta activates, we change this testnet’s name across the ecosystem to Ghostnet
  • again, shortly before K protocol activation, we coordinate with node operators to perform an user-activated upgrade to K protocol
  • K protocol will contain the logic testnet governance dictator key (only active on chains other than mainnet, obviously). It will also contain special code for Ithacanet/Ghostnet (identified by its chain id) that sets the dictator key to a key controlled by Oxhead Alpha, the company operating the teztnets.xyz platform
  • moving forward, Oxhead Alpha will perform Ghostnet upgrades administratively, using the dictator key. Testnet node operators no longer need to change their configuration at each upgrade (they do need to keep their nodes up-to-date with the latest versions of Octez).

This is the plan. Any comments or concerns, please let us know here.

Update: ghostnet is live. See details.

9 Likes
2

I would love it to happen and HOPE it does! But at this point I remain skeptical (politics always win) and will believe it when I see it. (Nothing would make me happier than to be proven wrong)

Fingers crossed and keep us posted! :crossed_fingers:

3

The Ithacanet PrimateE baker is on board and awaiting further instructions.

1 Like
4

Very excited for this, its massively needed for wallet developers or anything that requires interacting with a service/app outside of their own company. Its still quite painful to find dApps that do their full setup for each testnet. I still do most of my testing on Mainnet. I’d love to be able to deploy an automated testing suite hitting a testnet, but its only possible if we can do this and crack a few challenges

If I understand the current system correctly, there is a limited number of faucet accounts created with each new testnet, which eventually run dry. How will this new testnet ensure access to testnet-XTZ in a way that won’t run dry? and accessible to a new developer that can’t find a non-empty faucet account to get started with?

I haven’t checked lately, but previously a few dApps would deploy their contracts to a testnet, but due to a lack of other content, they would often create their own make shift tokens “token-1”, “token-2”, “token-3” and so on. These often didn’t give great test data. They didn’t have much variation in metadata, weren’t indexed by indexers, didn’t provide icons on IPFS etc. Once this testnet becomes official, it would be great to see a push to encourage devs to duplicate as much as possible. For example, have a testnet version of Kolibri, USDtz, tzBTC … Have the testnet dex’s use those tokens and not their own fake ones. Have indexers index them etc. The testnet will become truly useful then if we can get as close to a carbon copy as possible

Any thoughts around preventing bad actors/bad tests from, e.g. bleeding faucets dry, taking large amounts of liquidity from pools etc, either accidentally or maliciously. I remember at one point trying to test liquidity baking on testnet, with others only testing in one direction the exchange rate went to something like 100,000 XTZ == 1 tzBTC making it practically impossible to use. Situations like that need to be avoided in some way, and ideally automatic and not having to ping teams on slack

1 Like
5

Nice to see a permanent testnet finally come to fruition.

politics always win

That’s the beauty (or drawback, depending on your perspective) of having a dictator key. It doesn’t really matter what anyone else thinks.

I suppose if people are against the idea then they could shut down their bakers on Ithacanet, taking enough baking power with them to get the chain stuck. But it wouldn’t be difficult to move forward with Ghostnet by spinning up a new testnet chain where the dictator has the majority of staking power as well, so the dictator truly would have about as ultimate control over the chain as possible.

Once the governance dictator key is merged into the codebase, then resistance is futile. Ghostnet will happen if the dictator wills it, no matter who else is opposed.

That said, I’m not sure why anyone would be opposed to it.

6

This faucet https://tezos-testnet-faucet.netlify.app/ by @avysel is promising, we are currently trying to get its future development funded, it could become the default faucet of teztnets.xyz. My favorite part is that it is beacon-capable and does not require using the octez CLI.

It’s less clunky than the current faucet which relies on a limited set of 10k pre-accounts. This is just a big wallet, so it can be refilled over time if it gets empty.

Any thoughts around preventing bad actors/bad tests from, e.g. bleeding faucets dry,

We can look at what’s done elsewhere. We can use twitter as identity verification mechanism.

But what’s unique about tezos is that you need 6k tokens to become a baker, and we want anyone to be able to start baking on testnets without asking. So there is no way around giving people a lot of tokens at once.

7

I still don’t understand why you want to follow mainnet. (I don’t understand why the upgrade needs to be centralized either but let’s discuss one thing at a time.)

If we wait for a protocol to be activated to start testing dapps on it then protocol devs get feedback three months too late to fix the protocol.

IMO, what is really needed is a permanent mondaynet.

1 Like
8

Please lets dont start this fight again. Literally almost all Dapp devs are excited about this that its finally happening. And in the spirit of Tezos, if you think a permanent Mondaynet is needed then step up take the initiative and introduce a permanent Mondaynet - like Nicolas is doing now with Ghostnet.

1 Like
9

We will update Ithacanet to Jakarta protocol at block 765,952. The first Jakarta cycle will be cycle 187. This should happen on June 28th a few hours before the mainnet upgrades to Jakarta as well.

In order to remain on the network, every ithacanet baker must do the following:

  1. upgrade to octez 13.0
  2. turn on the jakarta baker alongside the ithaca baker
  3. configure the jakarta user-activated upgrade

Configuring the user activated upgrade requires a change to the tezos json configuration.

The tezos json configuration file is typically at ~/.tezos-node/config.json but may be in a different location.

To add the jakarta user-activated upgrade, issue the following command:

tezos-node config update --network https://teztnets.xyz/ithacanet

If your config file is not at the default location, add --config-file /path/to/config.json to the command above.

You should now see the following in your config.json:

  "network": {
    "chain_name": "TEZOS_ITHACANET_2022-01-25T15:00:00Z",
    "genesis": {
      "block": "BLockGenesisGenesisGenesisGenesisGenesis1db77eJNeJ9",
      "protocol": "Ps9mPmXaRzmzk35gbAYNCAw6UXdE2qoABTHbN2oEEc1qM7CwT9P",
      "timestamp": "2022-01-25T15:00:00Z"
    },
    "user_activated_upgrades": [
      {
        "level": 8191,
        "replacement_protocol": "Psithaca2MLRFYargivpo7YvUr7wUDqyxrdhC5CQq78mRvimz6A"
      },
      {
        "level": 765952,
        "replacement_protocol": "PtJakart2xVj7pYXJBXrqHgd82rdkLey5ZeeGwDgPp9rhQUbSqY"
      }
    ],
    "sandboxed_chain_name": "SANDBOXED_TEZOS",
    "default_bootstrap_peers":
        [ "ithacanet.teztnets.xyz", "ithacanet.smartpy.io",
          "ithacanet.boot.ecadinfra.com", "ithacanet.kaml.fr",
          "ithacanet.stakenow.de:9733", "ithacanet.visualtez.com" ],
    "genesis_parameters": {
      "values": {
        "genesis_pubkey": "edpkuYLienS3Xdt5c1vfRX1ibMxQuvfM67ByhJ9nmRYYKGAAoTq1UC"
      }
    }
  }

A restart of the node is required for the configuration change to take effect.

2 Likes
10

I agree with Rafoo’s point about following mainnet (although not sure if mondaynet is the right candidate). I think a permanent testnet is super important and has been massively lacking, but we do probably need it pointing towards a future protocol, or having 2 permanent testnets (1 following mainent, 1 set to whatever protocol makes it through the first round of voting)

I see advantages to having a stable testnet where devs can be assured the existing dApps and frontends are all working as normal, and they can be tested without having to spend real money. The current support for testnet apps is not great, forcing most users to test on mainnet. But that on its own still leaves a big gapping “how do I test my app won’t break in a month” problem, which brings us back to having to go searching for who has deployed something on the experimental network to test with, which is almost guaranteed to be painful

What do we think about deploying Ghostnet and then Ghostnet++ ?

1 Like
11

I propose the name Zombienet for the new protocol permanent testnet

12

@nicolasochem have any bakers setup full fledged bakers on ghostnet, just like mainnet, paying out delegation rewards? Had a quick look at TzKT and it doesn’t look like any baker has provided the info to be picked up as a “Public Baker” on Ghostnet yet: All Public Bakers in Tezos Ghostnet (i’m assuming thats the issue and its not an issue on TzKT’s side)

This is kind of a small, yet perfect example of the issues with lack of data/resources on the protocol-specific testnets, nobody bothers to do this kind of stuff. If i’m making a frontend that suggests recommended bakers … that frontend just doesn’t work on testnet.

Would be great to see a push for this, so that teams will be able to setup wallets with funds, that will grow over time letting them fund all their own testing. Maybe bakers should send their rewards / fees to whatever address controls the faucet bots, to keep them funded too

13

Someone has to write code to make any protocol update work. The issue with Ghostnet++/Zombienet/Permanent Mondaynet is that you have to spend time writing migrations between random states of the code for no good reason. It would be a bad allocation of engineering resources and a great way to lose your sanity :slight_smile:

The primary audience of ghostnet are NFT/DeFi app devs who want a stable testnet against which to test changes to their own platforms. They typically rely on libraries and indexers to interact with the chain. The teams behind these libraries and indexers are the first responders to any proto breakage, it’s their job to smooth it out for the upper layers.

Indexer/library teams can tolerate regular chain restarts. Right now on the teztnets platform we have a rudimentary way of injecting contracts at genesis. You can then run checks on this contracts each time the chain reinitializes. Actually Taquito authors at ECAD are doing just that and caught a regression.

We want to improve this and get more testing on Mondaynet. Indexer support would come a long way as well.

(I don’t understand why the upgrade needs to be centralized either but let’s discuss one thing at a time.)

@rafoo An in-band signal is better than out-of-band. We just updated ghostnet from ithaca to jakarta with an user-activated update and it was messy: some bakers inevitably fail to apply the update and the network partitions, with nodes banning each other, and logs start screaming. Plus, if they don’t do it on time, the storage is non-recoverable and you have to resync the node.

And we don’t even have tooling for such an out-of-band signal: we have tezos-node config update --network <url> but it has to be run again for every new update. Hard-coding the upgrade block in the software is also slow and lossy.

A protocol level signal has a guarantee of delivery and simplifies node operations a great deal: just keep the software up-to-date. And in the latest iteration of the code, we can even cancel an upgrade before it kicks in and submit another one if needed. So from an operational point of view, it’s perfect.

There is a FAQ in the tzip that addresses this point in more detail.

have any bakers setup full fledged bakers on ghostnet, just like mainnet, paying out delegation rewards? Had a quick look at TzKT and it doesn’t look like any baker has provided the info to be picked up as a “Public Baker” on Ghostnet yet

I like the idea. This seems like a good opportunity to revive the baker registry.

1 Like