Enterprise application software solutions within the financial services industry are required to meet strict standards and regulations. Tezos has many features (on-chain governance, formal verifiable smart contracts, proof-of-stake) which would greatly benefit the largest financial services companies however the industry needs to be satisfied that the Tezos development cycle meets or exceeds its own high standards.
A mature testing framework with test driven development is required for Tezos to gain acceptance within the financial services industry and this includes the following key architecture:
- Regression Testing Pack;
- Continuous Regression Testing; and
Having the resources to develop enterprise quality testing comparable to the financial services industry requires a significant investment but this is required for Tezos to gain widespread enterprise adoption. Investment in an independent testing team is an essential component of the testing strategy, with a team size around 10% of the core development teams. The purpose of this paper is to identify current gaps, suggest possible solutions and how these could be funded in the long-term. Many of the ideas would initially need to be bootstrapped by the Tezos Foundation but the ultimate goal would be to fund it organically from within the Tezos system.
Our background is in the financial services industry. We began our crypto journey in Ethereum as its smart contract platform interested us greatly due to the immediate benefits we could see in this technology. We had initially taken a keen interest in Bitcoin but is was the release of the Ethereum mainchain which convinced us to step outside the traditional financial services network and work in crypto on a full time basis and initially we were involved in helping financial clients navigate their way through blockchain and smart contract technology. Like many, our first big learning curve was to help secure the Ethereum network by filling a warehouse with GPU mining rigs.
It was this interest in mining and working out how to disperse all the heat from the rigs that made us realise that proof-of-stake consensus was the future. We decided to participate in the Tezos ICO to become a validator and gain experience as we believed that most chains would move towards the Tezos model. We have been very enthusiastic with the Tezos proof-of-stake system and have shared our experiences with our our local Tezos communities.
In this paper we concentrate on the on-chain governance process with a particular focus on testing to see what we can learn from the previous chain upgrades. We believe our experience in upgrading large complex capital market software systems, where billions of dollars flow through everyday, provides a useful insight when applying a similar process to upgrading a live chain. Many of the constraints are the same insofar as you have a 24 hour system where any downtime or errors can have large financial and reputational impacts. We have used test-driven agile based development processes and we believe Tezos development would benefit from adopting some of these techniques.
Our focus is on test-driven software and outlining a process for supporting the Tezos on-chain governance process. We are are not directly addressing the technical issues from the previous release, more how the process can be improved to an enterprise standard which can support developers to make the brave software changes required to advance Tezos. From an outside perspective, there appears to be an enormous amount of pressure on the development teams to keep new functionality rolling out; maybe it’s time to pause and consolidate the current position.
We mentioned in our reddit post that we were not fans of Tezos’ on-chain governance, not because of the process itself but because it’s extremely difficult to test. The current process leaves too much code untested and — in a community setting — only runs for the first time when it goes live. Until this changes we are likely to see many of the same problems recurring on each release. This may be acceptable in the current environment while the chain is not being heavily used but it won’t be acceptable when real world financial contracts are running. These mistakes have been costly in terms of reputational damage to our community and even now there is still plenty of confusion around the latest release on social media channels. Tezos users are early adopters and are willing to stick with Tezos but this is unlikely to be the case with the next wave of adopters.
For this reason we want to consider a disciplined and rigorous engineering approach to testing in order to deliver the enterprise level quality code required with a live chain. The process should also be iterative and we should constantly be looking for improvements so the process and tools will continue to be refined over time. This will help drive Tezos towards the goal of enterprise level quality code backed by comprehensive test coverage.
Introduction: Test-Driven Development — Everything starts with a test
One of Tezos’s unique features is on-chain governance however testing the entire workflow has proved difficult to date. The introduction of a testing framework would reduce the number of bugs that hit the mainchain on an upgrade.
Our paper starts with a worked example to give a flavour of how a change in the process will significantly reduce the number of bugs when deploying the software. This paper has been split into 5 testing improvement sections which will help steer constructive conversations within the community. This is not an exhaustive list but a place to start the discussion on how to improve the current process and the tools needed to support the process flow. The conclusion includes a number of concise actions points that were discussed in the paper.
Regression Testing Pack — Writing tests to test the code is a vital tool; developers have to check code works as expected. Tests can be grouped into two categories
- Unit tests — built by developers as part of the development process to test that small pieces of functionality are working as expected. Unit tests must all run green before any code is deployed; and
- Regression tests — built by an independent testing team to test all the major flow through the software.
A pack of regression tests contains all the unit tests and regression tests required to be run against the different code streams. These can be run by anyone with the appropriate set-up.
Continuous Regression Testing — Any code commit should automatically execute all the tests and results should be available to view by anyone in the community.
Testnets — On-chain governance adds extra complexity to the testing process and in order to get complete coverage a number of well defined testnets need to be available to the community. Clarity is key so users, application developers and bakers know exactly where to go to test new features.
Upgradable Functionality — Not all new functionality has to be switched on simultaneously. Consider delivering the upgrade first with most of the new functionality switched off.
On-chain Governance by Adoption — The software is not activated until 90% of validation nodes (weighted by votes) have signalled that they have upgraded. Doing this removes the need to have the final vote as the new software will only be adopted when the community is ready.
Test Driven Development — Worked Example
This is a worked example of how the test-driven software approach will eliminate the type of bugs that were seen in the last release. Before delving into the testing detail, a worked example will help contextualise the type of problem that is being addressed.
We acknowledge and thank Nomadic Labs and Cryptium Labs for producing an insightful and transparent lessons learnt document detailing the technical aspects of the Babylon upgrade. Transparency of the issues is crucial for process improvement and the continuous agile development cycle. Strengthening and improving the test-driven process is the particular focus of this paper.
In the development of large scale financial software systems developers can find themselves juggling many balls concurrently and it’s crucial that the process supports them. Even with a successful release, developers can find they are in the firing line on a minor issue but not receiving any credit for their successes. The key to solving this problem lies in the process and not with particular developers to remember to do one step. If you rely on the memory of a developer, or indeed any human, then your process will fail. One particular failure was addressed in the lessons learnt document: “ The failure of the mempool was caused by a bug which was known and fixed on the master branch of the Tezos code base but was incorrectly ported to the mainnet branch ” The solution suggested was as follows: “ Once again this kind of bug can be avoided using a simpler release process which we are currently implementing and will be in place before the next release ” Unfortunately, making the release process simpler doesn’t completely remove the chance of such mistakes recurring but in development if you leave a gap, however small, then someone else will inevitably fall into it again.
In this particular situation a change in process will fix this type of problem for good and this is where test-driven development comes to the rescue. Test-driven development means the first thing you do is build tests for the code you are going to write. So in this example, the first step is to construct a test case which will fail until this particular issue is solved. This test case is committed to each stream (i.e. dev, uat, prod) and the testing team never deploys a release until all test cases pass. The developer no longer needs to remember if they have merged their fixes into all the branches as it’s there for everyone to see. Not only does test-driven development provide a reminder of what hasn’t yet been fixed, it won’t break again because otherwise this test will fail. The test ensures that merge issues are eliminated and provides better quality code as a result of better test cases coverage whilst at the same time removing some of the pressure from the development team.
A bug should always be viewed as an opportunity to improve the quality of the code and should not be wasted by just fixing it. Improved code also prevents the embarrassment of the fix being merged out at a later release because it was never applied to the dev stream. Tools such as a bug tracking system should be built to reflect the development process flow; for example, a bug must have test cases with fixes to all the code streams completed before the test team signs it off.
It is this type of process change that will take the pressure off developers as it’s transparent to the whole team what needs to be achieved. If the process relies on developers remembering to do things then eventually it will come back to haunt you. If a developer decides to remove a test-case because it’s the easiest solution then the testing team process would pick that up and address it appropriately.
In our opinion, test-driven development should be used for the entire development process. Everything starts with a test; if it’s hard to write a test for the functionality then change the design so a test can be written. It’s amazing how many designs get altered when you have to write test-cases upfront. No code should ever be developed without first writing a test case upfront, no exceptions.