March 19, 2018 . 10 min read
In this series of tests, we want to measure the performance and stability of the validator nodes forming the consensus engine when the transactions are sent in bursts from multiple clients. The performance of the consensus engine is measured as the number of transactions processed per second. This is also called as consensus efficiency.
In our previous tests, we used 4 validator nodes to measure the consensus efficiency. This is the absolute minimum number of nodes required to tolerate Byzantine failure of 1 node. Such a setup offers minimum decentralization. In this test, we also want to increase the decentralization and see its effect on consensus efficiency.
Version 1 of DyPoS is partially built on top of Tendermint. The core consensus engine will be replaced eventually by BlockFin, Storecoin’s leaderless, fork-tolerant, high-throughput consensus protocol. Version 1 of DyPoS is used to model various transaction load patterns to better understand the behavior of the validator nodes. These models help identify deficiencies in traditional approaches to consensus, so they can be better addressed in BlockFin. In this series, we examine the behavior of validator nodes when bursts of transactions are sent from multiple clients.
Figure 1 shows Storecoin application architecture, which follows Tendermint's ABCI Protocol.
Since the purpose of the tests is to measure the consensus efficiency and stability of the validator nodes, the application logic is replaced with Tendermint’s tm-bench benchmarking tool. This tool is customized to generate bursts of transactions of different sizes.
With its Dynamic, Fork-tolerant, and Auditable Proof of Stake-based consensus protocol (DyPoS), Storecoin will secure free transactions, high throughput, and a decentralized governance system with built-in checks and balances inspired by the U.S. Constitution. Also inspired by the supply and demand principles of Uber Surge Pricing (blockchain economics) and Power of Attorney model (blockchain scaling), Storecoin will secure crypto-powered incentives and payments inside of apps.
In this series of tests, we set up 8-node and 21-node test benches and measure the consensus efficiency in each setup. The 21-node setup provides greater decentralization compared to all the previous tests performed. In typical blockchain deployments, the greater the decentralization, the worse the consensus efficiency will be. This is inevitable because large numbers of nodes result in increased peer-to-peer communication, thus affecting the consensus efficiency negatively. We want to verify that the resulting drop in consensus efficiency is acceptable and the efficiency doesn’t degrade to a level where it forces using fewer validator nodes.
The following tests are performed in this series.
4 clients, 8 validator nodes, fixed size transactions in multiple burst sizes
4 clients generate transactions of fixed size and send them to all the 8 validator nodes. The transactions are of size 500, 1,000, 5,000, and 10,000 bytes in bursts of size 500, 1,000, 2,000, 5,000, and 10,000 transactions per burst. Each burst lasts for 5 seconds. The test setup, results, and observations for all the tests with 8 validator nodes are captured in this report. Figure 2 visualizes the test setup.
Table 1 below summarizes the test results.
The best consensus efficiency of 80,000 transactions per second is obtained in test run 1.E. At higher transaction and burst sizes, the resulting transaction volume overwhelms the validator nodes, resulting in test failures. The test is characterized as Failed if the validator nodes are unable to keep up with incoming transactions and stop accepting them. Note that the validator nodes continue to run the consensus rounds for the transactions that are already accepted and at no time did they stop producing new blocks.
Test runs 3.D and 5.C are interesting. While 1.E produced the best consensus efficiency, it used smaller transaction size of 100 bytes. 3.D produced consensus efficiency of 38,316 transactions per second, but with transactions of size 1,000 bytes and burst size of 5,000. So, while the consensus efficiency is much lower than in test run 1.E, the run 3.D demonstrates processing large transactions in a large burst size.
The test run 5.C shows the effect of very large transaction sizes. The consensus efficiency dropped to 512 transactions per second. This demonstrates that the DyPoS consensus engine favors small to midsize transactions in small to large burst sizes. The typical transaction size in Storecoin is less than 500 bytes, so this behavior doesn’t affect the use cases that Storecoin is designed for.
8 clients, 8 validator nodes, fixed size transactions in multiple burst sizes
This test is similar to the test above, except 8 clients are used to generate transactions instead of 4. This test is intended to evenly saturate all 8 validator nodes, so they are not idling. The results are summarized in table 2 below.
Test run 6.E produced best consensus efficiency of 159,790 transactions per second. The consensus efficiency is nearly doubled in this case. This is because 8 clients are used to generate the transactions, so overall twice the total transactions are produced in this test. Test runs 8.D and 10.B are similar to test runs 3.D and 5.C in the previous test.
4 and 8 clients, 8 validator nodes, random size transactions in multiple burst sizes
In this test, random transaction sizes are used instead of fixed sizes used in the previous tests. The size distribution is done as shown in table 3 below. This test models real world settings where majority of transactions are of smaller sizes with a small percentage of medium and large size transactions.
The test results are described in table 4 below.
With 4 clients sending random sized transactions, the best consensus efficiency of 7,987 transactions per second is observed in test run 13.C. Notice the sharp drop in consensus efficiency. The drop is explained with the same reason as for test runs 3.D and 8.D with fixed size transactions. Since we are using transactions of random sizes, overall size of the burst is larger than that with the fixed size transactions. This results in lower consensus efficiency.
With 8 clients sending random sized transactions, the best consensus efficiency of 16,067 transactions per second is observed in test run 12.C. Notice that the consensus efficiency is nearly doubled because the validator nodes are being used to their capacity.
Random size transactions result in a decreased consensus efficiency because the overall transaction volume is much higher than the fixed size transactions.
2 clients, 21 validators 50 bursts, 500 txs/burst, random transaction sizes, 30 second pause between bursts
In this series of tests with 21 validator nodes, our goal is twofold:
The test setup, results, and observations for all the tests with 21 validator nodes are captured in this report. Figure 3 visualizes the test setup.
In this test, 2 clients generate random sized transactions with 500 transactions in a burst. Each burst lasts for 5 seconds as before. Each client sends 50 bursts with a 30 second pause between them. The consensus efficiency is measured per burst as before. The resulting list for 50 bursts is as follows.
The consensus efficiency ranges from a low of 4,750 to a high of 7,759 transactions per second. We theorize two reasons for the wide range in the consensus efficiency numbers.
The second claim above on transaction accumulation must be verified because depending on the size of accumulation, the consensus efficiency can increase or decrease. This claim is verified in the next test.
2 clients, 21 validators 50 bursts, 500 txs/burst, random transaction sizes, 120 second pause between bursts
In this test, the pause between the bursts is increased to 120 seconds, so the validator nodes have sufficient time to cool off. This should minimize transaction accumulation between the bursts. The test results are listed here.
The consensus efficiency ranges from a low 2,348 transactions per second to a high 2,654 transactions per second.
In other words, the consensus efficiency dropped with a longer pause between the bursts. This is because the longer pauses result in under-utilizing the validator nodes. This observation brings up an important characteristics of Storecoin DyPoS consensus engine. If the validator nodes are under-utilized or overwhelmed with transactions, the consensus efficiency drops. If, on the other hand, they are utilized to their capacity, the consensus efficiency would be at its peak. This leads us to the next test.
8 clients, 21 validators 50 bursts, 50 txs/burst, random transaction sizes, 30 second pause between bursts
This test is designed to verify the claim that the consensus efficiency peaks when validator nodes are loaded just right. In this test 8 clients are used to generate transactions instead of 2 in the previous tests. The burst size is reduced to 50 transactions per burst instead of 500 transactions. This results in smaller bursts arriving concurrently from a larger number of clients. The validator nodes get a continuous supply of transactions without overwhelming them and at the same time ensuring better utilization of them. The test results are available here.
The consensus efficiency ranges from a low of 5,592 transactions per second to a high 10,282 transactions per second.
This is a significant improvement compared to the consensus efficiencies observed in previous tests with 2 clients. The improved efficiency is due to optimal utilization of the validator nodes. It is also important to note that the consensus engine handles concurrent bursts from multiple clients without affecting the consensus efficiency.
In this series of tests we learned about the behavior of validator nodes with increased decentralization and continuous bursts of transactions. In typical blockchain deployments, greater decentralization severely affects consensus efficiency, resulting in centralized deployments. The tests performed here verified that Storecoin DyPoS is not affected by centralization. While consensus efficiency with 21-node setup did drop from the 8-node setup where transactions were sent in a single burst, the drop in consensus efficiency is respectable because multiple bursts of transactions are processed by a larger number of validator nodes.
In the test with large bursts of transactions, consensus efficiency was observed ranging from a low of 4,750 to a high of 7.759 transactions per second. In the test with smaller bursts of transactions but with greater concurrency, consensus efficiency was observed ranging from a low of 5,592 to a high of 10,282 transactions per second.
We are also able to characterize the behavior of the validator nodes when they are overwhelmed. The 8-node setup helped us model the failure cases resulting in low consensus efficiency or nodes refusing to accept incoming transactions.
With initial testing of DyPoS complete, the behavior of validator nodes is now understood under a variety of load conditions. This behavior can be summarized as follows:
Nothing herein is intended to be an offer to sell or solicitation of offer to buy, Storecoin tokens or rights to receive Storecoin tokens in the future. In the event that Storecoin conducts an offering of Storecoin tokens (or rights to receive Storecoin tokens in the future), Storecoin will do so in compliance with all applicable laws which may include the Securities Act of 1933 and the rules and regulations promulgated thereunder, as well as applicable state and foreign law. Any offering for sale to US Persons in a regulated transaction will be pursuant to a registration statement qualified by the Securities and Exchange Commission, or an applicable exemption from the registration requirements.