Merge pull request facebookresearch#20 from asonnino/master

More benchmark results
asonnino · Sep 28, 2021 · 3b46475 · 3b46475
2 parents 0719c64 + bcc27ae
commit 3b46475
Show file tree

Hide file tree

Showing 269 changed files with 11,021 additions and 4 deletions.
diff --git a/benchmark/data/README.md → benchmark/data/latest/README.md b/benchmark/data/README.md → benchmark/data/latest/README.md
@@ -1,6 +1,6 @@
 # Experimental Data
 
-This folder contains some raw data and plots obtained running a geo-replicated benchmark on AWS as explained in the [benchmark's readme file](https://github.com/facebookresearch/narwhal/tree/master/benchmark#readme). The results are taken running the code tagged as [v0.1.1](https://github.com/facebookresearch/narwhal/tree/v0.1.1).
+This folder contains some raw data and plots obtained running a geo-replicated benchmark on AWS as explained in the [benchmark's readme file](https://github.com/asonnino/narwhal/tree/master/benchmark#readme). The results are taken running the code tagged as [v0.2.0](https://github.com/asonnino/narwhal/tree/v0.2.0).
 
 ### Filename format
 The filename format of raw data is the following:
@@ -18,7 +18,7 @@ where:
 For instance, a file called `bench-0-50-1-True-100000-512.txt` indicates it contains results of a benchmark run with 50 nodes, 1 worker per node collocated on the same machine as the primary, 100K input rate, a transaction size of 512B, and 0 faulty nodes.
 
 ### Experimental step
-The content of our [settings.json](https://github.com/facebookresearch/narwhal/blob/master/benchmark/settings.json) file looks as follows:
+The content of our [settings.json](https://github.com/asonnino/narwhal/blob/master/benchmark/settings.json) file looks as follows:
 ```json
 {
     "key": {
@@ -28,7 +28,7 @@ The content of our [settings.json](https://github.com/facebookresearch/narwhal/b
     "port": 5000,
     "repo": {
         "name": "narwhal",
-        "url": "https://github.com/facebookresearch/narwhal",
+        "url": "https://github.com/asonnino/narwhal",
         "branch": "master"
     },
     "instances": {
@@ -37,7 +37,7 @@ The content of our [settings.json](https://github.com/facebookresearch/narwhal/b
     }
 }
 ```
-We set the following `node_params` in our [fabfile](https://github.com/facebookresearch/narwhal/blob/master/benchmark/fabfile.py):
+We set the following `node_params` in our [fabfile](https://github.com/asonnino/narwhal/blob/master/benchmark/fabfile.py):
 ```python
 node_params = {
     'header_size': 1_000,  # bytes

diff --git a/...data/tusk/bench-0-10-1-True-10000-512.txt → ...test/tusk/bench-0-10-1-True-10000-512.txt b/...data/tusk/bench-0-10-1-True-10000-512.txt → ...test/tusk/bench-0-10-1-True-10000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-100000-512.txt → ...est/tusk/bench-0-10-1-True-100000-512.txt b/...ata/tusk/bench-0-10-1-True-100000-512.txt → ...est/tusk/bench-0-10-1-True-100000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-110000-512.txt → ...est/tusk/bench-0-10-1-True-110000-512.txt b/...ata/tusk/bench-0-10-1-True-110000-512.txt → ...est/tusk/bench-0-10-1-True-110000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-120000-512.txt → ...est/tusk/bench-0-10-1-True-120000-512.txt b/...ata/tusk/bench-0-10-1-True-120000-512.txt → ...est/tusk/bench-0-10-1-True-120000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-130000-512.txt → ...est/tusk/bench-0-10-1-True-130000-512.txt b/...ata/tusk/bench-0-10-1-True-130000-512.txt → ...est/tusk/bench-0-10-1-True-130000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-140000-512.txt → ...est/tusk/bench-0-10-1-True-140000-512.txt b/...ata/tusk/bench-0-10-1-True-140000-512.txt → ...est/tusk/bench-0-10-1-True-140000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-150000-512.txt → ...est/tusk/bench-0-10-1-True-150000-512.txt b/...ata/tusk/bench-0-10-1-True-150000-512.txt → ...est/tusk/bench-0-10-1-True-150000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-160000-512.txt → ...est/tusk/bench-0-10-1-True-160000-512.txt b/...ata/tusk/bench-0-10-1-True-160000-512.txt → ...est/tusk/bench-0-10-1-True-160000-512.txt
diff --git a/...ata/tusk/bench-0-10-1-True-170000-512.txt → ...est/tusk/bench-0-10-1-True-170000-512.txt b/...ata/tusk/bench-0-10-1-True-170000-512.txt → ...est/tusk/bench-0-10-1-True-170000-512.txt
diff --git a/...data/tusk/bench-0-20-1-True-10000-512.txt → ...test/tusk/bench-0-20-1-True-10000-512.txt b/...data/tusk/bench-0-20-1-True-10000-512.txt → ...test/tusk/bench-0-20-1-True-10000-512.txt
diff --git a/...ata/tusk/bench-0-20-1-True-140000-512.txt → ...est/tusk/bench-0-20-1-True-140000-512.txt b/...ata/tusk/bench-0-20-1-True-140000-512.txt → ...est/tusk/bench-0-20-1-True-140000-512.txt
diff --git a/...ata/tusk/bench-0-20-1-True-150000-512.txt → ...est/tusk/bench-0-20-1-True-150000-512.txt b/...ata/tusk/bench-0-20-1-True-150000-512.txt → ...est/tusk/bench-0-20-1-True-150000-512.txt
diff --git a/...ata/tusk/bench-0-20-1-True-170000-512.txt → ...est/tusk/bench-0-20-1-True-170000-512.txt b/...ata/tusk/bench-0-20-1-True-170000-512.txt → ...est/tusk/bench-0-20-1-True-170000-512.txt
diff --git a/...ata/tusk/bench-0-20-1-True-180000-512.txt → ...est/tusk/bench-0-20-1-True-180000-512.txt b/...ata/tusk/bench-0-20-1-True-180000-512.txt → ...est/tusk/bench-0-20-1-True-180000-512.txt
diff --git a/...ata/tusk/bench-0-20-1-True-200000-512.txt → ...est/tusk/bench-0-20-1-True-200000-512.txt b/...ata/tusk/bench-0-20-1-True-200000-512.txt → ...est/tusk/bench-0-20-1-True-200000-512.txt
diff --git a/...data/tusk/bench-0-4-1-False-10000-512.txt → ...test/tusk/bench-0-4-1-False-10000-512.txt b/...data/tusk/bench-0-4-1-False-10000-512.txt → ...test/tusk/bench-0-4-1-False-10000-512.txt
diff --git a/...data/tusk/bench-0-4-1-False-40000-512.txt → ...test/tusk/bench-0-4-1-False-40000-512.txt b/...data/tusk/bench-0-4-1-False-40000-512.txt → ...test/tusk/bench-0-4-1-False-40000-512.txt
diff --git a/...data/tusk/bench-0-4-1-False-50000-512.txt → ...test/tusk/bench-0-4-1-False-50000-512.txt b/...data/tusk/bench-0-4-1-False-50000-512.txt → ...test/tusk/bench-0-4-1-False-50000-512.txt
diff --git a/...data/tusk/bench-0-4-1-False-60000-512.txt → ...test/tusk/bench-0-4-1-False-60000-512.txt b/...data/tusk/bench-0-4-1-False-60000-512.txt → ...test/tusk/bench-0-4-1-False-60000-512.txt
diff --git a/...data/tusk/bench-0-4-1-False-65000-512.txt → ...test/tusk/bench-0-4-1-False-65000-512.txt b/...data/tusk/bench-0-4-1-False-65000-512.txt → ...test/tusk/bench-0-4-1-False-65000-512.txt
diff --git a/...ata/tusk/bench-0-4-10-False-10000-512.txt → ...est/tusk/bench-0-4-10-False-10000-512.txt b/...ata/tusk/bench-0-4-10-False-10000-512.txt → ...est/tusk/bench-0-4-10-False-10000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-450000-512.txt → ...st/tusk/bench-0-4-10-False-450000-512.txt b/...ta/tusk/bench-0-4-10-False-450000-512.txt → ...st/tusk/bench-0-4-10-False-450000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-500000-512.txt → ...st/tusk/bench-0-4-10-False-500000-512.txt b/...ta/tusk/bench-0-4-10-False-500000-512.txt → ...st/tusk/bench-0-4-10-False-500000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-540000-512.txt → ...st/tusk/bench-0-4-10-False-540000-512.txt b/...ta/tusk/bench-0-4-10-False-540000-512.txt → ...st/tusk/bench-0-4-10-False-540000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-550000-512.txt → ...st/tusk/bench-0-4-10-False-550000-512.txt b/...ta/tusk/bench-0-4-10-False-550000-512.txt → ...st/tusk/bench-0-4-10-False-550000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-600000-512.txt → ...st/tusk/bench-0-4-10-False-600000-512.txt b/...ta/tusk/bench-0-4-10-False-600000-512.txt → ...st/tusk/bench-0-4-10-False-600000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-650000-512.txt → ...st/tusk/bench-0-4-10-False-650000-512.txt b/...ta/tusk/bench-0-4-10-False-650000-512.txt → ...st/tusk/bench-0-4-10-False-650000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-660000-512.txt → ...st/tusk/bench-0-4-10-False-660000-512.txt b/...ta/tusk/bench-0-4-10-False-660000-512.txt → ...st/tusk/bench-0-4-10-False-660000-512.txt
diff --git a/...ta/tusk/bench-0-4-10-False-670000-512.txt → ...st/tusk/bench-0-4-10-False-670000-512.txt b/...ta/tusk/bench-0-4-10-False-670000-512.txt → ...st/tusk/bench-0-4-10-False-670000-512.txt
diff --git a/...data/tusk/bench-0-4-4-False-10000-512.txt → ...test/tusk/bench-0-4-4-False-10000-512.txt b/...data/tusk/bench-0-4-4-False-10000-512.txt → ...test/tusk/bench-0-4-4-False-10000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-100000-512.txt → ...est/tusk/bench-0-4-4-False-100000-512.txt b/...ata/tusk/bench-0-4-4-False-100000-512.txt → ...est/tusk/bench-0-4-4-False-100000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-150000-512.txt → ...est/tusk/bench-0-4-4-False-150000-512.txt b/...ata/tusk/bench-0-4-4-False-150000-512.txt → ...est/tusk/bench-0-4-4-False-150000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-200000-512.txt → ...est/tusk/bench-0-4-4-False-200000-512.txt b/...ata/tusk/bench-0-4-4-False-200000-512.txt → ...est/tusk/bench-0-4-4-False-200000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-220000-512.txt → ...est/tusk/bench-0-4-4-False-220000-512.txt b/...ata/tusk/bench-0-4-4-False-220000-512.txt → ...est/tusk/bench-0-4-4-False-220000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-230000-512.txt → ...est/tusk/bench-0-4-4-False-230000-512.txt b/...ata/tusk/bench-0-4-4-False-230000-512.txt → ...est/tusk/bench-0-4-4-False-230000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-240000-512.txt → ...est/tusk/bench-0-4-4-False-240000-512.txt b/...ata/tusk/bench-0-4-4-False-240000-512.txt → ...est/tusk/bench-0-4-4-False-240000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-250000-512.txt → ...est/tusk/bench-0-4-4-False-250000-512.txt b/...ata/tusk/bench-0-4-4-False-250000-512.txt → ...est/tusk/bench-0-4-4-False-250000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-260000-512.txt → ...est/tusk/bench-0-4-4-False-260000-512.txt b/...ata/tusk/bench-0-4-4-False-260000-512.txt → ...est/tusk/bench-0-4-4-False-260000-512.txt
diff --git a/...ata/tusk/bench-0-4-4-False-270000-512.txt → ...est/tusk/bench-0-4-4-False-270000-512.txt b/...ata/tusk/bench-0-4-4-False-270000-512.txt → ...est/tusk/bench-0-4-4-False-270000-512.txt
diff --git a/...data/tusk/bench-0-4-7-False-10000-512.txt → ...test/tusk/bench-0-4-7-False-10000-512.txt b/...data/tusk/bench-0-4-7-False-10000-512.txt → ...test/tusk/bench-0-4-7-False-10000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-300000-512.txt → ...est/tusk/bench-0-4-7-False-300000-512.txt b/...ata/tusk/bench-0-4-7-False-300000-512.txt → ...est/tusk/bench-0-4-7-False-300000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-350000-512.txt → ...est/tusk/bench-0-4-7-False-350000-512.txt b/...ata/tusk/bench-0-4-7-False-350000-512.txt → ...est/tusk/bench-0-4-7-False-350000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-390000-512.txt → ...est/tusk/bench-0-4-7-False-390000-512.txt b/...ata/tusk/bench-0-4-7-False-390000-512.txt → ...est/tusk/bench-0-4-7-False-390000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-400000-512.txt → ...est/tusk/bench-0-4-7-False-400000-512.txt b/...ata/tusk/bench-0-4-7-False-400000-512.txt → ...est/tusk/bench-0-4-7-False-400000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-420000-512.txt → ...est/tusk/bench-0-4-7-False-420000-512.txt b/...ata/tusk/bench-0-4-7-False-420000-512.txt → ...est/tusk/bench-0-4-7-False-420000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-430000-512.txt → ...est/tusk/bench-0-4-7-False-430000-512.txt b/...ata/tusk/bench-0-4-7-False-430000-512.txt → ...est/tusk/bench-0-4-7-False-430000-512.txt
diff --git a/...ata/tusk/bench-0-4-7-False-440000-512.txt → ...est/tusk/bench-0-4-7-False-440000-512.txt b/...ata/tusk/bench-0-4-7-False-440000-512.txt → ...est/tusk/bench-0-4-7-False-440000-512.txt
diff --git a/...ata/tusk/bench-0-50-1-True-100000-512.txt → ...est/tusk/bench-0-50-1-True-100000-512.txt b/...ata/tusk/bench-0-50-1-True-100000-512.txt → ...est/tusk/bench-0-50-1-True-100000-512.txt
diff --git a/...ata/tusk/bench-0-50-1-True-150000-512.txt → ...est/tusk/bench-0-50-1-True-150000-512.txt b/...ata/tusk/bench-0-50-1-True-150000-512.txt → ...est/tusk/bench-0-50-1-True-150000-512.txt
diff --git a/...ata/tusk/bench-0-50-1-True-180000-512.txt → ...est/tusk/bench-0-50-1-True-180000-512.txt b/...ata/tusk/bench-0-50-1-True-180000-512.txt → ...est/tusk/bench-0-50-1-True-180000-512.txt
diff --git a/...ata/tusk/bench-0-50-1-True-190000-512.txt → ...est/tusk/bench-0-50-1-True-190000-512.txt b/...ata/tusk/bench-0-50-1-True-190000-512.txt → ...est/tusk/bench-0-50-1-True-190000-512.txt
diff --git a/...ata/tusk/bench-0-50-1-True-200000-512.txt → ...est/tusk/bench-0-50-1-True-200000-512.txt b/...ata/tusk/bench-0-50-1-True-200000-512.txt → ...est/tusk/bench-0-50-1-True-200000-512.txt
diff --git a/...data/tusk/bench-0-50-1-True-30000-512.txt → ...test/tusk/bench-0-50-1-True-30000-512.txt b/...data/tusk/bench-0-50-1-True-30000-512.txt → ...test/tusk/bench-0-50-1-True-30000-512.txt
diff --git a/...data/tusk/bench-0-50-1-True-50000-512.txt → ...test/tusk/bench-0-50-1-True-50000-512.txt b/...data/tusk/bench-0-50-1-True-50000-512.txt → ...test/tusk/bench-0-50-1-True-50000-512.txt
diff --git a/...data/tusk/bench-0-50-1-True-70000-512.txt → ...test/tusk/bench-0-50-1-True-70000-512.txt b/...data/tusk/bench-0-50-1-True-70000-512.txt → ...test/tusk/bench-0-50-1-True-70000-512.txt
diff --git a/...data/tusk/bench-1-10-1-True-10000-512.txt → ...test/tusk/bench-1-10-1-True-10000-512.txt b/...data/tusk/bench-1-10-1-True-10000-512.txt → ...test/tusk/bench-1-10-1-True-10000-512.txt
diff --git a/...ata/tusk/bench-1-10-1-True-100000-512.txt → ...est/tusk/bench-1-10-1-True-100000-512.txt b/...ata/tusk/bench-1-10-1-True-100000-512.txt → ...est/tusk/bench-1-10-1-True-100000-512.txt
diff --git a/...ata/tusk/bench-1-10-1-True-120000-512.txt → ...est/tusk/bench-1-10-1-True-120000-512.txt b/...ata/tusk/bench-1-10-1-True-120000-512.txt → ...est/tusk/bench-1-10-1-True-120000-512.txt
diff --git a/...ata/tusk/bench-1-10-1-True-130000-512.txt → ...est/tusk/bench-1-10-1-True-130000-512.txt b/...ata/tusk/bench-1-10-1-True-130000-512.txt → ...est/tusk/bench-1-10-1-True-130000-512.txt
diff --git a/...ata/tusk/bench-1-10-1-True-140000-512.txt → ...est/tusk/bench-1-10-1-True-140000-512.txt b/...ata/tusk/bench-1-10-1-True-140000-512.txt → ...est/tusk/bench-1-10-1-True-140000-512.txt
diff --git a/...ata/tusk/bench-1-10-1-True-150000-512.txt → ...est/tusk/bench-1-10-1-True-150000-512.txt b/...ata/tusk/bench-1-10-1-True-150000-512.txt → ...est/tusk/bench-1-10-1-True-150000-512.txt
diff --git a/...data/tusk/bench-1-10-1-True-50000-512.txt → ...test/tusk/bench-1-10-1-True-50000-512.txt b/...data/tusk/bench-1-10-1-True-50000-512.txt → ...test/tusk/bench-1-10-1-True-50000-512.txt
diff --git a/...data/tusk/bench-3-10-1-True-10000-512.txt → ...test/tusk/bench-3-10-1-True-10000-512.txt b/...data/tusk/bench-3-10-1-True-10000-512.txt → ...test/tusk/bench-3-10-1-True-10000-512.txt
diff --git a/...ata/tusk/bench-3-10-1-True-100000-512.txt → ...est/tusk/bench-3-10-1-True-100000-512.txt b/...ata/tusk/bench-3-10-1-True-100000-512.txt → ...est/tusk/bench-3-10-1-True-100000-512.txt
diff --git a/...ata/tusk/bench-3-10-1-True-110000-512.txt → ...est/tusk/bench-3-10-1-True-110000-512.txt b/...ata/tusk/bench-3-10-1-True-110000-512.txt → ...est/tusk/bench-3-10-1-True-110000-512.txt
diff --git a/...ata/tusk/bench-3-10-1-True-120000-512.txt → ...est/tusk/bench-3-10-1-True-120000-512.txt b/...ata/tusk/bench-3-10-1-True-120000-512.txt → ...est/tusk/bench-3-10-1-True-120000-512.txt
diff --git a/...data/tusk/bench-3-10-1-True-50000-512.txt → ...test/tusk/bench-3-10-1-True-50000-512.txt b/...data/tusk/bench-3-10-1-True-50000-512.txt → ...test/tusk/bench-3-10-1-True-50000-512.txt
diff --git a/benchmark/data/paper-data/README.md b/benchmark/data/paper-data/README.md
@@ -0,0 +1,52 @@
+# Experimental Data
+
+This folder contains the raw data and plots used in the evaluation section of the paper [Narwhal and Tusk: A DAG-based Mempool and Efficient BFT Consensus](https://arxiv.org/pdf/2105.11827.pdf). The data are obtained running a geo-replicated benchmark on AWS as explained in the [benchmark's readme file](https://github.com/asonnino/narwhal/tree/master/benchmark#readme). The results are taken running the code tagged as [v0.2.0](https://github.com/asonnino/narwhal/tree/v0.2.0).
+
+### Filename format
+The filename format of raw data is the following:
+```
+bench-FAULTS-NODES-WORKERS-COLLOCATE-INPUT_RATE-TX_SIZE.txt
+```
+where:
+- `FAULTS`: The number of faulty (dead) nodes.
+- `NODES`: The number of nodes in the testbed.
+- `WORKERS`: The number of workers per node.
+- `COLLOCATE`: Whether the primary and its worker are collocated on the same machine.
+- `INPUT_RATE`: The total rate at which clients submit transactions to the system.
+- `TX_SIZE`: The size of each transactions (in bytes).
+
+For instance, a file called `bench-0-50-1-True-100000-512.txt` indicates it contains results of a benchmark run with 50 nodes, 1 worker per node collocated on the same machine as the primary, 100K input rate, a transaction size of 512B, and 0 faulty nodes.
+
+### Experimental step
+The content of our [settings.json](https://github.com/asonnino/narwhal/blob/master/benchmark/settings.json) file looks as follows:
+```json
+{
+    "key": {
+        "name": "aws",
+        "path": "/absolute/key/path"
+    },
+    "port": 5000,
+    "repo": {
+        "name": "narwhal",
+        "url": "https://github.com/facebookresearch/narwhal",
+        "branch": "master"
+    },
+    "instances": {
+        "type": "m5d.8xlarge",
+        "regions": ["us-east-1", "eu-north-1", "ap-southeast-2", "us-west-1", "ap-northeast-1"]
+    }
+}
+```
+We set the following `node_params` in our [fabfile](https://github.com/asonnino/narwhal/blob/master/benchmark/fabfile.py):
+```python
+node_params = {
+    'header_size': 1_000,  # bytes
+    'max_header_delay': 200,  # ms
+    'gc_depth': 50,  # rounds
+    'sync_retry_delay': 10_000,  # ms
+    'sync_retry_nodes': 3,  # number of nodes
+    'batch_size': 500_000,  # bytes
+    'max_batch_delay': 200  # ms
+}
+```
+
diff --git a/benchmark/data/paper-data/baseline-hs/bench-0-10-1000-512.txt b/benchmark/data/paper-data/baseline-hs/bench-0-10-1000-512.txt
@@ -0,0 +1,50 @@
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 1,000 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 901 tx/s
+ Consensus BPS: 461,166 B/s
+ Consensus latency: 513 ms
+
+ End-to-end TPS: 900 tx/s
+ End-to-end BPS: 460,904 B/s
+ End-to-end latency: 15,434 ms
+-----------------------------------------
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 1,000 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 901 tx/s
+ Consensus BPS: 461,117 B/s
+ Consensus latency: 509 ms
+
+ End-to-end TPS: 900 tx/s
+ End-to-end BPS: 460,907 B/s
+ End-to-end latency: 15,410 ms
+-----------------------------------------
diff --git a/benchmark/data/paper-data/baseline-hs/bench-0-10-500-512.txt b/benchmark/data/paper-data/baseline-hs/bench-0-10-500-512.txt
@@ -0,0 +1,50 @@
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 500 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 10,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 399 tx/s
+ Consensus BPS: 204,365 B/s
+ Consensus latency: 572 ms
+
+ End-to-end TPS: 399 tx/s
+ End-to-end BPS: 204,205 B/s
+ End-to-end latency: 898 ms
+-----------------------------------------
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 500 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 10,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 399 tx/s
+ Consensus BPS: 204,269 B/s
+ Consensus latency: 537 ms
+
+ End-to-end TPS: 399 tx/s
+ End-to-end BPS: 204,205 B/s
+ End-to-end latency: 857 ms
+-----------------------------------------
diff --git a/benchmark/data/paper-data/baseline-hs/bench-0-10-750-512.txt b/benchmark/data/paper-data/baseline-hs/bench-0-10-750-512.txt
@@ -0,0 +1,50 @@
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 750 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 10,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 598 tx/s
+ Consensus BPS: 306,408 B/s
+ Consensus latency: 540 ms
+
+ End-to-end TPS: 598 tx/s
+ End-to-end BPS: 306,214 B/s
+ End-to-end latency: 828 ms
+-----------------------------------------
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 750 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 10,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 599 tx/s
+ Consensus BPS: 306,507 B/s
+ Consensus latency: 522 ms
+
+ End-to-end TPS: 598 tx/s
+ End-to-end BPS: 306,286 B/s
+ End-to-end latency: 807 ms
+-----------------------------------------
diff --git a/benchmark/data/paper-data/baseline-hs/bench-0-10-800-512.txt b/benchmark/data/paper-data/baseline-hs/bench-0-10-800-512.txt
@@ -0,0 +1,50 @@
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 800 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 799 tx/s
+ Consensus BPS: 409,078 B/s
+ Consensus latency: 505 ms
+
+ End-to-end TPS: 798 tx/s
+ End-to-end BPS: 408,464 B/s
+ End-to-end latency: 775 ms
+-----------------------------------------
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 800 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 798 tx/s
+ Consensus BPS: 408,586 B/s
+ Consensus latency: 499 ms
+
+ End-to-end TPS: 797 tx/s
+ End-to-end BPS: 408,313 B/s
+ End-to-end latency: 769 ms
+-----------------------------------------
diff --git a/benchmark/data/paper-data/baseline-hs/bench-0-10-900-512.txt b/benchmark/data/paper-data/baseline-hs/bench-0-10-900-512.txt
@@ -0,0 +1,50 @@
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 900 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 798 tx/s
+ Consensus BPS: 408,618 B/s
+ Consensus latency: 513 ms
+
+ End-to-end TPS: 798 tx/s
+ End-to-end BPS: 408,327 B/s
+ End-to-end latency: 816 ms
+-----------------------------------------
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 10 nodes
+ Input rate: 900 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 798 tx/s
+ Consensus BPS: 408,496 B/s
+ Consensus latency: 508 ms
+
+ End-to-end TPS: 797 tx/s
+ End-to-end BPS: 408,236 B/s
+ End-to-end latency: 810 ms
+-----------------------------------------
diff --git a/benchmark/data/paper-data/baseline-hs/bench-0-20-1000-512.txt b/benchmark/data/paper-data/baseline-hs/bench-0-20-1000-512.txt
@@ -0,0 +1,50 @@
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 20 nodes
+ Input rate: 1,000 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 798 tx/s
+ Consensus BPS: 408,441 B/s
+ Consensus latency: 597 ms
+
+ End-to-end TPS: 797 tx/s
+ End-to-end BPS: 408,308 B/s
+ End-to-end latency: 918 ms
+-----------------------------------------
+
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ + CONFIG:
+ Committee size: 20 nodes
+ Input rate: 1,000 tx/s
+ Transaction size: 512 B
+ Faults: 0 nodes
+ Execution time: 301 s
+
+ Consensus max payloads size: 50,000 B
+ Consensus min block delay: 10 ms
+ Mempool max payloads size: 600 B
+ Mempool min block delay: 10 ms
+
+ + RESULTS:
+ Consensus TPS: 798 tx/s
+ Consensus BPS: 408,543 B/s
+ Consensus latency: 559 ms
+
+ End-to-end TPS: 797 tx/s
+ End-to-end BPS: 408,275 B/s
+ End-to-end latency: 874 ms
+-----------------------------------------