From e96341a18aba2a6ea930cde97507750ff30a5a53 Mon Sep 17 00:00:00 2001 From: Alberto Sonnino Date: Fri, 22 Oct 2021 19:15:53 +0100 Subject: [PATCH 1/4] Nits --- README.md | 8 ++++---- benchmark/README.md | 18 +++++++++--------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index b26cd810..b08077a3 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Narwhal and Tusk -[![build status](https://img.shields.io/github/workflow/status/facebookresearch/narwhal/Rust/master?style=flat-square&logo=github)](https://github.com/facebookresearch/narwhal/actions) +[![build status](https://img.shields.io/github/workflow/status/asonnino/narwhal/Rust/master?style=flat-square&logo=github)](https://github.com/asonnino/narwhal/actions) [![rustc](https://img.shields.io/badge/rustc-1.51+-blue?style=flat-square&logo=rust)](https://www.rust-lang.org) [![license](https://img.shields.io/badge/license-Apache-blue.svg?style=flat-square)](LICENSE) @@ -10,7 +10,7 @@ This repo provides an implementation of [Narwhal and Tusk](https://arxiv.org/pdf The core protocols are written in Rust, but all benchmarking scripts are written in Python and run with [Fabric](http://www.fabfile.org/). To deploy and benchmark a testbed of 4 nodes on your local machine, clone the repo and install the python dependencies: ``` -$ git clone https://github.com/facebookresearch/narwhal.git +$ git clone https://github.com/asonnino/narwhal.git $ cd narwhal/benchmark $ pip install -r requirements.txt ``` @@ -52,9 +52,9 @@ This command may take a long time the first time you run it (compiling rust code ``` ## Next Steps -The next step is to read the paper [Narwhal and Tusk: A DAG-based Mempool and Efficient BFT Consensus](https://arxiv.org/pdf/2105.11827.pdf). It is then recommended to have a look at the README files of the [worker](https://github.com/facebookresearch/narwhal/tree/master/worker) and [primary](https://github.com/facebookresearch/narwhal/tree/master/primary) crates. An additional resource to better understand the Tusk consensus protocol is the paper [All You Need is DAG](https://arxiv.org/abs/2102.08325) as it describes a similar protocol. +The next step is to read the paper [Narwhal and Tusk: A DAG-based Mempool and Efficient BFT Consensus](https://arxiv.org/pdf/2105.11827.pdf). It is then recommended to have a look at the README files of the [worker](https://github.com/asonnino/narwhal/tree/master/worker) and [primary](https://github.com/asonnino/narwhal/tree/master/primary) crates. An additional resource to better understand the Tusk consensus protocol is the paper [All You Need is DAG](https://arxiv.org/abs/2102.08325) as it describes a similar protocol. -The README file of the [benchmark folder](https://github.com/facebookresearch/narwhal/tree/master/benchmark) explains how to benchmark the codebase and read benchmarks' results. It also provides a step-by-step tutorial to run benchmarks on [Amazon Web Services (AWS)](https://aws.amazon.com) accross multiple data centers (WAN). +The README file of the [benchmark folder](https://github.com/asonnino/narwhal/tree/master/benchmark) explains how to benchmark the codebase and read benchmarks' results. It also provides a step-by-step tutorial to run benchmarks on [Amazon Web Services (AWS)](https://aws.amazon.com) accross multiple data centers (WAN). ## License This software is licensed as [Apache 2.0](LICENSE). diff --git a/benchmark/README.md b/benchmark/README.md index 0b0e9b1a..75d73c96 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -5,7 +5,7 @@ This document explains how to benchmark the codebase and read benchmarks' result When running benchmarks, the codebase is automatically compiled with the feature flag `benchmark`. This enables the node to print some special log entries that are then read by the python scripts and used to compute performance. These special log entries are clearly indicated with comments in the code: make sure to not alter them (otherwise the benchmark scripts will fail to interpret the logs). ### Parametrize the benchmark -After cloning the repo and [installing all dependencies](https://github.com/facebookresearch/narwhal#quick-start), you can use [Fabric](http://www.fabfile.org/) to run benchmarks on your local machine. Locate the task called `local` in the file [fabfile.py](https://github.com/facebookresearch/narwhal/blob/master/benchmark/fabfile.py): +After cloning the repo and [installing all dependencies](https://github.com/asonnino/narwhal#quick-start), you can use [Fabric](http://www.fabfile.org/) to run benchmarks on your local machine. Locate the task called `local` in the file [fabfile.py](https://github.com/asonnino/narwhal/blob/master/benchmark/fabfile.py): ```python @task def local(ctx): @@ -104,7 +104,7 @@ $ ssh-keygen -f ~/.ssh/aws ``` ### Step 3. Configure the testbed -The file [settings.json](https://github.com/facebookresearch/narwhal/blob/master/benchmark/settings.json) (located in [narwhal/benchmarks](https://github.com/facebookresearch/narwhal/blob/master/benchmark)) contains all the configuration parameters of the testbed to deploy. Its content looks as follows: +The file [settings.json](https://github.com/asonnino/narwhal/blob/master/benchmark/settings.json) (located in [narwhal/benchmarks](https://github.com/asonnino/narwhal/blob/master/benchmark)) contains all the configuration parameters of the testbed to deploy. Its content looks as follows: ```json { "key": { @@ -114,7 +114,7 @@ The file [settings.json](https://github.com/facebookresearch/narwhal/blob/master "port": 5000, "repo": { "name": "narwhal", - "url": "https://github.com/facebookresearch/narwhal.git", + "url": "https://github.com/asonnino/narwhal.git", "branch": "master" }, "instances": { @@ -143,7 +143,7 @@ The third block (`repo`) contains the information regarding the repository's nam ```json "repo": { "name": "narwhal", - "url": "https://github.com/facebookresearch/narwhal.git", + "url": "https://github.com/asonnino/narwhal.git", "branch": "master" }, ``` @@ -159,12 +159,12 @@ The the last block (`instances`) specifies the [AWS instance type](https://aws.a The instance type selects the hardware on which to deploy the testbed. For example, `m5d.8xlarge` instances come with 32 vCPUs (16 physical cores), 128 GB of RAM, and guarantee 10 Gbps of bandwidth. The python scripts will configure each instance with 300 GB of SSD hard drive. The `regions` field specifies the data centers to use. If you require more nodes than data centers, the python scripts will distribute the nodes as equally as possible amongst the data centers. All machines run a fresh install of Ubuntu Server 20.04. ### Step 4. Create a testbed -The AWS instances are orchestrated with [Fabric](http://www.fabfile.org) from the file [fabfile.py](https://github.com/facebookresearch/narwhal/blob/master/benchmark/fabfile.pyy) (located in [narwhal/benchmarks](https://github.com/facebookresearch/narwhal/blob/master/benchmark)); you can list all possible commands as follows: +The AWS instances are orchestrated with [Fabric](http://www.fabfile.org) from the file [fabfile.py](https://github.com/asonnino/narwhal/blob/master/benchmark/fabfile.pyy) (located in [narwhal/benchmarks](https://github.com/asonnino/narwhal/blob/master/benchmark)); you can list all possible commands as follows: ``` $ cd narwhal/benchmark $ fab --list ``` -The command `fab create` creates new AWS instances; open [fabfile.py](https://github.com/facebookresearch/narwhal/blob/master/benchmark/fabfile.py) and locate the `create` task: +The command `fab create` creates new AWS instances; open [fabfile.py](https://github.com/asonnino/narwhal/blob/master/benchmark/fabfile.py) and locate the `create` task: ```python @task def create(ctx, nodes=2): @@ -189,13 +189,13 @@ This may take a long time as the command will first update all instances. The commands `fab stop` and `fab start` respectively stop and start the testbed without destroying it (it is good practice to stop the testbed when not in use as AWS can be quite expensive); and `fab destroy` terminates all instances and destroys the testbed. Note that, depending on the instance types, AWS instances may take up to several minutes to fully start or stop. The command `fab info` displays a nice summary of all available machines and information to manually connect to them (for debug). ### Step 5. Run a benchmark -After setting up the testbed, running a benchmark on AWS is similar to running it locally (see [Run Local Benchmarks](https://github.com/facebookresearch/narwhal/tree/master/benchmark#local-benchmarks)). Locate the task `remote` in [fabfile.py](https://github.com/facebookresearch/narwhal/blob/master/benchmark/fabfile.py): +After setting up the testbed, running a benchmark on AWS is similar to running it locally (see [Run Local Benchmarks](https://github.com/asonnino/narwhal/tree/master/benchmark#local-benchmarks)). Locate the task `remote` in [fabfile.py](https://github.com/asonnino/narwhal/blob/master/benchmark/fabfile.py): ```python @task def remote(ctx): ... ``` -The benchmark parameters are similar to [local benchmarks](https://github.com/facebookresearch/narwhal/tree/master/benchmark#local-benchmarks) but allow to specify the number of nodes and the input rate as arrays to automate multiple benchmarks with a single command. The parameter `runs` specifies the number of times to repeat each benchmark (to later compute the average and stdev of the results), and the parameter `collocate` specifies whether to collocate all the node's workers and the primary on the same machine. If `collocate` is set to `False`, the script will run one node per data center (AWS region), with its primary and each of its worker running on a dedicated instance. +The benchmark parameters are similar to [local benchmarks](https://github.com/asonnino/narwhal/tree/master/benchmark#local-benchmarks) but allow to specify the number of nodes and the input rate as arrays to automate multiple benchmarks with a single command. The parameter `runs` specifies the number of times to repeat each benchmark (to later compute the average and stdev of the results), and the parameter `collocate` specifies whether to collocate all the node's workers and the primary on the same machine. If `collocate` is set to `False`, the script will run one node per data center (AWS region), with its primary and each of its worker running on a dedicated instance. ```python bench_params = { 'nodes': [10, 20, 30], @@ -214,7 +214,7 @@ Once you specified both `bench_params` and `node_params` as desired, run: ``` $ fab remote ``` -This command first updates all machines with the latest commit of the GitHub repo and branch specified in your file [settings.json](https://github.com/facebookresearch/narwhal/blob/master/benchmark/settings.json) (step 3); this ensures that benchmarks are always run with the latest version of the code. It then generates and uploads the configuration files to each machine, runs the benchmarks with the specified parameters, and downloads the logs. It finally parses the logs and prints the results into a folder called `results` (which is automatically created if it doesn't already exists). You can run `fab remote` multiple times without fearing to override previous results, the command either appends new results to a file containing existing ones or prints them in separate files. If anything goes wrong during a benchmark, you can always stop it by running `fab kill`. +This command first updates all machines with the latest commit of the GitHub repo and branch specified in your file [settings.json](https://github.com/asonnino/narwhal/blob/master/benchmark/settings.json) (step 3); this ensures that benchmarks are always run with the latest version of the code. It then generates and uploads the configuration files to each machine, runs the benchmarks with the specified parameters, and downloads the logs. It finally parses the logs and prints the results into a folder called `results` (which is automatically created if it doesn't already exists). You can run `fab remote` multiple times without fearing to override previous results, the command either appends new results to a file containing existing ones or prints them in separate files. If anything goes wrong during a benchmark, you can always stop it by running `fab kill`. ### Step 6. Plot the results Once you have enough results, you can aggregate and plot them: From 589f95d205594f5f34ee8afe0a1c38f371060de5 Mon Sep 17 00:00:00 2001 From: Alberto Sonnino Date: Fri, 22 Oct 2021 19:18:07 +0100 Subject: [PATCH 2/4] Nits --- primary/README.md | 2 +- worker/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/primary/README.md b/primary/README.md index 6358759b..bab088f8 100644 --- a/primary/README.md +++ b/primary/README.md @@ -2,5 +2,5 @@ The diagram below illustrates the primary's architecture and could be useful to keep in mind while going through the code.

- +

diff --git a/worker/README.md b/worker/README.md index 970ac919..3f488861 100644 --- a/worker/README.md +++ b/worker/README.md @@ -2,5 +2,5 @@ The diagram below illustrates the worker's architecture and could be useful to keep in mind while going through the code.

- +

From 1d93ee0f25d940aa2fb359d1121448b2fe68b289 Mon Sep 17 00:00:00 2001 From: Alberto Sonnino Date: Fri, 22 Oct 2021 19:28:07 +0100 Subject: [PATCH 3/4] Nits --- benchmark/data/paper-data/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmark/data/paper-data/README.md b/benchmark/data/paper-data/README.md index 6f2f0f5f..7c6bf7ca 100644 --- a/benchmark/data/paper-data/README.md +++ b/benchmark/data/paper-data/README.md @@ -28,7 +28,7 @@ The content of our [settings.json](https://github.com/asonnino/narwhal/blob/mast "port": 5000, "repo": { "name": "narwhal", - "url": "https://github.com/facebookresearch/narwhal", + "url": "https://github.com/asonnino/narwhal", "branch": "master" }, "instances": { From 88c757e60a026ef6093a4b15e7661bdc96ab37c2 Mon Sep 17 00:00:00 2001 From: Alberto Sonnino Date: Fri, 22 Oct 2021 19:30:24 +0100 Subject: [PATCH 4/4] Update settings.json --- benchmark/settings.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/benchmark/settings.json b/benchmark/settings.json index d13629f2..e1471a2c 100644 --- a/benchmark/settings.json +++ b/benchmark/settings.json @@ -1,12 +1,12 @@ { "key": { - "name": "aws-fb", - "path": "/Users/asonnino/.ssh/aws-fb.pem" + "name": "aws", + "path": "/Users/asonnino/.ssh/aws.pem" }, "port": 5000, "repo": { "name": "narwhal", - "url": "https://github.com/facebookresearch/narwhal", + "url": "https://github.com/asonnino/narwhal", "branch": "master" }, "instances": {