add a make target to retrieve the blockstore and prometheus data store #11
Conversation
| @@ -45,6 +45,14 @@ runload: | |||
| --ip-list `grep ' name=' ./ansible/hosts | cut -d' ' -f1 | paste -s -d, -` \ | |||
| --seed-delta $(shell echo $$RANDOM) | |||
|
|
|||
| retrieve-data: | |||
There was a problem hiding this comment.
Question:
IIUC, we are retrieving the experiment data into our laptop.
Then, we would have to transfer it to the DO "space", where we decided we'd keep it.
Looks suboptimal to me, but, on the other hand, as we have two different DO accounts (Informal and IG), I don't see a more efficient way of doing this.
So, just a reflection
There was a problem hiding this comment.
At the moment, I'm not sure the easiest way to get the data to the DO storage. IMO, we should store it in a digital ocean block store, which they call spaces. Spaces allow upload of data, but I'm not currently seeing an easy way to upload to them using ansible. You need credentials to upload to them as well, so the credentials need to be present on the machine where the data is generated. Perhaps just ssh, scp the credentials onto the machine and run doctl ...?
There was a problem hiding this comment.
Yeah, that's what I was thinking, on top of the fact that we have two DO accounts, and probably we want to have all data in both for convenience, so I don't see how we would do that other than manually
There was a problem hiding this comment.
Created an issue in this repo to track
#14
Makefile
Outdated
| retrieve-data: | ||
| @DIR=`date "+%Y-%m-%d-%H_%M_%S%N"`; \ | ||
| mkdir -p "./experiments/$${DIR}"; \ | ||
| cd ansible && ansible-playbook -i hosts -u root retrieve-blockstore.yaml --limit `ansible -i hosts --list-hosts validators | tail -1| sed 's/ //g'` -e "dir=../experiments/$${DIR}/blockstore.db.zip"; \ |
There was a problem hiding this comment.
I'm testing the value of the --limit argument, and it outputs the last of the validators. Is this what you want to do?
If yes, I'd retrieve everything, in case the chosen node was late... AFAIU, disk space in "DO spaces" is cheap
There was a problem hiding this comment.
Updated to fetch all. It now places each in a directory under the experiments dir, of the format ./experiments/<date>/seed01/blockstore.db.zip.
places the
blockstore.dband the prometheus data into a directory in a newexperimentsdirectory. Each new fetch generates a new directory in theexperimentsdirectory of the current time, e.g.experiments/2022-09-07-13_21_59595212278.