Use db_backup + management commands to implement data dump/load/seed

[KevinMind]

Relates to: mozilla/addons#15020
Unblocks: #22663

Description

• migrate data backup management to use django manmagement commands and db-backup under the hood
• add test coverage for the behaviour
• add synchronization of the storage volume with the database snapshot as those two data sources are tightly coupled.

Context

In order to include data initialization into the make up flow, we need a better way to dump/load/seed data. This PR moves the mechanism for exporting/importing data to use the db-backup lib orchestrated by management commands, includes test coverage for these behaviours and removes the concept from our make files.

db-backup is a light wrapper around django database connectors to use mysql commands for dumping/loading. This side steps issues we have with using native django dump/load data commands. Before this patch, we had a db connector and could implement this logic ourselves.. but looking at this lib it is well maintained, has a lot of activity and is very transparent looking through the source code. I cannot imagine implementing something much different then what they already have.

This unblocks the initialization part because the command for initializing data in make up will use seeding (which uses dumping) and loading in order to make sure the database is in the "correct" state based on the the arguments passed by the developer.

This can be extracted as a separate PR to make review and testing easier as this part is relatively low impact.

From this PR we also get synchronization of the ./storage directory with the database backup which squashes a whole range of bugs related to mismatches from db file references and the underlying file system.

Testing

1. run a data dump.

make data_dump

Expect a directory with the timestamp name is added to ./backups. It should include a file db.sql and a storage file storage.tar which is an exact replica of the ./storage directory.

Optionally pass an argument ARGS=--name foo to put the data in a specific named directory.

2. run data dump force

Assuming you dumped data to a directory with the name "foo" if you try to dump data there again, it will fail. You can make it pass by adding ARGS="--name foo --force" which will clean the foo directory before dumping.

3. Seed data

Now run the command

make initialize

This should reset your database to the initial data.

Note. any data you previously had in your DB will now be gone. Expect no add-ons for the local_admin user.

5. Load data

You should be able to load data from a directory by passing the directory name. Use the name from step 1.

Note, this should restore the data you had originally

make data_load ARGS=--name foo

If you don't pass a name or pass an invalid name the command will raise.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[eviljeff]

iirc @diox had concerns that django's dumpdata wasn't going to work for us, but I don't know the details.

[KevinMind]

iirc @diox had concerns that django's dumpdata wasn't going to work for us, but I don't know the details.

I ran into these issues myself. I think it could be possible to mitigate but I found a library that does mysql level dumping/loading and also handles files.. so pretty much exactly what we are aiming for..

It's a dev dep and only installed on dev so I think it's relatively low risk to add. Also seems well maintained and I read through the code fairly easily.

[diox]

Using django-dbbackup seems the way to go, yeah. dumpdata and loaddata are a pain to work with when you have complex relationships (especially the kind that we have).

[eviljeff]

@KevinMind I'm attempting to follow the testing instructions but I don't think they're up to date? make data_seed doesn't exist any longer, and there's no mention of the changes to make initialize

[KevinMind]

@KevinMind I'm attempting to follow the testing instructions but I don't think they're up to date? make data_seed doesn't exist any longer, and there's no mention of the changes to make initialize

@eviljeff updated. You should run make initialize this will in turn run the seed management command

[eviljeff]

r+wc - for me, just need to apply the suggestion to reintroduce create_db (or a similar change)