WIP: Add backup & restore commands

[lunny]

This is a new method to backup database into a fixture-like yml files and then restoring to other databases become more easier.

How to backup a database into a fixture directory? Run

./gitea backup --dir-path=./backup

This will generate yml files per Gitea's database tables. One table one yaml file.

$ ls
access.yml                    commit_status.yml             language_stat.yml             project_issue.yml             star.yml
access_token.yml              commit_status_index.yml       lfs_lock.yml                  protected_branch.yml          stopwatch.yml
action.yml                    dbfs_data.yml                 lfs_meta_object.yml           protected_tag.yml             system_setting.yml
...

How to restore a directory with these fixtures files into a database? Run

./gitea restore --dir-path=./backup

ToDo List:

• Keep columns sequence in yml file when backup
• Add tests for same database
• Add tests for different database

[silverwind]

Sounds interesting, this could enable easy migration between databases.

[a1012112796]

@lunny I had writen a fixture dumpper also, would you like to have a look :) #26594

[lng2020]

Testfixture itself support the backup/restore command. Maybe we can reuse those commands?

# load
testfixtures -d postgres -c "postgres://user:password@localhost/database" -D testdata/fixtures
# dump
testfixtures --dump -d postgres -c "postgres://user:password@localhost/database" -D testdata/fixtures

ref(https://github.com/go-testfixtures/testfixtures/blob/master/cmd/testfixtures/testfixtures.go)

[wxiaoguang]

Is it the right way to use yaml like this?

Suppose the user have a large database (I know some users have a >1G action table), would the large yaml files be unmarshaled into memory and cause problems?

[lunny]

@wxiaoguang I don't think we read all records from database to memory. Take a look at this, only 100 records every time. For restoring, testfixture will read the whole yaml content into memory when generating SQL files. That maybe consumes much memories.

[wxiaoguang]

For restoring, testfixture will read the whole yaml content into memory when generating SQL files. That maybe consumes much memories.

I meant this

[lunny]

Close as I'm not sure if it's a good idea for a big instance. Maybe improve the current SQL based implementation is better.