|
@@ -1,340 +0,0 @@
|
|
-# Migrating to the new Pixelfed Docker setup
|
|
|
|
-
|
|
|
|
-There is [*a lot* of changes](https://github.com/pixelfed/pixelfed/pull/4844) in how Pixelfed Docker/Docker Compose images work - it's a complete rewrite - with a bunch of breaking changes.
|
|
|
|
-
|
|
|
|
-## No more anonymous Docker Compose volumes
|
|
|
|
-
|
|
|
|
-The old `docker-compose.yml` configuration file [declared four anonymous volumes](https://github.com/pixelfed/pixelfed/blob/b1ff44ca2f75c088a11576fb03b5bad2fbed4d5c/docker-compose.yml#L72-L76) for storing Pixelfed related data within.
|
|
|
|
-
|
|
|
|
-These are no longer used, instead favoring a [Docker bind volume](https://docs.docker.com/storage/bind-mounts/) approach where content are stored directly on the server disk, outside
|
|
|
|
-of a Docker volume.
|
|
|
|
-
|
|
|
|
-The consequence of this change is that *all* data stored in the - now unsupported - Docker volumes will no longer be accessible by Pixelfed.
|
|
|
|
-
|
|
|
|
-* The `db-data` volume *definitely* contain important data - it's your database after all!
|
|
|
|
-* The `app-storage` volume *definitely* contain important data - it's files uploaded to - or seen by - your server!
|
|
|
|
-* The `redis-data` volume *might* contain important data (depending on your configuration)
|
|
|
|
-* The `app-bootstrap` volume does not contain any important data - all of it will be generated automatically in the new setup on startup. We will *not* be migrating this!
|
|
|
|
-
|
|
|
|
-### Migrating off anonymous Docker Compose volumes
|
|
|
|
-
|
|
|
|
-#### Caveats and warnings
|
|
|
|
-
|
|
|
|
-> [!IMPORTANT]
|
|
|
|
-> This is a best-effort guide to help migrate off the old system, the operation is potentially rather complicated (and risky), so please do be careful!
|
|
|
|
-
|
|
|
|
-> [!CAUTION]
|
|
|
|
-> ***PLEASE MAKE SURE TO BACKUP YOUR SERVER AND DATA BEFORE ATTEMPTING A MIGRATION***
|
|
|
|
->
|
|
|
|
-> **YOUR INSTANCE WILL BE *DOWN* WHILE DOING THE MIGRATION, PLEASE PLAN ACCORDINGLY, DEPENDING ON DATA SIZE IT COULD TAKE ANYWHERE FROM 5 *MINUTES* TO 5 *HOURS***
|
|
|
|
-
|
|
|
|
-> [!WARNING]
|
|
|
|
-> **It's important to note that this is a *copy* operation - so disk usage will (temporarily) double while you migrate**
|
|
|
|
-> We provide a "migration container" for your convenience that can access both the new and old volumes, allowing you to copy the data into the setup.
|
|
|
|
-
|
|
|
|
-#### Step 0) Backup, rollout, and rollback plan
|
|
|
|
-
|
|
|
|
-1. Make sure to backup your server (ideally *after* step 1 below has completed, but *before* is better than not at all!)
|
|
|
|
-1. Capture the current Git version / Pixelfed release you are on (e.g. `git --no-pager log -1` outputs the commit reference as the 2nd word in first line)
|
|
|
|
-1. Backup your `.env` file (we will do this in step 3 as well)
|
|
|
|
-1. Backup your `docker-compose.yml` file (`cp docker-compose.yml docker-compose.yml.old`)
|
|
|
|
-1. Read through the *entire* document before starting
|
|
|
|
-
|
|
|
|
-#### Step 1) Migrate your ".env" file
|
|
|
|
-
|
|
|
|
-The new `.env` file for Docker is a bit different from the old one (many new settings!) so the easiest is to grab the new `.env.docker` file and modify it from scratch again.
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ cp .env .env.old
|
|
|
|
-$ wget -O .env.new https://raw.githubusercontent.com/pixelfed/pixelfed/dev/.env.docker
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-Then open your old `.env.old` configuration file, and for each of the key/value pairs within it, find and update the key in the new `.env.new` configuration file.
|
|
|
|
-
|
|
|
|
-Don't worry though, the file might *look* different (and significantly larger) but it behaves *exactly* the way the old file did, it just has way more options!
|
|
|
|
-
|
|
|
|
-> [!TIP]
|
|
|
|
-> Don't worry if a key is missing in `.env.new`, you can add those key/value pairs back to the new file - ideally in the `Other configuration` section near the end of the file - but anywhere *should* be fine.
|
|
|
|
-
|
|
|
|
-This is a great time to review your settings and familiarize you with all the new settings.
|
|
|
|
-
|
|
|
|
-> [!NOTE]
|
|
|
|
-> In *particular* the following sections
|
|
|
|
->
|
|
|
|
-> * `PHP configuration` section (near the end of the file) where
|
|
|
|
-> * The `DOCKER_APP_PHP_VERSION` settings controls your PHP version
|
|
|
|
-> * The `PHP_MEMORY_LIMIT` settings controls your PHP memory limit
|
|
|
|
-> * `Docker Specific configuration` section (near the end of the file) where
|
|
|
|
-> * The `DOCKER_ALL_HOST_DATA_ROOT_PATH` setting dictate where the new migrated data will live.
|
|
|
|
-> * The `DOCKER_APP_RUN_ONE_TIME_SETUP_TASKS` controls if the `One time setup tasks` should run or not. We do *not* want this, since your Pixelfed instance already is set up!
|
|
|
|
-> * [Frequently Asked Question / FAQ](faq.md)
|
|
|
|
-> * [How do I use my own Proxy server?](faq.md#how-do-i-use-my-own-proxy-server)
|
|
|
|
-> * [How do I use my own SSL certificate?](faq.md#how-do-i-use-my-own-ssl-certificate)
|
|
|
|
-
|
|
|
|
-#### Step 2) Stop all running containers
|
|
|
|
-
|
|
|
|
-> [!CAUTION]
|
|
|
|
-> This will take your Pixelfed instance offline
|
|
|
|
-
|
|
|
|
-Stop *all* running containers (web, worker, redis, db)
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose down
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Step 3) Pull down the new source code
|
|
|
|
-
|
|
|
|
-Update your project to the latest release of Pixelfed by running
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ git pull origin $release
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-> [!NOTE]
|
|
|
|
-> The `$release` can be any valid git reference like `dev`, `staging` or a [tagged release](https://github.com/pixelfed/pixelfed/releases) such as `v0.12.0`.
|
|
|
|
-
|
|
|
|
-#### Step 4) Run the migration container
|
|
|
|
-
|
|
|
|
-You can access the Docker container with both old and new volumes by running the following command
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose -f docker-compose.migrate.yml run migrate bash
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-This will put you in the `/migrate` directory within the container, containing 8 directories like shown here
|
|
|
|
-
|
|
|
|
-```plain
|
|
|
|
-|-- app-storage
|
|
|
|
-| |-- new
|
|
|
|
-| `-- old
|
|
|
|
-|-- db-data
|
|
|
|
-| |-- new
|
|
|
|
-| `-- old
|
|
|
|
-`-- redis-data
|
|
|
|
- |-- new
|
|
|
|
- `-- old
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Step 5) Check the folders
|
|
|
|
-
|
|
|
|
-##### Old folders
|
|
|
|
-
|
|
|
|
-The following commands should all return *SOME* files and data - if they do not - then there might be an issue with the anonymous volume binding.
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ ls app-storage/old
|
|
|
|
-$ ls db-data/old
|
|
|
|
-$ ls redis-data/old
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-##### New folders
|
|
|
|
-
|
|
|
|
-The following commands should all return *NO* files and data - if they contain data - you need to either delete it (backup first!) or skip that migration step.
|
|
|
|
-
|
|
|
|
-If you haven't run `docker compose up` since you updated your project in step (2) - they should be empty and good to go.
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ ls app-storage/new
|
|
|
|
-$ ls db-data/new
|
|
|
|
-$ ls redis-data/new
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Step 6) Copy the data
|
|
|
|
-
|
|
|
|
-> [!WARNING]
|
|
|
|
-> This is where we potentially will double your disk usage (temporarily)
|
|
|
|
-
|
|
|
|
-Now we will copy the data from the old volumes, to the new ones.
|
|
|
|
-
|
|
|
|
-The migration container has [`rsync`](https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories) installed - which is perfect for that kind of work!
|
|
|
|
-
|
|
|
|
-**NOTE** It's important that the "source" (first path in the `rsync` command) has a trailing `/` - otherwise the directory layout will turn out wrong!
|
|
|
|
-
|
|
|
|
-**NOTE** Depending on your server, these commands might take some time to finish, each command should provide a progress bar with rough time estimation.
|
|
|
|
-
|
|
|
|
-**NOTE** `rsync` should preserve ownership, permissions, and symlinks correctly for you as well for all the files copied.
|
|
|
|
-
|
|
|
|
-Lets copy the data by running the following commands:
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ rsync -avP app-storage/old/ app-storage/new
|
|
|
|
-$ rsync -avP db-data/old/ db-data/new
|
|
|
|
-$ rsync -avP redis-data/old/ redis-data/new
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Step 7) Sanity checking
|
|
|
|
-
|
|
|
|
-Lets make sure everything copied over successfully!
|
|
|
|
-
|
|
|
|
-Each *new* directory should contain *something* like (but not always exactly) the following - **NO** directory should have a single folder called `old`, if they do, the `rsync` commands above didn't work correctly - and you need to move the content of the `old` folder into the "root" of the `new` folder like shown a bit in the following sections.
|
|
|
|
-
|
|
|
|
-The **redis-data/new** directory might also contain a `server.pid`
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ ls redis-data/new
|
|
|
|
-appendonlydir
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-The **app-storage/new** directory should look *something* like this
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ ls app-storage/new
|
|
|
|
-app debugbar docker framework logs oauth-private.key oauth-public.key purify
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-The **db-data/new** directory should look *something* like this. There might be a lot of files, or very few files, but there *must* be a `mysql`, `performance_schema`, and `${DB_DATABASE}` (e.g. `pixelfed_prod` directory)
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ ls db-data/new
|
|
|
|
-aria_log_control ddl_recovery-backup.log ib_buffer_pool ib_logfile0 ibdata1 mariadb_upgrade_info multi-master.info mysql performance_schema pixelfed_prod sys undo001 undo002 undo003
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-If everything looks good, type `exit` to leave exit the migration container
|
|
|
|
-
|
|
|
|
-#### Step 8) Starting up the your Pixelfed server again
|
|
|
|
-
|
|
|
|
-With all an updated Pixelfed (step 2), updated `.env` file (step 3), migrated data (step 4, 5, 6 and 7) we're ready to start things back up again.
|
|
|
|
-
|
|
|
|
-But before we start your Pixelfed server back up again, lets put the new `.env` file we made in step 1 in its right place.
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ cp .env.new .env
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-##### The Database
|
|
|
|
-
|
|
|
|
-First thing we want to try is to start up the database by running the following command and checking the logs
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose up -d db
|
|
|
|
-$ docker compose logs --tail 250 --follow db
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-if there are no errors and the server isn't crashing, great! If you have an easy way of connecting to the Database via a GUI or CLI client, do that as well and verify the database and tables are all there.
|
|
|
|
-
|
|
|
|
-##### Redis
|
|
|
|
-
|
|
|
|
-Next thing we want to try is to start up the Redis server by running the following command and checking the logs
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose up -d redis
|
|
|
|
-$ docker compose logs --tail 250 --follow redis
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-if there are no errors and the server isn't crashing, great!
|
|
|
|
-
|
|
|
|
-##### Worker
|
|
|
|
-
|
|
|
|
-Next thing we want to try is to start up the Worker server by running the following command and checking the logs
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose up -d worker
|
|
|
|
-$ docker compose logs --tail 250 --follow worker
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-The container should output a *lot* of logs from the [docker-entrypoint system](customizing.md#running-commands-on-container-start), but *eventually* you should see these messages
|
|
|
|
-
|
|
|
|
-* `Configuration complete; ready for start up`
|
|
|
|
-* `Horizon started successfully.`
|
|
|
|
-
|
|
|
|
-If you see one or both of those messages, great, the worker seems to be running.
|
|
|
|
-
|
|
|
|
-If the worker is crash looping, inspect the logs and try to resolve the issues.
|
|
|
|
-
|
|
|
|
-You can consider the following additional steps:
|
|
|
|
-
|
|
|
|
-* Enabling `DOCKER_APP_ENTRYPOINT_DEBUG` which will show even more log output to help understand whats going on
|
|
|
|
-* Enabling `DOCKER_APP_ENSURE_OWNERSHIP_PATHS` against the path(s) that might have permission issues
|
|
|
|
-* Fixing permission issues directly on the host since your data should all be in the `${DOCKER_ALL_HOST_DATA_ROOT_PATH}` folder (`./docker-compose-state/data` by default)
|
|
|
|
-
|
|
|
|
-##### Web
|
|
|
|
-
|
|
|
|
-The final service, `web`, which will bring your site back online! What a journey it has been.
|
|
|
|
-
|
|
|
|
-Lets get to it, run these commands to start the `web` service and inspect the logs.
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose up -d web
|
|
|
|
-$ docker compose logs --tail 250 --follow web
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-The output should be pretty much identical to that of the `worker`, so please see that section for debugging tips if the container is crash looping.
|
|
|
|
-
|
|
|
|
-If the `web` service came online without issues, start the rest of the (optional) services, such as the `proxy`, if enabled, by running
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker compose up -d
|
|
|
|
-$ docker compose logs --tail 250 --follow
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-If you changed anything in the `.env` file while debugging, some containers might restart now, thats perfectly fine.
|
|
|
|
-
|
|
|
|
-#### Step 9) Verify
|
|
|
|
-
|
|
|
|
-With all services online, it's time to go to your browser and check everything is working
|
|
|
|
-
|
|
|
|
-1. Upload and post a picture
|
|
|
|
-1. Comment on a post
|
|
|
|
-1. Like a post
|
|
|
|
-1. Check Horizon (`https://${APP_DOMAIN}/horizon`) for any errors
|
|
|
|
-1. Check the Docker compose logs via `docker compose logs --follow`
|
|
|
|
-
|
|
|
|
-If everything looks fine, yay, you made it to the end! Lets do some cleanup
|
|
|
|
-
|
|
|
|
-#### Step 10) Final steps + cleanup
|
|
|
|
-
|
|
|
|
-With everything working, please take a new snapshot/backup of your server *before* we do any cleanup. A post-migration snapshot is incredibly useful, since it contains both the old and new configuration + data, making any recovery much easier in a rollback scenario later.
|
|
|
|
-
|
|
|
|
-Now, with all the data in the new folders, you can delete the old Docker Container volumes (if you want, completely optional)
|
|
|
|
-
|
|
|
|
-List all volumes, and give them a look:
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker volume ls
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-The volumes we want to delete *ends* with the volume name (`db-data`, `app-storage`, `redis-data`, and `app-bootstrap`.) but has some prefix in front of them.
|
|
|
|
-
|
|
|
|
-Once you have found the volumes in in the list, delete each of them by running:
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ docker volume rm $volume_name_in_column_two_of_the_output
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-You can also delete the `docker-compose.yml.old` and `.env.old` file since they are no longer needed
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ rm docker-compose.yml.old
|
|
|
|
-$ rm .env.old
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-### Rollback
|
|
|
|
-
|
|
|
|
-Oh no, something went wrong? No worries, we you got backups and a quick way back!
|
|
|
|
-
|
|
|
|
-#### Move `docker-compose.yml` back
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ cp docker-compose.yml docker-compose.yml.new
|
|
|
|
-$ cp docker-compose.yml.old docker-compose.yml
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Move `.env` file back
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ cp env.old .env
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Go back to old source code version
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-$ git checkout $commit_id_from_step_0
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Start things back up
|
|
|
|
-
|
|
|
|
-```bash
|
|
|
|
-docker compose up -d
|
|
|
|
-```
|
|
|
|
-
|
|
|
|
-#### Verify it worked
|
|
|