Editing data in a Docker volume to fix a container that shuts down immediately after exit

Today I was faced with a problem whereby I had edited configuration in a local development Postgres Docker container in such a way that caused it to fail on start, exiting immediately.

Context

The configuration change was in postgresql.conf to test a behaviour around connection starvation.

max_connections = 100

max_connections = 3

The container then failed to start since this value was incompatible with the reserved super user number of connections.

Normally you can used docker exec to get a shell into a container and then modify data on the attached volume, but if the container won't stay up, this is a problem. My containers were running under Docker Desktop with Docker compose.

Resolution

Keeping the Container Running

To resolve I researched ways of keeping a container running on failure. The method that worked for me was to append tail -f /dev/null to the end of a docker run command (credit).

Attaching the Volume

We can find the name of our volume using docker volume ls.

We then map the volume to the correct path inside the container as we would in our Docker Compose file (name_of_volume_postgres:/var/lib/postgresql/data).

docker run -d -v name_of_volume_postgres:/var/lib/postgresql/data postgres:13.7-bullseye tail -f /dev/null 

Editing the Data

We can now use docker exec -it <container ID> /bin/bash to get a shell into the running container. There is no text editor included with the stock Postgresql image, so I used sed to make the replacement.

The postgresql.conf file can be found at /var/lib/postgresql/data/postgresql.conf this can be checked using the SHOW config_file when Postgres is running.

# sed -i 's/max_connections = 3/max_connections = 5/g' postgresql.conf

This fixed the issue and now when I start the container under Docker compose, it runs without immmediately exiting.