This issue is very strange. I still did not comprehend what happened exactly, but my brain tells me that somehow the docker-compose project changed it's automatic instance name generation from using hyphens to dashes.
I do not want to go and replicate the issue right now on the live server, but I was able to track down the most relevant error message from the search history:
ActiveRecord::NoDatabaseError: could not translate host name "mastodon_db_1" to address: Name or service not known
What does that mean?
How I got here
I am really not sure. I was doing some server maintenance around docker services and then the errors start to pour, even from the ones that I did not touch. Now when all the issues are resolved I feel strange, because things work but obviously I do not know exactly why.
On the other hand, no one can know everything and without professional training I am left to tinker and hack around the docker world. And again, lessons are learned until things start to work. The happy part of this is I was able to get everything back to normal. The sad thing is I still do not know what the hell has happened.
Tracing the problem
The obvious part was that the Mastodon instance would not open in the web tab with the HTTP 5xx error code. There were various repeating errors in the docker log, but all of them were stemming from the error message above. Yet I had no way to know which single one error message from that wall of text was the culprit.
The obvious part was that the Mastodon instance would not open in the web tab with the HTTP 5xx error code. There were various repeating errors in the docker log, but all of them were stemming from the error message above. Yet I had no way to know which single one error message from that wall of text was the culprit.
The solution
Fortunately I have noticed that error message and what it says is actually
quite clear. There is no host with name mastodon_db_1 to connect to.
Right. All the other error messages were probably there because the
database is not accessible. I knew the services were defined in
docker-compose.yml file. And I knew equally well it worked few hours
before. And I knew I was not touching that folder at all, so something else
had changed.
What was especially striking was that the services the docker-compose were
started differently - in the form of mastodon-db-1. Ripgrepping around
the repository finally led me to the .env.production file that contained
the value mastodon_db_1. The solution?
- DB_HOST=mastodon_db_1
+ DB_HOST=mastodon-db-1
- REDIS_HOST=mastodon_redis_1
+ REDIS_HOST=mastodon-redis-1
- ES_HOST=mastodon_es_1
+ ES_HOST=mastodon-es-1
Similar changes needed to be done in other projects so I believe there had to be some update in the docker ecosystem somewhere, but I was not able to pinpoint it exactly yet. Searching does not show any breaking change in past month or two. Maybe I do not search hard enough ...