Orchestrator fails to start after reboot using SystemD
While testing in an orchestrator lab I saw that none of my Orchestrator on-raft nodes were coming online after a reboot. This is the status report from SystemD.
$ sudo systemctl status orchestrator * orchestrator.service - orchestrator: MySQL replication management and visualization Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2020-04-03 09:30:05 UTC; 30s ago Docs: https://github.com/github/orchestrator Main PID: 957 (code=exited, status=1/FAILURE) Apr 03 09:30:05 orchestrator-1 systemd[1]: Started orchestrator: MySQL replication management and visualization. Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 ERROR dial tcp 127.0.0.1:3306: connect: connection refused Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 FATAL dial tcp 127.0.0.1:3306: connect: connection refused Apr 03 09:30:05 orchestrator-1 systemd[1]: orchestrator.service: main process exited, code=exited, status=1/FAILURE Apr 03 09:30:05 orchestrator-1 systemd[1]: Unit orchestrator.service entered failed state. Apr 03 09:30:05 orchestrator-1 systemd[1]: orchestrator.service failed.In this cluster, I'm using a MySQL backend for my Orchestrator instances. It seems that Orchestrator tried to start BEFORE my MySQL service was available. I did research on how I could set dependencies for the services in SystemD. I came across this article in Fedora Magazine. In our case, the
Wants relationship is not good, as this only guarantees that both services will be started, but we need MySQL to be available before Orchestrator can start and we don't want to place a hard requirement on MySQL as Orchestrator works fine with SQLite. This made me choose for the
Before of
After relationships. Since we are configuring Orchestrator and not MySQL we will update the Orchestrator service config. This means Orchestrator needs to start
After MySQL is started. SystemD offers a convenient way to configure overrides by using
systemctl edit <service>. When you have no overrides configured yet, it will open an editor with an empty file. Dependencies are configured in the
[Unit] section of the service definition. The following example shows how we can define our
orchestrator service to be started after the
mysqld service.
$ sudo systemctl edit orchestrator <add this content> [Unit] After=mysqld.serviceNote: You can read more about the dependency configuration in this manual. Let's test our changes by rebooting the machine.
$ sudo reboot ... $ sudo systemctl status orchestrator * orchestrator.service - orchestrator: MySQL replication management and visualization Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/orchestrator.service.d +- override.conf Active: active (running) since Fri 2020-04-03 09:36:32 UTC; 52s ago Docs: https://github.com/github/orchestrator Main PID: 1483 (orchestrator) CGroup: /system.slice/orchestrator.service +- 1483 /usr/local/orchestrator/orchestrator http You can now see my reboot Orchestrator started up fine.To end this problem, I went looking in the Orchestrator issue tracker and I found this issue where the same problem is described. I have submitted a pull-request and when accepted this issue will be addressed. Until then, if you have not found the issue, I hope this blog has saves you troubleshooting time looking for a solution.
