Orchestrator fails to start after reboot using SystemD

Posted in: MySQL, Open Source, Technical Track

While testing in an orchestrator lab I saw that none of my Orchestrator on-raft nodes were coming online after a reboot.

This is the status report from SystemD.

$ sudo systemctl status orchestrator
* orchestrator.service - orchestrator: MySQL replication management and visualization
   Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-04-03 09:30:05 UTC; 30s ago
     Docs: https://github.com/github/orchestrator
 Main PID: 957 (code=exited, status=1/FAILURE)

Apr 03 09:30:05 orchestrator-1 systemd[1]: Started orchestrator: MySQL replication management and visualization.
Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 ERROR dial tcp 127.0.0.1:3306: connect: connection refused
Apr 03 09:30:05 orchestrator-1 orchestrator[957]: 2020-04-03 09:30:05 FATAL dial tcp 127.0.0.1:3306: connect: connection refused
Apr 03 09:30:05 orchestrator-1 systemd[1]: orchestrator.service: main process exited, code=exited, status=1/FAILURE
Apr 03 09:30:05 orchestrator-1 systemd[1]: Unit orchestrator.service entered failed state.
Apr 03 09:30:05 orchestrator-1 systemd[1]: orchestrator.service failed.

In this cluster, I’m using a MySQL backend for my Orchestrator instances. It seems that Orchestrator tried to start BEFORE my MySQL service was available. I did research on how I could set dependencies for the services in SystemD. I came across this article in Fedora Magazine. In our case, the Wants relationship is not good, as this only guarantees that both services will be started, but we need MySQL to be available before Orchestrator can start and we don’t want to place a hard requirement on MySQL as Orchestrator works fine with SQLite. This made me choose for the Before of After relationships. Since we are configuring Orchestrator and not MySQL we will update the Orchestrator service config. This means Orchestrator needs to start After MySQL is started.

SystemD offers a convenient way to configure overrides by using systemctl edit <service>. When you have no overrides configured yet, it will open an editor with an empty file. Dependencies are configured in the [Unit] section of the service definition. The following example shows how we can define our orchestrator service to be started after the mysqld service.

$ sudo systemctl edit orchestrator
<add this content>
[Unit]
After=mysqld.service

Note: You can read more about the dependency configuration in this manual.

Let’s test our changes by rebooting the machine.

$ sudo reboot
...
$ sudo systemctl status orchestrator
* orchestrator.service - orchestrator: MySQL replication management and visualization
   Loaded: loaded (/etc/systemd/system/orchestrator.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/orchestrator.service.d
           +- override.conf
   Active: active (running) since Fri 2020-04-03 09:36:32 UTC; 52s ago
     Docs: https://github.com/github/orchestrator
 Main PID: 1483 (orchestrator)
   CGroup: /system.slice/orchestrator.service
           +- 1483 /usr/local/orchestrator/orchestrator http

You can now see my reboot Orchestrator started up fine.

To end this problem, I went looking in the Orchestrator issue tracker and I found this issue where the same problem is described. I have submitted a pull-request and when accepted this issue will be addressed. Until then, if you have not found the issue, I hope this blog has saves you troubleshooting time looking for a solution.

email

Interested in working with Matthias? Schedule a tech call.

About the Author

Lead Database Consultant
Matthias has been passionate about computers since the age of 10. He has been working with them ever since. Currently he's a Lead Database Consultant in one of the MySQL teams at Pythian where he's the technical lead for his team. Together with his team he works to provide the best possible service to the customers.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *