Today marks a new page for my personal infra: a vital server was decommissioned and replaced by another one.
Old server
The initial VPS was brought up some time around 2018, which makes it my oldest server system. Recently the old VPS started to have disk I/O issues and required manual fsck from the recovery partition on every boot. The server plays an important role in the integrity of my virtual network and various services (Prometheus, Grafana, etc.) and just disabling it was not an option.
Verdict: the server needs to go. Quick and clean.
Replacement process
First I made a comprehensive list of services and redirects that were running on the server (not that I didn’t have the list, I just needed to be sure). Since a lot of connections are dependent on this, a way to plug-and-play the new server should be developed.
As a reference, below you can see an approximate network diagram which shows just how important VPS is in the network.
The plan was to copy over everything that’s actively running on the old machine to the new one and then redirect domain registrar DNS records to the new hosting provider DNS servers which should result in full server replacement with minimal downtime. Which it did!
After the server was fully copied over and DNS redirected, I wrote a simple script that sent me a Telegram message when the IP for osetr.su would change in Google DNS. After I received the message, DNS cache was (almost) instantly updated on all other servers and with a VPN connection downtime of only 7 minutes everything clicked together and the migration was finished.
I’m sure I could’ve had an even shorter downtime if DNS setting on my servers would be properly synchronized and managed.
Conclusion
With the new server in place, the following was improved:
- No more annoying I/O errors!
- Increased VPN security (better firewall)
- Nice and fresh OS on a clean server (always feels so nice)
- Better DNS management (personal DNS server was decommissioned)
I’ve wanted to do this for so long and I’m so happy I finally found time to do that! Just beautiful.