How to continuously upgrade everything (including some Debian distro release) in a distro in a safe way?

Why I ask this question

I ask this question since I feel like I'm doing something wrong when my Nginx environment installation scripts become larger and larger, and take more and more time to maintain.

Maintaining the scripts is something I can and love to handle with because of learning, practice, and communal sharing, even though I must go out to work as an employee soon and will have less time to maintain these. Given these circumstances I always have this fears like:

-- What if my current distro LTS is no longer supported.

-- Even if it's Debian stable and not something else I might have security breaches because of an old Distro just a few months forward but not more than that.

-- What if a well configured unattended-upgrades won't suffice giving me security quiet because the distro got old and so forth.

As for now I pay 5 USDs on a droplet in DigitalOcean and I really don't know what's the best solution for me to have a LEMP stack with these characteristics:

  • Freely accessible Bash.
  • Unattended-upgrades.
  • CSF-LFD, and Maldet.
  • Certbot-handled webapp confs.
  • Immediate and automated cron backups as I have created in cron (available in my scripts).
  • Automatic distro upgrades (say from Ubuntu 16.04 to 18.04 and from 18.04 to 20.04 at the time and so forth), without me moving the data manually, even with rsync.
  • WordPress websites
  • WP-CLI
  • All in not more than 10 USDs for VPS.

I'm not sure Ansible or Salt are the answer (Ansible doesn't seem to me very easy to learn even though it is marketed and rumored to be easy to learn and I'm afraid I don't have the time to study it in depth in the next 3 months).

I'm not sure Azure is the answer (only because of price, I understand an environment like I desire would cost at least 15USDs in Azure).

Maybe I need a distro with continuous delivery in every basic aspect (assuming there is one).

That's the story basically.

closed as too broad by Michael Homer, garethTheRed, Stephen Kitt, Jeff Schaller, Romeo Ninov Jan 10 '18 at 11:24

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • If the unlikely happened and this question was closed or locked for any reason - please reply in comments or in chat with tagging my profile. I would thank you dearly for your advice. – Arcticooling Jan 10 '18 at 6:33
  • 2
    Release upgrades (like Ubuntu 16.04 to 18.04) are major changes which should not be automated IMHO. Packages necessary for you may no longer be available after an upgrade or be in an incompatible version, it requires a reboot and might possibly be broken afterwards or in need of manual intervention. After all, it only occurs every 2 or 4 years that you have to do an LTS upgrade, I see no need to try automating this unless you have to do it on dozens of machines of the same type. Otherwise look into rolling distributions like Arch which do not really have release versions at all. – Byte Commander Jan 10 '18 at 9:53
  • Hi @ByteCommander (nice nick), sadly DigitalOcean doesn't allow Arch droplets but there might be another hosting provider that allow these, or allow installation of whatever Linux I want. Will you say FreeBSD (which DO provides) suffice for that purpose? – Arcticooling Jan 10 '18 at 10:42
  • Can't say, never worked with that. As Oli pointed out in his answer though, rolling will mean no major release updates, but even more frequent smaller updates, so the overall change of encountering updates that break anything is probably similar. – Byte Commander Jan 10 '18 at 10:51

Good CI (Continuous Integration) relies on at least three things:

  1. Scripted deployment
  2. An out of band testing environment
  3. Automated tests to cover the entire surface.

You currently only have #1. Another VPS could serve as #2 but you don't have any (obvious) testing.

But before we go any further, just a quick note in favour of cadence-based releases.

Ubuntu LTS releases gets updates for five years after release. Not every package but everything required for LEMP. These updates backport security fixes from future versions so you get the old major version (where possible) with the modern fixes. This means things keep working.

A rolling release updates everything, at any time. You don't know when a configuration-breaking change (eg systemd) will land, or how much work it'll take to fix. If you're installing updates automatically, that also means there's no predictable interval on time demands.

With an Ubuntu LTS, you know that once every 4-5 years you'll need to spend an hour or two on your server guiding it through an upgrade. You get to pick when you do this within a window of 3 years after the next LTS is released. That's a major feature, not a flaw.

For your battle plan, I would —perhaps controversially— disable automatic updates completely and replace them with a relatively simple script that:

  • Spins up a second VPS (using your vendor's API so you only pay for it while in action). Based on a template of the same OS version (eg Ubuntu 16.04 if your main VPS is on that).
  • Updates to latest versions of your software, deploys your stuff and restores databases
  • Runs your test suite and reports back a positive or negative for all services
  • Emails you on fail, runs apt dist-upgrade on the main VPS on success.

It's key that you get loud feedback if you aren't getting updates. And yes, you'll need to test that too.

You could move to Ansible (et al) but that isn't your main problem here. Writing tests is.

This is a lot of work but this stuff puts you in good stead to scale up and out too. If you get to the point where you have a load balanced cluster, you can separate one, update it and test before reintroducing it to the cluster.

  • 1
    "For your battle plan, I would —perhaps controversially— disable automatic updates completely"... I disagree with the "controversially. It is the best method. Updates should be done at a convenient time. Problems resulting from an update are not an issue; those can and will always happen. It is the TIME you need to fix the problems when they arise. If you do not have that then there is a big problem. – Rinzwind Jan 10 '18 at 10:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.