24

Is there a tool or even an entire distribution that supports rolling-back changed packages after an update?

As an example: I upgraded packages A, B and C. After working with those packages for several days, I encounter a bug in B that is deal-breaking.

While I'd submit a bugreport, I'd also need to downgrade B to the previous version so that I can finish what I was about to do. Meanwhile A is dependent on B, so it'd need to be downgraded as well, but C is independent of both, so it could stay at its current version.

Is there a tool or a distribution that supports this?

I know that most distributions have a way of downgrading a package but that's usually kind of sketchy or not even possible because the previous package was removed from repositories and it some cases (for example after upgrading the X server and Mesa) it gets really... messy.

Improve this question
2
  • 3
    Thing to note: if the package version change was minor, answers below may apply; however note that that bigger package changes might upgrade on-disk data which won't work correctly afterwards in older version. For example, mysql-server (or joomla) major upgrade will add and modify fields and upgrade SQL tables, inn2 upgrade might change database type, or some distro kernel upgrade might upgrade ext3 to ext4 filesystem, or some package upgrade will convert config files, etc. The only protection against those "unrollbackable" changes are LVM/btrfs/etc snapshots (or much slower backup/restore).
    – Matija Nalis
    Commented Mar 11, 2016 at 17:15
  • @MatijaNalis +1 for mentioning that!
    – Steffen Winkler
    Commented Mar 11, 2016 at 17:40

7 Answers 7

Reset to default
21

NixOS supports upgrade rollbacks, although as I understand it, it doesn't go quite as far as you'd like: if you upgrade A, B and C in one operation, you can roll that entire operation back, but not just A and B. (You should be able to roll A, B and C back, and then upgrade C...) That makes sense from a transactional perspective though.

Debian (in combination with the snapshot archive if you no longer have the old packages) will allow you to downgrade B, and tools like apt or aptitude will in many cases figure out that A also needs to be downgraded (once you've convinced them that you don't want to simply upgrade B). But as you say that tends to be somewhat messy, and package downgrades are unsupported in Debian anyway (which means that most of the time they work, but if they break it's not a bug).

Improve this answer
5
  • 1
    that looks interesting! The community looks active and healthy. I'll definetly give this a try, thank you! If by Friday evening there is no better/different answer, I'll mark your answer.
    – Steffen Winkler
    Commented Mar 10, 2016 at 9:09
  • Rollbacks on NixOS are awesome, but the real power comes from the declarative approach: The packages (and system) description can be taken from a git repository, thus you can manage your system the same way you'd manage a software project, including branches and merges etc. (and due to atomic upgrades and purity you never break stuff)
    – Daniel Jour
    Commented Mar 10, 2016 at 19:04
  • The Nix package manager can also be run on other Linux distros, side-by-side with their 'native' package manager. It also runs on OSX, and I've seen claims that it might work on Windows. You can also tell Nix to use 'native' versions of some packages, rather than installing its own duplicates, although you lose some of its guarantees that way (e.g. it may not notice that you've swapped out some dependency).
    – Warbo
    Commented Mar 10, 2016 at 19:29
  • just FYI, I managed to install NixOS on my laptop (the KDE4 image gave me a Kernel Panic, but the small (~390 MB) image booted fine. For someone who only ever installed Debian based distributions this was quite interesting/fun. However, I do not yet understand certain things about the package manager, especially when it comes to desktop environments. I installed gdm/gnome-shell but it wouldn't work. I then enabled gnome3/gdm in the configuration.nix file and on rebuilding it, both packages and their dependencies were downloaded again. It did work after a reboot but I don't understand why.
    – Steffen Winkler
    Commented Mar 15, 2016 at 7:25
  • 1
    @SteffenWinkler When using NixOS, you do not normally install things with nix-env, instead you specify things in configuration.nix and then run nixos-rebuild switch. This has the benefit that all of your system configuration is in one place, and it's easy to backup your entire system's configuration (just backup the configuration.nix file).
    – Pauan
    Commented Nov 29, 2017 at 3:57
15

On any yum based distribution (e.g. Red Hat EL, CentOS, etc), you can:

  1. examine the history of changes to system using sudo yum history list

    Loaded plugins: fastestmirror
ID     | Login user               | Date and time    | Action(s)      | Altered
------------------------------------------------------------------------------
    10 | Administrator <admin>    | 2016-03-08 09:08 | Install        |   11   
     9 | Administrator <admin>    | 2016-03-03 16:48 | Install        |    1   
     8 | Administrator <admin>    | 2016-03-03 16:09 | Install        |    5   
     7 | Administrator <admin>    | 2016-02-26 18:13 | Install        |    1   
     6 | Administrator <admin>    | 2016-02-26 15:12 | Install        |   27   
     5 | Administrator <admin>    | 2016-02-26 15:07 | Install        |    1   
     4 | Administrator <admin>    | 2016-02-26 15:05 | Install        |    3  <
     3 | Administrator <admin>    | 2016-02-26 15:03 | Install        |    1 > 
     2 | Administrator <admin>    | 2016-02-26 15:01 | I, U           |   49   
     1 | System <unset>           | 2016-02-26 14:38 | Install        |  296   
history list

  2. check the details, using sudo yum history info 10

  3. rollback to a previous point in the history, using sudo yum history rollback 9

Warning

There are some obvious caveats:

  1. If the old package isn't available anymore, you are toast (to quote @vonbrand),
  2. If you install anything outside yum, you could break the history.

In my example, that < in the row with ID 4 (in the last column), means I cannot rollback past that point. 

sudo yum history rollback 2
Loaded plugins: fastestmirror
Transaction history is incomplete, before 4.
 You can use 'history rollback force', to try anyway.
Error: Failed history rollback, incomplete
Improve this answer
6
  • 3
    interesting feature! But due to the 'toast' doesn't fit my bill.
    – Steffen Winkler
    Commented Mar 10, 2016 at 12:46
  • AFAIK this is only possible in RHEL/CentOS 6 or above. If you're still using RHEL/CentOS 5 you're SOL.
    – Wildcard
    Commented Mar 11, 2016 at 15:56
  • And by the way, in a corporate environment with managed repos, this approach is quite workable as you always have the old packages around, even if you need to use an --enablerepo flag to enable an otherwise disused old repository for the downgrade.
    – Wildcard
    Commented Mar 11, 2016 at 15:58
  • What's the benefit of rolling back versus just installing the older version again?
    – Bratchley
    Commented Mar 11, 2016 at 16:30
  • @Bratchley automation! yum simply takes care of keeping the list of packages and versions installed and checks for dependencies when reinstalling old versions. Obviously you can do it by hand.
    – andcoz
    Commented Mar 11, 2016 at 16:45
7

On OpenSUSE you can easily use Snapper with the Btrfs filesystem.

If you use the standard file-system configuration during installation, it is enabled by default.

Once Snapper is enabled, it is fully integrated with yast2 and zypper. It will create a filesystem snapshot every time you'd install or upgrade something (or create an user, etc).

To rollback the system to a previous condition, you have only to run yast2 snapper.

enter image description here

Improve this answer
7
  • interesting tool indeed. Although I'm quite fond of ext4. Will research this! Am I correct in assuming that Snapper is not 'bound' to OpenSUSE but to btrfs?
    – Steffen Winkler
    Commented Mar 10, 2016 at 12:44
  • Snapper is developed by SUSE. It is a collection of tools that automate the creation of the Brtfs snapshots on "events". I suppose you can use it on other distributions but I am not sure. In any case, you can manually create Brtfs snapshots on any distribution.
    – andcoz
    Commented Mar 10, 2016 at 13:34
  • 4
    Just keep in mind that if your volume contains any data, not just binaries and scripts, that rolling back a snapshot will roll back your own data too. This option sounds risky, best suited for those that really know the layout of their filesystem. For me, snapshots have always been for consistent backups, replication, and total recovery situations.
    – jimp
    Commented Mar 10, 2016 at 18:03
  • 1
    Note also that many packages have files and/or directories in /var so it has to be rolled back along with / even if it is a separate fs or subvolume. It is far better to thoroughly test an upgrade before applying it to production servers than to rely on features like OS upgrade rollback (which is trivially easy to do in a half-arsed manner but is an extremely difficult problem to solve correctly).
    – cas
    Commented Mar 11, 2016 at 3:20
  • 1
    @Jimp A good advice. In any case, OpenSuSE doesn't enable snapper unless /home is in a separate non snapshoting filesystem.
    – andcoz
    Commented Mar 11, 2016 at 10:54
7

AIX is very good at rolling back updates. Well - we're on Unix/Linux site and you've never specified you want Linux :)

Each single AIX update saves all the modified files in a separate subdirectory inside /var filesystem. The update can be reverted with a simple native command, and the revert doesn't need the network to be up, it doesn't need any media/packages, it doesn't re-install anything and it doesn't depend on any snapshot technology - the effect is simply that the files re-appear as they were before the update.

As a bonus, there is one trivial native command mksysb to create a bootable stand-alone system backup. The file that can be simply booted on a completely dysfunctional system that doesn't boot up due to some malfunction/corruption.

And it's all proven technology with decades of history :)

Improve this answer
3
  • Do you often run into problems with programs restored in that way being in an inconsistent state and rollback (ie, a database whose storage engine has changed or a service that switched to a new config file format), or are there good ways to work about that?
    – Josh Rumbut
    Commented Mar 10, 2016 at 19:00
  • 1
    This is the model I used when I built a rudimentary package management system for software that had traditionally been distributed in compressed/gzipped tarballs. Any time it installed updates, it first built a "rollback package" of everything that was to be modified. This rollback package could then be used to revert, provided that no other update packages had been applied in the interim. I have often wondered whether proper package managers could do this, but given the constraints of rolling back only in reverse order... maybe we should just use git instead.
    – Monty Harder
    Commented Mar 10, 2016 at 19:49
  • damn it. I should've specified that I meant GNU/Linux distributions. But interestingly, AIX seems to be capable of running GNU/Linux 'programs' nowadays, guess I'll take a closer look.
    – Steffen Winkler
    Commented Mar 11, 2016 at 8:35
5

In Fedora (and I'm sure in other distributions too) you can ask to go back to a previous version:

dnf downgrade <packages>

gets you the next-to-last version of the packages, and you can ask for a specific one by:

dnf downgrade <package>.<version>

This only works if the package(s) are still available in the repositories. The functionality isn't unheard off, by any means. It has its snags, if part of the upgrade was to change configurations, the rollback won't necessarily be to the exact past version.

Improve this answer
6
  • You could also use dnf history undo #thing to undo
    – Clearer
    Commented Mar 10, 2016 at 10:55
  • This only works if the package(s) are still available in the repositories. yup, that's exactly the problem. I'm kind of surprised that there doesn't seem to be a 'broader' solution for this, especially with the amount of rolling-release distributions. As of now it seems like NixOS is my best option, or I'd need some kind of system imaging tool that works on differences only and can restore the system to a specific point in time over the last 20(?) updates.
    – Steffen Winkler
    Commented Mar 10, 2016 at 10:56
  • @SteffenWinkler, if the old package isn't available anymore, you are toast. Obviously. Unless you have some sort of local backup.
    – vonbrand
    Commented Mar 10, 2016 at 11:34
  • 2
    @MTilsted, because it is already too old? Repositories don't contain all versions from the dawn of time.
    – vonbrand
    Commented Mar 10, 2016 at 17:07
  • 2
    In dnf.conf (same as yum.conf) has keepcache=true Old packages available, as lond cache is not cleared manually. But it went to "bloat" package cache.
    – mmv-ru
    Commented Mar 11, 2016 at 15:54
2

Arch Linux also supports downgrading packages and the kernel. You can also install the downgrader and downgrade tools to automate the process. The btrfs solution also works, I have used it to do a manual rollback before.

How I rollback my system:

sudo -i
mount /dev/sda3 /mnt/hd #mount the top btrfs subvolume
ls #find the version you want
mv @ @-old #move the '/' subvolume (I named mine '@')
btrfs sub snap @-<date> @ #replace @ with the backup from <date>
sync
reboot #the changes will take effect once the system restarts

One benefit of btrfs is that you can use subvolumes and dynamic "partitions." For instance, I have a subvolume for / (called @), /tmp (@tmp), and /home (@home). It is then easy to back up and roll back any of these. I have /tmp in a separate subvolume because backing it up with the rest of the system seems pointless, since it is erased on almost every reboot.

Improve this answer
3
  • I followed a link in the wiki and got here. That looks pretty cool and from checking the repository, there are files going back to 2013! Why use the btrfs way when this exist? Because of files that are 'upgraded'? Or is there some other reason?
    – Steffen Winkler
    Commented Mar 15, 2016 at 7:29
  • I now switched permanently to Arch Linux after paying NixOS a short visit. It (Arch Linux) is not nearly as unstable as I always thought it'd be and it has a healthy community.
    – Steffen Winkler
    Commented Nov 7, 2016 at 11:59
  • @SteffenWinkler That is correct. Also, the Arch downgrade options are non "officially" supported and downgrading will probably ignore dependencies (which can get quite messy if something goes wrong).
    – Caleb Reister
    Commented Nov 23, 2016 at 1:53
2

I use Arch Linux and it stores all downloaded packages to /var/cache/pacman/pkg/ so you can downgrade any package anytime (of you are unable to boot, use a live usb). From Arch Wiki:

pacman -U <file_name_of_the_package>

To prevent package from being upgraded, include the package name in /etc/pacman.conf, like:

IgnorePkg=linux

To save space you can clear the cache folder with:

pacman -Sc

Which will delete all older packages and keep the newest one, or use -Scc to remove all.

Improve this answer
2
  • Note that this (ignoring packages) amounts to a partial upgrade, which is unsupported...
    – jasonwryan
    Commented Mar 14, 2016 at 23:40
  • I followed a link in the wiki and got here. That looks pretty cool and from checking the repository, there are files going back to 2013! Why use the btrfs way when this exist? Because of files that are 'upgraded'? Or is there some other reason?
    – Steffen Winkler
    Commented Mar 15, 2016 at 7:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .