19

Every now and then I need to reinstall Linux system (mainly Ubuntu based distros). The process of reinstalling every piece of software I need is really boring and time consuming. Is there any software that can help me out? For Windows there is Ninite, is there something else for Linux?

Edit: Thanks for all the answers! I went with Ansible and it is an amazing tool.

  • 18
    Why do you need to reinstall your system so often? – Billy Apr 27 at 4:06
  • 2
    As @Martin have answered, using the tools that comes with your Ubuntu or Debian (dpkg and apt) work really well, as they are part of your system (you don't have to install them, they are already there). You don't need to deal with more complicated tools as Ansible and Nix - although it is a good idea to learn about them. – Henrique Apr 27 at 18:07
  • 1
    Also, thank you for the Ninite tip! I didn't know about it! Neat! – Henrique Apr 27 at 18:08
  • 1
    Can you get away with something as simple as a bash script containing a single command: "sudo apt install X Y Z" ? – Jonathan Hartley Apr 27 at 21:41
  • 2
    @Henrique If Martin already have answered the question, why did you then ignore the text that says "Avoid answering questions in comments."? It's there for a reason. – pipe Apr 27 at 23:21
10

Ansible is an open-source software provisioning, configuration management, and application-deployment tool. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration

(From Wikipedia.) Homepage (Github).

There are several others in the same category. Reading about should give you vocabulary to search for the others, and compare, if needed. Nix is a newer contender. Some say "more complex, but maybe just right.". is also on the scene.

Ansible example for hostname myhost, module apt (replace with yum or whatever):

ansible -K -i myhost, -m apt -a "name=tcpdump,tmux state=present" --become myhost

The list "tcpdump,tmux" can be extended with commas. (The fact, that the hostname myhost is twice in the command-line, because we are not using a fixed host inventory list, but an ad-hoc one, with the trailing comma.)

This only scratches the surface, Ansible has an extensive module collection.

| improve this answer | |
  • Using a tool like ansible also helps with maintaining configuration of installed applications. – BillThor Apr 26 at 14:21
  • 1
    Ansible is a great idea, though you can (and probably should) go a lot farther with it. I have a 319-line playbook plus some ancillary files and templates that installs and configures my local laptop from a fresh OS installation. When I change something, I change the playbook and run it. When I have to reinstall the OS, I then rerun the playbook and everything is restored (unless I forgot something, which does happen). – Michael Hampton Apr 26 at 22:06
  • 4
    Ansible is to Nix what Bash is to Haskell. If you have the time and energy to learn Nix it is vastly superior to Ansible in several ways: reproducibility, actually being declarative rather than just a thin layer on top of shell commands, isolated versions (so you can install for example two versions of system software like Bash or Python at the same time). Nix takes about the same time to learn as Ansible, but is more abstract and has fewer limitations and surprises (this is after using Ansible for two years and Nix for about three months). – l0b0 Apr 27 at 3:09
  • 1
    Hi, could the down-voter please elaborate on the reason, please? – Alex Stragies Apr 27 at 5:42
  • 2
    @l0b0: Nix is great but unless something has vastly changed in regards to what it is, it has nothing to do with using distro packages and is not a substitute or competitor for ansible. Rather it's a substitute for the whole Debian (and other distros) package system. – R.. GitHub STOP HELPING ICE Apr 27 at 19:25
41

Save and restore all packages

On Debian/Ubuntu based systems you can dump the list of installed packages to a file

dpkg --get-selections > my_package_list.txt

and install them again by running

apt-cache dumpavail | sudo dpkg --merge-avail
sudo dpkg --set-selections < my_package_list.txt
sudo apt-get dselect-upgrade

The first line ensures dpkg's database is up to date (nowadays most people use apt to install packages which maintains its own database), the second line imports your saved selections and the third command installs the selected packages. See the Debian Administrator's Handbook for details.

In case you have installed packages from third-party repositories you need to add these repositories before following the above steps for restoring.

Save and restore only explicitly installed packages

Aptitude automatically installs dependencies (e.g. a library required by an application). These packages are flagged as "automatic" and can be automatically removed when they aren't required anymore. In case you want to preserve these "automatic" flags we can't use dpkg since it doesn't track automatically installed packages. Instead we have to use the slightly more cryptic

LC_ALL=C aptitude search -F '%p %C' '~i!~M' > my_package_list.txt

This will search all packages that are installed (~i) and not (!) flagged automatic (~M). For each matching package the package name (%p) and the status (%C) will be printed (this mimics the output of dpkg --get-selections). LC_ALL=C ensures that all output is done in English without translation to a native language. Use the commands mentioned above for installing the packages from this list. Refer to the aptitude user's manual for details.

| improve this answer | |
  • Do I understand correctly that, usually, apt install updates dpkg's internal availability status info, so if you were to omit the dpkg --merge-avail step, it would tell you it doesn't know those packages despite them being in apt's cache (since I assume apt update was run on the new system)? – Luc Apr 27 at 9:24
3

If you just want to install a bunch of packages a simple one-liner could do like:

sudo bash -c 'for package in "tmux" "htop" "gimp"; do apt install -y --no-upgrade "$package"; done'

The loop is not strictly necessary, but without it, if apt fails to find any of the programs in the list, it will fail to install any of the other packages. This can happen for example if you switch to a more recent version of your distro and older packages are not within the repos anymore. If you prefer all or nothing use

sudo apt install -y --no-upgrade tmux htop gimp

If you also want to save your configurations the search term would be "dotfiles". Thats what the configurations in Unix like systems are called since they mostly start with a ".".

A quick and dirty way to save those is just by copying all those configurations directory to your new system. A better way would be to place them under version control with tools like git. I use a combination of git, dotbot and hand written scripts to setup my system.

Update

One point that is missing from the discussion so far is that apt is typically not the only package management system one needs for anything beyond the bare basics. Other package management tools might be snap, pip, conda, cargo and many more. This is implicitly addressed in the answer by Alex Stragies. Ansible contains a vast ammount of modules including modules to manage packages apart from apt like snap and pip. As my answer is focused on write-your-own-script I'd like to expand on that. A well tested framework such as Ansible should generally be prefered for most tasks, but self-written code gives an advantage in terms of flexibility in my eyes.

Small example framework

I've written a small code in python which shall examplify how such a framework could look.

#!/usr/bin/env python3

import os
import re
import sys
import subprocess

def read_package_list(path):
    package_list=[]
    try:
        with open(os.path.realpath(path)) as f:
            for line in f:
                match = re.search(r'^(?!\s*$)(?!#)\w+',line)
                if match:
                    package_list.append(match.group(0))
            return package_list
    except Exception as e:
        print(e.message)
        print(e.args)
        sys.exit(1)    
    return package_list

def install_packages(command,package_list,err_log):
    try:
        with open(err_log,'w+') as f:
            for p in package_list:
                print('executing '+command+' '+str(p))
                out=subprocess.run(command+' '+p,shell=True,stderr=f)
    except Exception as e:
        print(e.message)
        print(e.args)
        sys.exit(1)

def main():
    args = sys.argv[1:]
    package_list = read_package_list(args[1])
    err_log=os.path.realpath(args[2])
    install_packages(args[0],package_list,err_log)

if __name__ == '__main__':
    main()

The basic ingredients are a function to process a list of packages separated by newlines (read_package_list) and a function to execute the installer command in a shell (install_packages). Lines with only whitespace and lines starting with # are ignored when reading in the package list. The main processes the arguments which can be given on the command line as installer command, packagefile, errorlog.

What does that give me?

Well you can just use any installer command you like

./installerscript.py 'apt install --dry-run' myaptpackages.txt apt_err.log
./installerscript.py 'snap install' mysnaps.txt snap_err.log
./installerscript.py 'pip install --user' mypy.txt py_err.log
./installerscript.py 'git clone' repos.txt git_err.log

This might be helpful if one keeps a list of packages which should all be treated in the same way. Once such a framework exist it is easy to improve on it. One could, for example, customize the way the installation process is logged or customize the processing of the command line arguments. Another aspect is that the script probably shouldn't execute each command as root (if run as root) as it currently does.

| improve this answer | |
  • 2
    Silently ignoring error messages by doing a for loop seems hacky at best. If apt fails to find a package, I'd like to know of it. When installing a laptop/desktop I want to find a replacement or at least I know I will be missing a package, or for unattended deployments the automatic install should fail because the system environment changed and the script needs updating. – Luc Apr 27 at 9:34
  • 1
    @Luc executing apt in a loop does not discard lines written to stderr. They will still be shown on the command line, although admittedly harder to find between the stdout output. One could add something like 2>errors.log to get a basic form of error logging. – Max1 Apr 28 at 10:28
1

If you install software from the command line, then doing

grep "^sudo apt install" ~/.bash_history > system-setup.sh

once the system setup is complete will provide you with a script, which (after some editing) can be reused to set up a freshly installed system next time you need it.

| improve this answer | |
  • 7
    What if your history file doesn't go back till the beginning of time? What if you use apt-get sometimes? What if you used a visual front-end? What if you also want packages installed as part of an installation guide that used "apt update && apt install x"? I'm not sure this is a solution, especially 7 hours after Martin posted Debian's solution. That said, I do have a system-setup script myself where I put my favorite packages for different purposes (one base install, one for graphical environments, one with IDEs, one with WiFi tools if the system has WiFi, etc.), but that wasn't the question. – Luc Apr 27 at 9:31
  • @Luc The OP presumably reinstalls the system regularly, so there's a good chance they will have everything still in history. Ideally this should be done right after the system setup is complete. Grepping for apt-get instead of apt is trivial. – Dmitry Grigoryev Apr 27 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.