How to disable `apt-daily.service` on Ubuntu cloud VM image?

Question

The Ubuntu 16.04 server VM image apparently starts the "apt-daily.service" every 12 hours or so; this service performs various APT-related tasks like refreshing the list of available packages, performing unattended upgrades if needed, etc.

When starting from a VM "snapshot", the service is triggered immediately, as (I presume) systemd realizes quickly that the timer should have gone off long ago.

However, a running APT prevents other apt processes from running as it holds a lock on /var/lib/dpkg. I need to disable this automated APT task until Ansible has completed the machine setup (which typically involves installing packages); see https://github.com/gc3-uzh-ch/elasticluster/issues/304 for more info and context.

I have tried various options to disable the "unattended upgrades" feature through a "user data" script for cloud-init, but all of them have failed so far.

1. Disable the systemd task

systemd task apt-daily.service is triggered by apt-daily.timer. I have tried to disable one or the other, or both, with various cobinations of the following commands; still, the apt-daily.service is started moments after the VM becomes ready to accept SSH connections::

    #!/bin/bash

    systemctl stop apt-daily.timer
    systemctl disable apt-daily.timer
    systemctl mask apt-daily.service
    systemctl daemon-reload

2. Disable config option APT::Periodic::Enable

Script /usr/lib/apt/apt.systemd.daily reads a few APT configuration variables; the setting APT::Periodic::Enable disables the functionality altogether (lines 331--337). I have tried disabling it with the following script::

    #!/bin/bash

    # cannot use /etc/apt/apt.conf.d/10periodic as suggested in
    # /usr/lib/apt/apt.systemd.daily, as Ubuntu distributes the
    # unattended upgrades stuff with priority 20 and 50 ...
    # so override everything with a 99xxx file
    cat > /etc/apt/apt.conf.d/99elasticluster <<__EOF
    APT::Periodic::Enable "0";
    // undo what's in 20auto-upgrade
    APT::Periodic::Update-Package-Lists "0";
    APT::Periodic::Unattended-Upgrade "0";
    __EOF

However, despite APT::Periodic::Enable having value 0 from the command line (see below), the unattended-upgrades program is still run...

    ubuntu@test:~$ apt-config shell AutoAptEnable APT::Periodic::Enable
    AutoAptEnable='0'

3. Remove /usr/lib/apt/apt.systemd.daily altogether

The following cloud-init script removes the unattended upgrades script altogether::

    #!/bin/bash

    mv /usr/lib/apt/apt.systemd.daily /usr/lib/apt/apt.systemd.daily.DISABLED

Still, the task runs and I can see it in the process table! although the file does not exist if probed from the command line::

ubuntu@test:~$ ls /usr/lib/apt/apt.systemd.daily
ls: cannot access '/usr/lib/apt/apt.systemd.daily': No such file or directory

It looks as though the cloud-init script (together with the SSH command-line) and the root systemd process execute in separate filesystems and process spaces...

Questions

Is there something obvious I am missing? Or is there some namespace magic going on which I am not aware of?

Most importantly: how can I disable the apt-daily.service through a cloud-init script?

Answer 1

Yes, there was something obvious that I was missing.

Systemd is all about concurrent start of services, so the cloud-init script is run at the same time the apt-daily.service is triggered. By the time cloud-init gets to execute the user-specified payload, apt-get update is already running. So the attempts 2. and 3. failed not because of some namespace magic, but because they altered the system too late for apt.systemd.daily to pick the changes up.

This also means that there is basically no way of preventing apt.systemd.daily from running -- one can only kill it after it's started.

This "user data" script takes this route::

#!/bin/bash

systemctl stop apt-daily.service
systemctl kill --kill-who=all apt-daily.service

# wait until `apt-get updated` has been killed
while ! (systemctl list-units --all apt-daily.service | fgrep -q dead)
do
  sleep 1;
done

# now proceed with own APT tasks
apt install -y python

There is still a time window during which SSH logins are possible yet apt-get will not run, but I cannot imagine another solution that can works on the stock Ubuntu 16.04 cloud image.

Answer 2

Woudn't ist be easier to mask the unit

systemctl mask apt-daily.service

?