49

The Ubuntu 16.04 server VM image apparently starts the "apt-daily.service" every 12 hours or so; this service performs various APT-related tasks like refreshing the list of available packages, performing unattended upgrades if needed, etc.

When starting from a VM "snapshot", the service is triggered immediately, as (I presume) systemd realizes quickly that the timer should have gone off long ago.

However, a running APT prevents other apt processes from running as it holds a lock on /var/lib/dpkg. The error message indicating this looks like this:

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

I need to disable this automated APT task until Ansible has completed the machine setup (which typically involves installing packages); see https://github.com/gc3-uzh-ch/elasticluster/issues/304 for more info and context.

I have tried various options to disable the "unattended upgrades" feature through a "user data" script for cloud-init, but all of them have failed so far.

1. Disable the systemd task

systemd task apt-daily.service is triggered by apt-daily.timer. I have tried to disable one or the other, or both, with various cobinations of the following commands; still, the apt-daily.service is started moments after the VM becomes ready to accept SSH connections::

    #!/bin/bash

    systemctl stop apt-daily.timer
    systemctl disable apt-daily.timer
    systemctl mask apt-daily.service
    systemctl daemon-reload

2. Disable config option APT::Periodic::Enable

Script /usr/lib/apt/apt.systemd.daily reads a few APT configuration variables; the setting APT::Periodic::Enable disables the functionality altogether (lines 331--337). I have tried disabling it with the following script::

    #!/bin/bash

    # cannot use /etc/apt/apt.conf.d/10periodic as suggested in
    # /usr/lib/apt/apt.systemd.daily, as Ubuntu distributes the
    # unattended upgrades stuff with priority 20 and 50 ...
    # so override everything with a 99xxx file
    cat > /etc/apt/apt.conf.d/99elasticluster <<__EOF
    APT::Periodic::Enable "0";
    // undo what's in 20auto-upgrade
    APT::Periodic::Update-Package-Lists "0";
    APT::Periodic::Unattended-Upgrade "0";
    __EOF

However, despite APT::Periodic::Enable having value 0 from the command line (see below), the unattended-upgrades program is still run...

    ubuntu@test:~$ apt-config shell AutoAptEnable APT::Periodic::Enable
    AutoAptEnable='0'

3. Remove /usr/lib/apt/apt.systemd.daily altogether

The following cloud-init script removes the unattended upgrades script altogether::

    #!/bin/bash

    mv /usr/lib/apt/apt.systemd.daily /usr/lib/apt/apt.systemd.daily.DISABLED

Still, the task runs and I can see it in the process table! although the file does not exist if probed from the command line::

ubuntu@test:~$ ls /usr/lib/apt/apt.systemd.daily
ls: cannot access '/usr/lib/apt/apt.systemd.daily': No such file or directory

It looks as though the cloud-init script (together with the SSH command-line) and the root systemd process execute in separate filesystems and process spaces...

Questions

Is there something obvious I am missing? Or is there some namespace magic going on which I am not aware of?

Most importantly: how can I disable the apt-daily.service through a cloud-init script?

  • 2
    This isn't going to help you until it gets rolled into an official package update, but please see the patch I just posted to Debian bug #844453. – zwol Mar 27 '17 at 14:18
  • Maybe you were missing the --now flag in the systemctl disable command in order to make the change effective immediately. That was my issue. – Daniel F Aug 19 '18 at 15:47
  • @DanielF no, because disable --now is equivalent to stop followed by disable. – sourcejedi Aug 19 '18 at 16:02
34

Yes, there was something obvious that I was missing.

Systemd is all about concurrent start of services, so the cloud-init script is run at the same time the apt-daily.service is triggered. By the time cloud-init gets to execute the user-specified payload, apt-get update is already running. So the attempts 2. and 3. failed not because of some namespace magic, but because they altered the system too late for apt.systemd.daily to pick the changes up.

This also means that there is basically no way of preventing apt.systemd.daily from running -- one can only kill it after it's started.

This "user data" script takes this route::

#!/bin/bash

systemctl stop apt-daily.service
systemctl kill --kill-who=all apt-daily.service

# wait until `apt-get updated` has been killed
while ! (systemctl list-units --all apt-daily.service | egrep -q '(dead|failed)')
do
  sleep 1;
done

# now proceed with own APT tasks
apt install -y python

There is still a time window during which SSH logins are possible yet apt-get will not run, but I cannot imagine another solution that can works on the stock Ubuntu 16.04 cloud image.

  • this worked for me on aws ubuntu 16.04, thanks for the solution – krisdigitx Aug 2 '17 at 16:03
  • Yes, I am going the road of creating a custom AMI. This also speeds up the installation of common services. – giorgiosironi Dec 15 '17 at 14:53
  • This doesn't seem to be enough, I find there are still straggling instances of apt-get -o Acquire::http::AllowRedirect=false update – Edward Z. Yang Apr 2 at 18:19
8

Note: Unfortunately part of the solution below doesn't work on Ubuntu 16.04 systems (such as that of the questioner) because the suggested systemd-run invocation only works on Ubuntu 18.04 and above (see the comments for details). I'll leave the answer here because this question is still a popular hit regardless of which Ubuntu version you are using...

On Ubuntu 18.04 (and up) there may be up to two services involved in boot time apt updating/upgrading. The first apt-daily.service refreshes the list of packages. However there can be a second apt-daily-upgrade.service which actually installs security critical packages. An answer to the "Terminate and disable/remove unattended upgrade before command returns" question gives an excellent example of how to wait for both of these to finish (copied here for convenience):

systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true

(note this has to be run as root). If you are trying to disable these services on future boots you will need to mask BOTH services:

systemctl mask apt-daily.service apt-daily-upgrade.service

Alternatively you can systemctl disable both services AND their associated timers (i.e. apt-daily.timer and apt-daily-upgrade.timer).

Note the masking/disabling techniques in this answer only prevent the update/upgrade on future boots - they won't stop them if they are already running in the current boot.

  • 2
    Excellent answer, thank you! Although, note that systemd-run on Ubuntu 16.04 is too old to support the --wait option, but it should not really be necessary for the purpose at hand. (According to the man page, --wait waits for the termination of a unit but it's enough to wait for its start which is the default behavior of systemd-run.) – Riccardo Murri Nov 18 '18 at 5:43
  • I stand corrected: the given systemd-run incantation does not work on Ubuntu 16.04 at all; it dies with error message Unknown assignment After=apt-daily.service apt-daily-upgrade.service. It looks like some unit properties were not available in systemd-run, see for example here – Riccardo Murri Nov 18 '18 at 6:28
  • @riccardo-murri you got me :-)! I was actually wondering about 16.04/18.04 differences myself (hence the weaselly "up to two") and then forgot to put the caveat in. What change would you suggest? – Anon Nov 18 '18 at 6:28
  • @riccardo-murri ah that's too bad I'll add a big warning to the top of the answer saying it can't be used on Ubuntu 16.04 – Anon Nov 18 '18 at 6:30
  • Disabled the services and restarted and it works! – digz6666 Dec 18 '18 at 5:14
3

You can disable this via the "bootcmd" cloud-init module. This runs before network is brought up, which is required before apt update can get a chance to run.

#cloud-config
bootcmd:
    - echo 'APT::Periodic::Enable "0";' > /etc/apt/apt.conf.d/10cloudinit-disable
    - apt-get -y purge update-notifier-common ubuntu-release-upgrader-core landscape-common unattended-upgrades
    - echo "Removed APT and Ubuntu 18.04 garbage early" | systemd-cat

Once you ssh into the instance, you should also wait for the final phases of cloud-init to finish, since it moves apt sources / lists around.

# Wait for cloud-init to finish moving apt sources.list around... 
# a good source of random failures
# Note this is NOT a replacement for also disabling apt updates via bootcmd
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do
    echo 'Waiting for cloud-init to finish...'
    sleep 3
done

This is also helpful to see how early the bootcmd runs:

# Show microseconds in systemd journal
journalctl -r -o short-precise

You can verify this worked as follows:

apt-config dump | grep Periodic

# Verify nothing was updated until we run apt update ourselves.
cd /var/lib/apt/lists
sudo du -sh .   # small size
ls -ltr         # old timestamps
1

Woudn't ist be easier to mask the unit

systemctl mask apt-daily.service

?

  • Doesn't work -- see section 1. Disable the systemd task in the question's text. But thanks anyway for the suggestion! :-) – Riccardo Murri Oct 10 '16 at 20:48
  • 2
    disable and mask a service is not the same. mask create a Link to /dev/null. ls -al /etc/systemd/system/ | grep alsa lrwxrwxrwx 1 root root 9 Sep 1 13:17 alsa-init.service -> /dev/null the Data is empty. – user192526 Oct 10 '16 at 21:23
  • 2
    i get rid unattended upgrade sudo dpkg-reconfigure -plow unattended-upgrades and forbit it. So the status of unit apt-daily.service is dead. – user192526 Oct 11 '16 at 11:56
  • Hi @Bahamut thanks for your efforts! The question, however, is how to disable apt-daily.service from a cloud-init script and before it starts after VM reboot: this means: (1) it must be done non-interactively, (2) it must be done before apt-daily.service fires for the first time. (If my understanding of systemd is correct, (2) cannot actually be accomplished as cloud-init and apt-daily run concurrently -- see my own reply for more.) – Riccardo Murri Oct 11 '16 at 14:14
  • 1
    I tried this on a normal physical machine (i.e. not a VM) and can confirm it doesn't work. You need to at also stop the timer: systemctl stop apt-daily.timer; systemctl disable apt-daily.timer – happyskeptic Aug 6 '17 at 7:01
0

This waits for 1sec in a whil loop and checks if the lock is released.

while : ; do
                sleep 1
                echo $( ps aux | grep -c lock_is_held ) processes are using apt.
                ps aux | grep -i apt
                [[ $( ps aux | grep -c lock_is_held ) > 2 ]] || break
        done
        echo Apt released

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.