wget downloaded file names with absolute path for use with apt-get

Question

I'm looking for a way of updating a machine that is offline. For that purpose I found this explanation using --print-uris from apt-get. I have successfully installed new packages (and dependencies) with it. Very handy.

However, when I tried to update the package list using the method described in the previous link, I ended up with a bunch of files all named Packages.bz2.*, where the * takes values from 1 to 23. As far I understand I have to extract them and the resulting file copy it to /var/lib/apt/lists/.

Is there a way to download the files with wget but instead of all called Packages.bz2 with names similar (or preferably equal) to the ones originally used by apt-get archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages, so when extracting them they stay with readable names.

After doing that, is there anything more that I have to do in order to apt-get find updates (I intent to do an upgrade printing uris and then installing the packages).

---

An execution example:

sudo apt-get update --print-uris -y > update.list

This gives a file containing lines like:

'http://security.ubuntu.com/ubuntu/dists/trusty-security/main/binary-amd64/Packages.bz2' security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages 0 :
'http://security.ubuntu.com/ubuntu/dists/trusty-security/restricted/binary-amd64/Packages.bz2' security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages 0 :

This format can't be fed to wget, so we extract the urls by executing (this can be done directly in the previous command as shown in the linked page):

cat update.list | grep ^\' | cut -d\' -f2 > update.cut

Then we get a file with lines like:

http://security.ubuntu.com/ubuntu/dists/trusty-security/main/binary-amd64/Packages.bz2
http://security.ubuntu.com/ubuntu/dists/trusty-security/restricted/binary-amd64/Packages.bz2

Now this can be feed into wget using the option --input-file. I executed:

mkdir lists
cd lists
wget --input-file ../update.cut

An ls shows:

Packages.bz2     Packages.bz2.14  Packages.bz2.2   Packages.bz2.4  Translation-en.bz2     Translation-en.bz2.4
Packages.bz2.1   Packages.bz2.15  Packages.bz2.20  Packages.bz2.5  Translation-en.bz2.1   Translation-en.bz2.5
Packages.bz2.10  Packages.bz2.16  Packages.bz2.21  Packages.bz2.6  Translation-en.bz2.10  Translation-en.bz2.6
Packages.bz2.11  Packages.bz2.17  Packages.bz2.22  Packages.bz2.7  Translation-en.bz2.11  Translation-en.bz2.7
Packages.bz2.12  Packages.bz2.18  Packages.bz2.23  Packages.bz2.8  Translation-en.bz2.2   Translation-en.bz2.8
Packages.bz2.13  Packages.bz2.19  Packages.bz2.3   Packages.bz2.9  Translation-en.bz2.3   Translation-en.bz2.9

After this point the site I linked gives no more clues other than this should be put in /var/lib/apt/lists/. For that I do (after I copied the files to the offline machine):

sudo rm -R /var/lib/apt/lists/*
bunzip2 Packages.bz2.14
sudo cp Packages.bz2.14.out /var/lib/apt/lists/

But after doing that I can't search packages with apt-cache so I suppose that something went wrong ( I checked that the file Packages.bz2.14.out lists a package name 0ad and searched for that).

For now I'm trying to get wget downloaded files with human readable names (I think some should be gpg files but don't know which ones because of the name).

After that I want to be able to do packages searches with the new package list (in order to do upgrades and software installations). I'm not sure what happens if the gpg files are missing (maybe that's why I can't find packages) or should y check the gpg before copying it to /var/lib/apt/lists/.

Answer 1

The simple approach

If you are using another machine, one which is connected to the internet, to upgrade your target computer, you can simply use the same sources.list file on the two machines, run apt-get update and then copy the package lists over:

$ sudo mv /etc/apt/sources.list /etc/apt/sources.list.orig
$ sudo mv target_computer_sources.list /etc/apt/sources.list
$ sudo apt-get update

apt stores its repository file lists in /var/lib/apt/lists/, so afer running the command above, you cam copy everything in /var/lib/apt/lists/ to the target machine and then revert to the original sources.list:

$ sudo mv /etc/apt/sources.list.orig /etc/apt/sources.list 

The complex way

I for whatever reason the method above does not work for you, you can do it the way you were suggesting, you just need to parse the output of apt-get update --print-uris. The following are from my LMDE system but the idea is the same:

$ sudo apt-get update --print-uris -y | head
'ftp://ftp.debian.org/debian/dists/testing/main/source/Sources.bz2' ftp.debian.org_debian_dists_testing_main_source_Sources 0 :
'ftp://ftp.debian.org/debian/dists/testing/contrib/source/Sources.bz2' ftp.debian.org_debian_dists_testing_contrib_source_Sources 0 :
'ftp://ftp.debian.org/debian/dists/testing/non-free/source/Sources.bz2' ftp.debian.org_debian_dists_testing_non-free_source_Sources 0 :
'ftp://ftp.debian.org/debian/dists/testing/InRelease' ftp.debian.org_debian_dists_testing_InRelease 0 
'http://packages.linuxmint.com/dists/debian/main/source/Sources.bz2' packages.linuxmint.com_dists_debian_main_source_Sources 0 :
'http://packages.linuxmint.com/dists/debian/upstream/source/Sources.bz2' packages.linuxmint.com_dists_debian_upstream_source_Sources 0 :
'http://packages.linuxmint.com/dists/debian/import/source/Sources.bz2' packages.linuxmint.com_dists_debian_import_source_Sources 0 :
'http://packages.linuxmint.com/dists/debian/main/binary-amd64/Packages.bz2' packages.linuxmint.com_dists_debian_main_binary-amd64_Packages 0 :
'http://packages.linuxmint.com/dists/debian/upstream/binary-amd64/Packages.bz2' packages.linuxmint.com_dists_debian_upstream_binary-amd64_Packages 0 :
'http://packages.linuxmint.com/dists/debian/import/binary-amd64/Packages.bz2' packages.linuxmint.com_dists_debian_import_binary-amd64_Packages 0 :

In the output above, the 1st field is the URL and the 2nd is the name the file will be saved under. As I mentioned before, apt stores its repository file lists in /var/lib/apt/lists/, you can verify this by running locate on one of the 2nd fields above:

$ locate packages.linuxmint.com_dists_debian_main_source_Sources
/var/lib/apt/lists/packages.linuxmint.com_dists_debian_main_source_Sources

So, if you want to download and update each of those lists, you will need to parse the output of apt-get update --print-uris and tell wget to use the 2nd field as the output name:

$ sudo apt-get update --print-uris -y | sed "s/'//g" | cut -d ' ' -f 1,2 | 
   while read url target; do wget $url -O ./$target; done 

This will download each package list and save it in the current directory using the appropriate name. You can now copy these files over to the /var/lib/apt/lists/ directory of the target machine.

Answer 2

This may not directly apply to what you wanted to do. But since it appears wget is not preserving the path of the files you downloaded, you may want to use the Directory Option -x:

-x
--force-directories
       The opposite of -nd---create a hierarchy of directories, even if
       one would not have been created otherwise.  E.g. wget -x
       http://fly.srk.fer.hr/robots.txt will save the downloaded file to
       fly.srk.fer.hr/robots.txt.