3

I use EC2 on Amazon Web Services. The OS of the t2.micro instance is a customized “Amazon Linux” with 1 GiB RAM and 1 vCPU. When accessing this instance via their Cloud9 IDE I find that by default already 73% of the available file space (7.8G on /dev/xvda1) is occupied, and I can only use the remaining 2.2G.

My requirements:

  • I need to execute a Python script and write output data locally.
  • I can do without GUI since I am working on the command line.

What components of their OS can be safely removed in order to free up some space?

  • 1
    Is running a different distribution an option? I run an Ubuntu Server on EC2 with less than 4.5 GB for the OS. – Philip Couling Mar 19 '19 at 16:15
  • You could a) add an extra empty volume to an instance b) copy existing image to a bigger volume and extend the file system (this is a bit tricky but still possible) c) create your own image with software packages of your choice – Serge Mar 20 '19 at 14:38
  • @PhilipCouling That might be a workaround, but would require me to set up the instance from scratch. Can you confirm that this actually yields you more (remaining) free space? – marianoju Mar 20 '19 at 19:43
  • @PhilipCouling I had to find out by myself and started two t2.micro instances with Amazon Linux and Ubuntu Server 18.04 LTS respectively. OS occupies 5.7G/9.8G (60%) and 6.4G/9.7G (66%), and default available file space on startup is 4.0G (40%) and 3.3G (34%). How come your Ubuntu Server on EC2 weighs less than 4.5G? – marianoju Aug 8 '19 at 12:19
  • @marianoju Similar to the accepted answer. I start with a minimal image, its a while so I can't say which one. Then I strip things out. From a fresh install I interrogate the output of apt-mark showmanual. I mark for possible removal as much as I can: apt-mark auto <module>. Eg: apt-mark auto $(apt-mark showmanual | grep ^lib) sets all libs to auto meaning they will only be kept if something else needs them. Finally I apt-get autoremove to remove anything no longer needed. – Philip Couling Aug 8 '19 at 13:01
2

1. remove dispensable packages

Amazon Linux instances manage their software using the yum package manager. The yum package manager can install, remove, and update software, as well as manage all of the dependencies for each package. – Managing Software on Your Linux Instance

I have executed the following to produce a list of the 20 largest packages in the system:

rpm -qa --queryformat '%10{size} - %-25{name} \t %{version}\n' | sort -nr | head -n 20

To remove packages with all of its dependencies I have then installed the yum plugin remove-with-leaves and then repeatedly removed the largest packages (including dependencies) which I deemed dispensable (see below for list):

sudo yum remove package_name --remove-leaves

2. remove obsolete kernel

  1. Identified current kernel:uname -mrs
  2. Listed all kernels: rpm -q kernel
  3. Manually removed obsolete Linux kernel: sudo yum remove kernel-4.9.76-3.78.amzn1.x86_64

3. remove unused packages

Identified packages that can be removed without affecting anything else (in debian-speak these are called “orphaned packages”) and removed quietly.

sudo package-cleanup --quiet --leaves | sudo xargs -l1 yum -y remove 

Findings

While I am actively only using Python 3.6.5 it is not possible to remove the default python (Python 2.7.14).

Python is required by many of the Linux distributions. Many system utilities the distro providers combine (both GUI based and not), are programmed in Python. The version of python the system utilities are programmed in I will call the "main" python. [...] Because of the system utilities that are written in python it is impossible to remove the main python without breaking the system. – How to yum remove Python gracefully?

Space occupied by python27 packages amounts to 115819035 bytes (~116 MB).

Results

  • A total of ~0.5 GB was reclaimed (7% of disk space on /dev/xvda1).
  • 214 packages with a total of 633427867 bytes were removed:
    java-1.7.0-openjdk emacs-common mysql55-server java-1.7.0-openjdk-devel git 
    mysql55 vim-common perl compat-libicu4 aws-apitools-ec2 emacs v8 ruby20-libs 
    perl-Encode nodejs-devel aws-apitools-elb aws-apitools-as nodejs 
    aws-apitools-mon perl-DBD-SQLite dejavu-sans-fonts subversion subversion-libs 
    subversion-perl python36-devel dejavu-serif-fonts vim-enhanced libtool autoconf 
    perl-DBI rubygem20-rdoc automake libX11-common perl-libs gyp cvs libX11 git-svn 
    alsa-lib gnutls dejavu-sans-mono-fonts perl-Net-SSLeay npm libyaml-devel 
    xorg-x11-fonts-Type1 perl-IO-Compress rsync libxcb libpng perl-Test-Harness 
    rubygems20 perl-Pod-Simple fontconfig aws-amitools-ec2 lcms2 perl-DBD-MySQL55 
    git-cvs xorg-x11-font-utils libXfont perl-podlators perl-IO-Socket-SSL git-p4 
    v8-devel perl-YAML perl-Storable rubygem20-json perl-Git-SVN perl-PathTools 
    nodejs-hawk perl-Pod-Perldoc ruby20-irb perl-File-Temp libuv-devel libserf 
    system-rpm-config autogen-libopts perl-Getopt-Long perl-Compress-Raw-Zlib 
    perl-Filter perl-GSSAPI dejavu-fonts-common libuv perl-Net-Daemon libICE cvsps 
    perl-Socket rubygem20-psych perl-Digest-SHA git-email perl-Authen-SASL ttmkfdir 
    perl-HTTP-Tiny perl-Data-Dumper nodejs-ctype perl-threads emacs-git 
    perl-Time-HiRes perl-IO-Socket-IP libXext giflib rubygem20-bigdecimal libSM 
    nodejs-async perl-threads-shared perl-PlRPC nodejs-hoek node-gyp libXi perl-Git 
    nodejs-request nodejs-fstream perl-Scalar-List-Utils ruby20 nodejs-mime 
    perl-Exporter perl-TermReadKey perl-Compress-Raw-Bzip2 nodejs-tar 
    perl-Digest-MD5 perl-File-Path perl-Error http-parser perl-Net-LibIDN 
    perl-Pod-Usage perl-Time-Local libfontenc libXrender libXau 
    nodejs-npm-registry-client nodejs-minimatch nodejs-boom nodejs-http-signature 
    nodejs-semver libXcomposite nodejs-glob nodejs-nopt perl-Digest perl-Carp 
    libXtst perl-Thread-Queue nodejs-npmconf libffi-devel perl-constant gpm-libs 
    perl-Pod-Escapes nodejs-normalize-package-data nodejs-packaging 
    nodejs-read-package-json nodejs-promzard nodejs-lockfile nodejs-asn1 
    nodejs-ansi perl-Text-ParseWords copy-jdk-configs nodejs-form-data nodejs-sntp 
    nodejs-fstream-npm nodejs-node-uuid nodejs-config-chain perl-Digest-HMAC 
    nodejs-retry nodejs-graceful-fs nodejs-sigmund nodejs-npmlog http-parser-devel 
    nodejs-read-installed nodejs-lru-cache nodejs-init-package-json nodejs-qs 
    nodejs-slide nodejs-combined-stream nodejs-assert-plus nodejs-fstream-ignore 
    nodejs-block-stream perl-parent nodejs-delayed-stream nodejs-ini nodejs-sha 
    nodejs-cmd-shim nodejs-tunnel-agent nodejs-mute-stream nodejs-rimraf 
    nodejs-read nodejs-osenv nodejs-mkdirp perl-macros nodejs-which nodejs-abbrev 
    perl-Net-SMTP-SSL nodejs-archy nodejs-uid-number nodejs-aws-sign 
    nodejs-forever-agent nodejs-opener nodejs-json-stringify-safe nodejs-proto-list 
    nodejs-cryptiles nodejs-editor nodejs-child-process-close 
    nodejs-github-url-from-git nodejs-cookie-jar nodejs-npm-user-validate 
    nodejs-chmodr nodejs-chownr nodejs-once nodejs-inherits nodejs-oauth-sign 
    aws-apitools-common mysql-config vim-filesystem ruby git-all 
    fontpackages-filesystem 
    

Resources

  1. Amazon Linux AMI
  2. GAD3R's answer to how to remove all installed dependent packages while removing a package in centos 7?
  3. How to remove old unused kernels on CentOS Linux
  4. jtoscarson's answer to Remove unused packages
  5. Owen Fraser-Green's answer to How can I remove Orphan Packages in Fedora?
  • 2
    nice work and nice explanation. this seems however too much work for too small gain (unless whole process is automated on vm creation). – Archemar Jul 24 '19 at 8:35
  • Compressing my output remotely at regular intervals turned out to be the solution to my original problem which I failed to state: I was regularly running out of space. My output was highly redundant and could be compressed by >80% using gzip and >95% using xz. Runtime of the compression was negligible in this scenario. – marianoju Aug 8 '19 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.