1

I am using CloudLab to create an openstack setup, and the filesystem structure on the controller node looks like this:

root@ctl:/users/Jasim9# df -h
Filesystem                                      Size  Used Avail Use% Mounted on
udev                                            126G     0  126G   0% /dev
tmpfs                                            26G   58M   26G   1% /run
/dev/sda1                                        16G  4.8G   11G  32% /
tmpfs                                           126G  4.0K  126G   1% /dev/shm
tmpfs                                           5.0M     0  5.0M   0% /run/lock
tmpfs                                           126G     0  126G   0% /sys/fs/cgroup
ops.clemson.cloudlab.us:/proj/live-migrate-PG0  100G  2.7G   98G   3% /proj/live-migrate-PG0
ops.clemson.cloudlab.us:/share                   97G  1.2G   88G   2% /share
/dev/mapper/openstack--volumes-swiftv1          4.0G   33M  4.0G   1% /storage/mnt/swift/swiftv1
/dev/mapper/openstack--volumes-swiftv1--2       4.0G   33M  4.0G   1% /storage/mnt/swift/swiftv1-2
tmpfs                                            26G     0   26G   0% /run/user/0

The problem is that the root disk is only 16GB and with openstack logs it is filling up fast so I need to find a way to increase the space for /var/log directory. I tried the following approach but it does not seem to work because it causes some services to stop working.

sudo lvcreate -L 200G -n logs-lv openstack-volumes
sudo mkfs.ext3 /dev/openstack-volumes/logs-lv
sudo mkdir /mnt/tmplogs
sudo mount -t ext3 /dev/openstack-volumes/logs-lv /mnt/tmplogs/
df -h
sudo mv /var/log/* /mnt/tmplogs/
sudo umount /dev/openstack-volumes/logs-lv
sudo mount -t ext3 /dev/openstack-volumes/logs-lv /var/log/

I found this link and thought of using it but it suggests stopping all services using /var/log but I have a bunch of services running when I run that command lsof +D /var/log | awk '!/COMMAND/{print $1 | "sort -u"}' (about 40) and I did not want to mess with them.

Is there a smoother way to increase the /var/log capacity using some trick like mount --bind or moving mount points or something? I only have ssh access to the machine.

/dev/sda1 is not an LVM partition. and /dev/sda4 is being used for LVM.

Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x90909090

Device     Boot    Start        End    Sectors   Size Id Type
/dev/sda1  *        2048   33556479   33554432    16G 83 Linux
/dev/sda2       33556480   39847935    6291456     3G  0 Empty
/dev/sda3       39847936   46139391    6291456     3G 82 Linux swap / Solaris
/dev/sda4       46139392 1953525167 1907385776 909.5G  0 Empty
  • Symlink /var/log to a partition with more space. See man ln for information on how to use it. – Mioriin Nov 8 '17 at 6:02
  • @Mioriin Typically logging already starts before all partitions are mounted. I'm not sure how to deal with that. – Philippos Nov 8 '17 at 6:50
  • The system keeps logs in memory until /var/log becomes available, then flushes the buffer. If the partition that houses the logs is mounted at startup, you shouldn't have problems with it. (some systems have the entire /var directory on a separate drive to keep the logs since forever) – Mioriin Nov 8 '17 at 6:56
  • I just thought of something; you already created an LV or partition for the logs, yes? Then simply add it to fstab as mount point for /var or /var/log, stop all the logging services, move the existing logs, then reboot the server. Then go see what it is that's filling the logs so fast. – Mioriin Nov 8 '17 at 7:11
  • I could try that, but when I tried mounting and reboot previously, the services would just stop working and would not start back up for some reason. – Wajahat Nov 8 '17 at 7:56
1

Are you sure that "more space" is the best answer to this problem? I don't know the openstack logs, but typically logs can be drastically reduced in size by compression, so I'd suggest to configure your /etc/logrotate.d/openstack like this:

/var/log/openstack.log {
  rotate 10
  hourly
  compress
  delaycompress
  minsize 1G
  maxsize 1200M
}

This example would keep you around 10 GB of logs, but with eight of them compressed with a ration of maybe 1:10, less than 3 GB are actually used.

If you can't adapt it to your needs, you could add a postrotate script to move old logs to another volume.

  • I guess this will help, but I still think that 11G that is left on the root drive might run out eventually. I just wanted to know if there is a way to accomplish what I was trying without disrupting services. – Wajahat Nov 8 '17 at 7:54
  • Of course you can adapt the suggested numbers according to how much logs you really need and how much space to are willing to reserve for the logs. If it doesn't matter if the log is split into many small files, so you need less space for the uncompressed logs. You can manually test the compression ratio of those logs to see how much space you really could save by compression. – Philippos Nov 8 '17 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.