1

I have an app registered as a service in Ubuntu 16.04

If I type:

journalctl -u myapp.service

I can see the logs for my app.

I am moving my app to a new VM where the same service will be in place. Is it possible to migrate the log files to the new VM so that if I type journalctl -u myapp.service it will show all my old logs and any new logs seamlessly?

I have tried to swap in the contents of the old /var/log/journal directory into the new VM, and restarting the systemd-journald service, but it doesn't seem to work.

More Details:

  • The logs are stored in /var/log/journal/< machine-id >/
  • the directory contents look like this:

$ ls /var/log/journal/05b6b1e76c6040cc99b4d34977a98eca/ system@0005b3c2d13e0c14-f5dafb4f81b14546.journal~ system@0005bccfbf5ba0a3-489b00efc9586207.journal~ system@0005bd32e69bb664-572fb5c4ae0871b3.journal~ system@0005c4ce02fb0068-2e040749ddd4a06f.journal~ system@9b08b416ae4c47a78c24b4ed77c39ea2-0000000000000001-0005b3c2d1380bae.journal system@9b08b416ae4c47a78c24b4ed77c39ea2-0000000000000248-0005bccbf3f7d7c1.journal system@e5c655526bb54aa886764039cd37f897-0000000000000001-0005c4ce02f66caf.journal system.journal user-1000@0005b3c2d1bad476-9ffb04f7ed320462.journal~ user-1000@0005bccdb3bf35c7-f73e5bfc47f269c9.journal~ user-1000@0005bd32e724853a-f40e7f8131217c05.journal~ user-1000@0005c4cf22d203b5-66cc27013c97695d.journal~ user-1000@7b4df282ccfe4816a30db088f2621493-00000000000000ab-0005b3c2d1be9c3b.journal user-1000.journal

  • There are no files in /run/log/
  • both old and new VMs share the same service, service-running user, and machine-id (this is because ultimately both VMs stem from (were copied from) the same VM just at different times and with minor software/configuration settings)
  • the user I access the logs with 'ubuntu' has the same groups in both VMs:

$ groups

ubuntu adm dialout cdrom floppy sudo audio dip video plugdev netdev lxd

  • journal version on both VMs is:

$ journalctl --version

systemd 229

+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

  • I am not looking to hold the logs longer than what is configured in the journal settings

Whats Working

  • replacing the contents of the journal folder entirely
  • adjusting file ownership to root:systemd-journal as they were originally
  • 'root' can now see the full log
  • 'ubuntu' can only see some 'current' logs, which are confusing, as they are not what was copied over
4
  • Hi fei0x and welcome to unix.stackexchange.com ! The first question that comes to mind is: why would you want to migrate log files from one system to another ? (...in particular if the log's contents are the result of a specific host's journald doing its job.) The question is real. I am left wondering ... whether this could be an XY problem.
    – Cbhihe
    Jun 22 at 20:36
  • Most of the log files get rotated out over time, anyhow. If you want to save something long-term, you should copy it to a separate text file.
    – C. M.
    Jun 23 at 1:47
  • turns out the file owners were not set properly after copying. I believe what I did works after setting the file owner to match the source.
    – fei0x
    Jun 23 at 13:06
  • hmm... actually turns out i can only see all the logs as root when I do this.... the ubuntu user can only see the new logs.. more to figure out
    – fei0x
    Jun 23 at 16:10
1

All entries for which you will find a relation using journalctl -u <myapp.service> correspond to the contents of the systemd journal for a given host. HOWEVER (from man journalctl):

Output is interleaved from all accessible journal files, whether they are rotated or currently being written, and regardless of whether they belong to the system itself or are accessible user journals.

The issue may boil down to the fact that on the "new VM" (a remote PaaS for instance, or even a VM on a local host) your $USER on that VM may not be granted access to anything but his/her own, private journals. For any other journals (provided your migrated log files are located in the right place) your $USER will need to be part of a few special authorized groups or to be root.

That would apply to any imported journal files since those would not be recognized as your $USER's private journals on the new host. Special groups include systemd-journal, adm, and wheel. Ubuntu may include others.

To access your migrated files you should start making sure that:

  • you migrated your journals to the single central journal in /{run,var}/log/journal/. I'm not certain which of /run or /var begins the path on your Ubuntu VM; you will have to see that by yourself. In any case see journald.conf(5), by issuing $ man 5 journald.conf in a terminal. You will find that your files may be at:

    /run/log/journal/machine-id/*.journal,
    /run/log/journal/machine-id/*.journal~,
    /var/log/journal/machine-id/*.journal,
    /var/log/journal/machine-id/*.journal~
    

From the Ubuntu man page:

systemd-journald writes entries to files in either /run/log/journal/machine-id/ or /var/log/journal/machine-id/ with the ".journal" suffix. If the daemon is stopped uncleanly, or if the files are found to be corrupted, they are renamed using the ".journal~" suffix, and systemd-journald starts writing to a new file.
[...]
/run/... is used when /var/log/journal is not available, or when Storage=volatile is set in the VM's /etc/systemd/journald.conf configuration file.

  • your VM's $USER is member of the wheel group (for instance).

Other recommendations and steps may apply but I do not have enough info to run on to proceed. Please report in a comment, so I may complete this answer if need be.

7
  • Thanks, you got me thinking about permissions. and I realized that the files after being copied were not owned by the same user as they were previously. they were all ubuntu:ubuntu ... but after changing them to root:systemd-journal it seems to have worked. If you want to update your answer to include checking this I will mark yours the answer.
    – fei0x
    Jun 23 at 13:03
  • hmm... actually turns out i can only see all the logs as root when I do this.... the ubuntu user can only see the new logs.. more to figure out
    – fei0x
    Jun 23 at 16:10
  • I've added some more details to my post.
    – fei0x
    Jun 23 at 16:24
  • sudo usermod -a -G systemd-journal ubuntu seemed to work... not great, but I think I can live with this.
    – fei0x
    Jun 23 at 16:47
  • @fei0x: This is actually one of the things I suggested: to add your $USER to one of the restricted groups that have default access to the systemd journal: groups root, wheel and systemd-journal for the systemddaemon each do that. Good you solved it in any case, but I am still stumped as to why you would ever want to migrate systemd logs from one host to the next. I just cannot imagine the use case for that, unless you just mean to keep raw data around for statistical purposes. I still think this might be an XY problem. Good luck.
    – Cbhihe
    Jun 23 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.