8

I have read some question, that ask advice how to rsync sparse files efficiently mentioning the files /var/log/lastlog and /var/log/faillog. Indeed I myself have stumpled over those files being an "issue" as their being backup via rsync turns them to become "unsparse".

What I hence wonder is, what is the need/backgrounding motivation to have those files as sparse, huge files (in my case it was 1.1TB)?

Also in relationship to this a follow up: Since I was assuming them to be logfiles I do not care about excesively I truncated those files, did I corrupt anything with truncating those files ?

  • configure logrotate – user1133275 Jul 14 '19 at 0:25
  • 3
    It's a sparse file since the user-id's are used for indexing into it, and those can be large values, e.g., when using Samba. – Thomas Dickey Jul 14 '19 at 20:56
  • @ThomasDickey according to your comment, the answer by waitinator that originally hints that truncating the file is the original problem would not ring truth, right? as soon as there is a "strangly" large uid the file will become apparently huge (though taking few bytes being "sparse"), correct? – humanityANDpeace Jul 15 '19 at 10:01
9

What I hence wonder is, what is the need/backgrounding motivation to have those files as sparse, huge files (in my case it was 1.1TB)?

This is how it's supposed to be.

/var/log/lastlog is not a log file like /var/log/syslog, and its name should be read as "last logins list" rather than "last logfile".

It's maintained by the pam_lastlog(8) module, and it's basically an array like this:

struct lastlog {
    time_t  ll_time;    // 4
    char    ll_line[UT_LINESIZE];   // 32
    char    ll_host[UT_HOSTSIZE];   // 256
} entry[UINT_MAX];

Sizes of the fields on a typical x86-64 machine are in comments; an entry should be 4 + 32 + 256 = 292 bytes.

Every time a program using the pam_lastlog(8) pam module is logging a user in, it will seek to uid * sizeof(struct lastlog) and overwrite the entry corresponding to that user.

did I corrupt anything with truncating those files ?

You did corrupt the output of the lastlog(1) command, which nobody is using anyway ;-)

  • I am happy I refrained from accepting an earlier answer, that mentioned the reason for huge apparent file size being truncating of the file. Your answer seems to provide better, more correct advice, well done! – humanityANDpeace Jul 15 '19 at 10:07
  • 1
    Couriousity has me here, this concept to use sparse files and an then using a arraykey /offset to map into the correct small peace of information (leading to sparse files) is something that I only ever encountered with /var/log/lastlog and /var/log/faillog, making me feel prompted to ask here. Is this a common technique, i.e. something other software in an normal unix/linux setup do also? – humanityANDpeace Jul 15 '19 at 10:11
  • No, most common software don't do that, because that doesn't work on windows ;-) – mosvy Jul 15 '19 at 10:16
  • This doesn't answer the why of the question. Why did they chose to use a sparse file here when almost nobody else uses it for this kind of purpose, and hardly anyone expects to have a gigantic sparse file in this log directory? – Edmund Sep 10 '20 at 23:55
  • 1
    @Edmund That's a loaded question. Let's have a different take: why would someone NOT use sparse files and instead of it re-invent the wheel (or use some heavy-weight general purpose database) if they're sure that their software will never have to work on broken/primitive OSs without support for sparse files? – mosvy Sep 11 '20 at 16:52
1

I have seen this behaviour after we have introduced some (large) LDAP UID numbers co-existing with "normal" UNIX UID numbers in a linux (RHEL 7) machine.

Also, in the "CAVEATS" section of the manual page for "lastlog" it states that

  *"Large gaps in UID numbers will cause the lastlog program to run longer with no output to the screen"*

"lastlog" will appear to hang as it processes entries with the intermediate unused UIDs

Maybe this could be also related to the observed problem, as FreeIPA LDAP UID numbers assigned were much larger than the ones from Linux

  • That's one of those things which hadn't aged well. The code was written for much smaller machines, many, many years ago. One of these days I should join the maintenance team. Or re-do a bunch of that code from scratch. – Julie in Austin Sep 23 '20 at 21:21
-1

Those 2 files are "data" type files, if you check using ls -l or ls -lh command then you will see it's occupying very huge amount of disk space, but actually it's not

"ls -s" is the actual command to check the actual size of these files.

[root@LinuxServer ~]# file /var/log/lastlog /var/log/faillog
/var/log/lastlog: data
/var/log/faillog: data
[root@LinuxServer ~]# ls -l /var/log/lastlog /var/log/faillog
-rw------- 1 root osg       3166464 Jul  8 21:20 /var/log/faillog
-rw-r--r-- 1 root root 304854077980 Jul 14 21:38 /var/log/lastlog
[root@LinuxServer ~]# ls -lh /var/log/lastlog /var/log/faillog
-rw------- 1 root osg  3.1M Jul  8 21:20 /var/log/faillog
-rw-r--r-- 1 root root 284G Jul 14 21:38 /var/log/lastlog
[root@LinuxServer ~]#

The actual size is only few KB's, if you check with the "ls -s" command.

[root@LinuxServer ~]# ls -s /var/log/lastlog /var/log/faillog
3100 /var/log/faillog   748 /var/log/lastlog
[root@LinuxServer ~]# ls -s /var/log/faillog
3100 /var/log/faillog
[root@LinuxServer ~]#
  • 2
    An apparent size larger than the amount of data stored is the very definition of a sparse file. – ilkkachu Jul 14 '19 at 21:27
  • thank you! I understand you are quite new to this stackexchange community. Most likely you meant very well providing some insight to this issue, but as @ilkkachu suggest the very question mentions the fact that we deal with sparse files hence I knew that they actually use less space on disk. Do not be too discouraged by the downvotes. – humanityANDpeace Jul 15 '19 at 10:05
-1

By "truncating the file", without informing (or restarting) the programs that are writing to the log file, YOU have caused the sparse file.

The best fix is to find and fix the problem causing all those log messages.

Another way to avoid the problem is Stop the logging service, use lsof to ensure no other processes have the file open, then truncate the file, and restart logging.

As requested, here's a model of how it (doesn't) work:

A program opens the file for "append", is told that the file ends at block X, and writes block X+1. You "truncate the file", but the program "knows" the next block to write is X+2. Thus, the file will might gave data at the beginning, but definitely has data at block X+2. Ta-Da! Sparse file.

  • So then I must have gotten myself into an endless loop, since my cure for when those files have grown excessively huge has since been to truncate them back to size (which as if I read correctly your answer is supposed to be the original cause of the huge size in and on itself?). I am puzzled how this however reaches sizes up to 1TB since, I would supposed imparative reboots for kernel updates would have interrupted any logging processes anyway. I want to believe you, but I am struggling to really grasp the particularities of this, i..e. how a file becomes sparse + huge, can you enlight me? – humanityANDpeace Jul 14 '19 at 8:10
  • How do you know the files ended up being sparse because of truncating without restarting the programs? Give some reasoning for that claim. – ilkkachu Jul 14 '19 at 21:30
  • Actually, writing in append mode doesn't create a sparse file like that, the writes go to the end of the file, where ever it is at that point. (A write in normal mode would work as you described though.) But anyway, you've described how a file becomes sparse if it's truncated while being open, but where does it say that they truncated the file without restarting the relevant processes? Also, given that the man page for lastlog explicitly says that the file is sparse, are you sure it's used like a regular log file to begin with? – ilkkachu Jul 14 '19 at 22:03
  • 1
    lastlog has nothing to do with log files. Your answer is completely off. Try man lastlog and man pam_lastlog. – mosvy Jul 14 '19 at 23:25
  • @ilkkachu lastlog is not a log file like /var/log/messages. It's normal to be a big sparse binary file, since it's basically an array of structs indexed by user id, and the user ids are not contiguous. – mosvy Jul 14 '19 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.