Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How do I read or seek from a file that's bigger than the maximum off64_t? The problem arises because the address space of a process is represented in the /proc/.../as file, which is a huge sparse file for 64-bit processes. It's really very big: on a sample process on Solaris x86-64, argv's address is 0xFFFFFD7FFFxxxxxx, that is, the very top of the address space is used. Pointers are unsigned, but an off64_t is signed, so can't reach anything in the top half of the address space file.

This obviously depends on the layout of the address space. On 32-bit systems, this isn't a problem (a long offset isn't big enough, but an off64_t easily works), and on x86-64 on linux (for example), the top of the process is 0x7fffxxxxxxxx (48 bits), so again an off64_t can refer to anything in the process's address space.

So, it's a shame that Solaris on x86-64 seems to use the whole 64-bit address space, when 50 bits is more than enough. Sun's examples of using psinfo_t.pr_argv just don't seem to work except on SPARC and x86. Is there any way around this problem?

share|improve this question
    
Can't you fseek from SEEK_END with a negative offset? –  Mat Oct 29 '12 at 12:22
    
I wish. It's not quite so straightforward. If we were to try and seek backwards from the end, we'd have to know the filesize. stat64 and ftello64 give me some slightly funny results; it's perhaps not quite as simple as pretending that the off64_t they give is infact a ulonglong. Example size output from fstat64 on the address space file is 0xFFFFFD7FFF1F28FA, but pr_argv is 0xFFFFFD7FFFDFFC98 (bigger!). (pr_argv is definitely correct: it matches printf("%p", argv) from within the process. –  Nicholas Wilson Oct 29 '12 at 15:05
1  
It's a bit odd. ftello64 has funny behaviour on very large files (bigger than an off64_t). If I seek to the end, I get no error from fseeko64, but ftello64 gives me 4726784 as the location, an honest positive number, not wrapped. Assuming I can use wizardly logic (eg inspect for negative fstat size) to "correct" it, it's not clear even what it's saying about the size of the file. –  Nicholas Wilson Oct 29 '12 at 17:31
1  
Thanks, lseek returns much more sensible values than fseek. offset_t seems to be a holdover from an older implementation of LFS, but lseek64(-500,SEEK_SET) does seem to seek to 2**64-500. tail and dd can't handle offsets bigger than 2**63, as far as I could tell from playing around. –  Nicholas Wilson Oct 30 '12 at 10:48
1  
So, from fixr it seems Solaris uses u_offset_t for filesizes internally (vattr), but stores the file_t offset in an offset_t which it allows to overflow without any checks. It seems to work, but I'd certainly accept any answer that can reassure me it's OK to do this... What if there are other things that might go wrong if the absolute offset is negative? –  Nicholas Wilson Oct 30 '12 at 10:59

1 Answer 1

up vote 0 down vote accepted

There are on Solaris x86-64 some very large files, whose size exceeds 263, that is, the maximum size representable in an off64_t. This includes the file representing a process's address space in proc (/proc/<pid>/as).

To deal with these files:

  1. Don't use fopen, fseek, etc. Don't trust the libc stream routines, which (on the versions of Solaris I tested) mangle badly the "illegal" offsets.
  2. Use open64, read.
  3. To seek:

    static off64_t lseeku64(int file, uint64_t offset /* eg from pr_argv */)
    {
    #ifndef __sun
      if (offset > 0x7FFFFFFFFFFFFFFFllu) return -1;
    #endif
      return lseek64(file, offset, SEEK_SET);
    }
    

    That is, on Solaris, we know that we can do this cast because of inspection of the OpenSolaris sources, but we should avoid assuming it works on other platforms with psinfo and pr_argv (eg AIX).

    But, pass in your very large offset, and it does all "just work".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.