3

The problem with reading /proc file system is that it is not atomic. I tried using a library (procfs in Python) that wishes to read it, but it has glitches about 5% of the time because the process that I am after is now gone. So, I wish to copy everything in /proc to another location, say /proc_clone/<GUID>/. A GUID can be generated using the uuidgen command.

The naive approach would be trying to use cp -r, but I am getting a whole bunch of 'permission denied' errors and it is taking more than a second. I am only after a subset /proc/<pid> and even then, only the read-only stuff that can be turned into a string (ultimately I wish to build a single JSON file that contains all of this).

I am looking for a script that would do that for me and is also short, simple, very fast, can deal with processes dying mid-flight by skipping them, however, I would like to avoid writing C code - I would much rather stick to bash, existing utilities and maybe Perl / Python. The goal here is just to take a snapshot of a subset of /proc, not produce the JSON file itself.

I have heard opinions that "one should not ever try to copy /proc but just read from it". Well, the non-atomic nature of it means that one has to throw a bunch of try/catch all over the code when trying to do very simple operations. Using a high level language like Python (what I ultimately want to use) to iterate over /proc is a slow method, which involves both IO and CPU time. This increases the risk of seeing a process die on you (I certainly see it often; I have a script that interrogates /proc every minute and I run into plenty of exceptions). I would like to build a library that outputs a single JSON file that contains cpu and memory usage information about processes in a human-readable format (e.g. using seconds and not jiffies, bytes and not pages, have units in addition to values. I am not concerned that it would take some time to create that file out of a directory dump; I just want to make sure that the snapshot is as accurate as possible. So. if I should not just copy /proc over, then which other method should I use?

5
  • 2
    What exactly do you want to do? Copying a complete process will take a lot of time...
    – vonbrand
    Commented Apr 26, 2013 at 23:57
  • @vonbrand, I believe I stated what I want to do. Say I want to copy reliably and fast all of the /proc/*/cmdline and all of the /proc/*/statm and /proc/*/status files. Can you suggest a tool/script that can do that for me?
    – Leonid
    Commented Apr 28, 2013 at 16:41
  • I understand the problem. But why can't you catch the exception where the file being read is gone (and terminate whatever you wanted to do there)?
    – Nils
    Commented Apr 29, 2013 at 13:45
  • @Nils, A) Because I want to minimize the window during which I interact with the /proc file system, B) Initially I was using a nice high-level library for Python called procfs. I would prefer to use that library, except hack it to point to a new root.
    – Leonid
    Commented Apr 29, 2013 at 17:04
  • So you should catch the exception. I looked into some mount-options (mount --make-private) that looked as if you could do a snapshot of a FS-mount, but this does not seem to work.
    – Nils
    Commented Apr 30, 2013 at 6:50

2 Answers 2

4

You need to have a look at the man pages for proc. Look at /proc/[pid]/* entries and pick which files you want.

You do not want to copy everything. As an example you have /proc/[pid]/mem which is all the virtual memory for a process including shared etc. – i.e. size of all your memory. Further you are not able to read it unless the process own mem or it is ptraced and read with higher privileges. (And then only Virtual Size - /proc/pid/status -> VmSize) – and so on.

Or /proc/PID/exe which simply is a link to the executable, /proc/PID/fd/* which holds file descriptors for stdin, stdout, stderr and other files, pipes etc.

There is a lot of other special files as well, and a lot of links (If you were to follow them all you'll never be done). Read the man mentioned above, documentation from kernel.org, etc.


Further the files are not block-device files. They reside in memory as proc is a memory based filesystem. In one way you can call it a window into the kernel arranged for easy access.

Thus this is also a big advantage when it comes to the reads it looks like you want – as the information reside in memory there is no disk I/O and therefore very fast. Off course if you choose to write the data to disk this is lost – but is still much faster then disk to disk.


Look at it as shared memory and try to not duplicate more then necessary. For your task you would probably also look at files at the root level of /proc. Look at the source code for ps, top, free etc. to get a quick introduction to how other tools do it.


Have written a few other answers that might more or less touch some of the aspects. You might find some of it useful:


Edit:

As for comment. You could do something like:

save_to=/some/path

pushd /proc
for d in [0-9]*; do
    cp "$d/cmdline" "$save_to/$d.cmdline"
    cp "$d/status" "$save_to/$d.status"
done
popd

But I would have used some lower level like e.g. C to parse the files as the method above quite possible not would be quick enough for what one want.

If it is a one time copy every hour or so, OK, but not for e.g. seconds or even a few minutes interval.

1
  • so say I want to copy over only /proc/<pid>/cmdline, /proc/<pid>/statm, /proc/<pid?/status reliably and fast. Is there a command-line utility that would help me do that?
    – Leonid
    Commented Apr 28, 2013 at 16:40
0

In a similar vein, I wrote the following script, which snapshots a mockup mimicry of /proc into a local ./proc folder as an alternative to mounting a /proc filesystem when chrooting. (Well, not that much of an alternative, just enough to fool autotools and provide /sys to peek at my topology but its completely non-functional.)

{
  echo 'exec 3>/dev/null; set +e'
  find -P /proc/1/ /proc/2/ /proc/[^0-9]* -xdev \
    \( \
      -type d -printf "mkdir -p '.%p'\n" \
      -o -type f -printf "dd 'if=%p' 'of=.%p' bs=4096 conv=noerror count=16 status=none 2>&3 &\n" \
      -o -type l -printf "L='%l'; ln -snf -- "'"${L:-/}"'" '.%p'\n" \
    \) \
    -o -prune 2>/dev/null
  # We need at least two mockup processes for the best trickery mockup:
  #  a process with PID 1 for probing system boot info or whatever and
  #  a separate /proc/self as PID 2 that's us so we're not /sbin/init
  echo 'ln -snf 2 proc/self; ln -snf 2/task/2 proc/thread-self; wait'
} | sh -s

Notice the sh -s above. It's at least a few hundred times faster to pipe commands into one shell than spawn a new process every time, and the whole process finishes copying 2000-something files on my system in less than a second if I replace it with busybox sh -s (which uses its own builtin dd as fast as clone3ing a thread instead of spawning a new process every time.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .