0

Context

I am trying to design a tool that would more or less automatically generate a Makefile from a standard shell script.

The idea would be that for each command executed by the script: I register the files that are read from and written to, then generate the associated dependencies in a Makefile along side with the associated command. I do not want to limit the scope of such command, they could be: a compiler call in an esoteric language, a test suite, a packaging system, etc...

In practice, the Makefile is only a cache for the tool. It provides a suggestion of commands that can run in parallel. The first run will assume complete sequential dependency and observe which files are read/written by which command. The following runs only run commands that need to be re-run, with a mechanism à la make. The subtlety is that if a command is trying to read a file that is not up-to-date yet (new dependency), the intercepted call will stall until the file is updated.

Problem

I need to intercept, in real time, access to files to detect which files are accessed by which command.

I did consider using LD_PRELOAD, but this assume all commands use libc calls to perform system calls. And I considered implementing my own kernel module to capture those system calls but then my tool will require undue permissions.

I see two solutions:

  • run the commands in debug mode
  • run the commands in a virtual environment using OS virtualization capabilities

Would debug mode provide me with enough control to intercept those calls? For instance, would it stack with a command that already run a sub-process in debug mode?

If not, how much work would involve using virtualization capabilities of the processor? That is, the virtualized operating system is mostly transparent (except for the stalling behavior), will I need to implement a lot of unrelated system calls?

Assumptions

I am only interested in files accessed by user space commands, if a program is run in privileged mode I do not expect to be able to "spy" it. I will assume a file is either produced/updated by exactly one command or it is a seed.

  • Why not use strace -fe open? – Stéphane Chazelas Sep 15 '16 at 16:29
  • 3
    Kind of meta to your question: what you describe seems like a solution searching for a problem. There are already way too many build environments, helpers, generators, make-alike programs, and what not. None of these tools use a shell script to do their work. So why would anybody use your tool, instead of using one of these make-alike programs, or indeed writing a plain Makefile? – Satō Katsura Sep 15 '16 at 16:41
  • To do something similar to what you want to do, the installwatch program uses LD_PRELOAD, I believe. – Mark Plotnick Sep 15 '16 at 17:16
  • @SatoKatsura Plain Makefiles need to be maintained in parallel of the code. It is one of the main reason why most people use generators (tracking header files is redundant work). I am not aware of generators that are language oblivious. I believe it is possible to make one in such a straight forward way that it could simply take a bash script. Note that if I catch all system calls I could get rid of the "bash script" requirement and bootstrapping the system from a Makefile would be feasible. – Nonyme Sep 15 '16 at 17:34
  • @MarkPlotnick Thank you for the reference, I didn't know this tool. Indeed, installwatch relies on LD_PRELOAD to catch libc function calls. I guess it is probably sufficient for my use case. But there are much more libc function calls manipulating files than linux system calls. – Nonyme Sep 15 '16 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.