1

I'm trying to figure this out. Scenario is this:

A threaded tcp socket daemon with a sqlite back-end stores information in a unique sqlite2 file per IMEI ID, in an unique sub-directory per day (pseudo: 2012/11/01/event_$imei.sqlite). About 3000 of them for each day. As you know, there is a phase that an sqlite journal file is being created and deleted while receiving and storing incoming data.

This application only writes in either today's or yesterdays directory, (small) datafeeds can take up to 255 seconds. So there is a phase-over at 00:00 UTC. During a connection multiple writes can happen, the network link stays open and the assets can send additional bursts of new data while the connection is open. No explicit locking is going on at all, sqlite handles itself, a connection is simply unique per imei, so never there are 2 threads writing to the same file. With other words, the controlling daemon doesn't have a clue of what sqlite does nor does it care.

I want to count the number of sqlite journal files ever created as a 2 totals at a given point in time of course. Essentially they are about the same but never really the same, they only live for very short amount of time anyway. Monitoring systems usually count the number of present files as a snapshot in time. That is not the sort of metric I'm looking for.

So How can you accurately do this sort of count with the smallest footprint on the load ? There must be a trick, it's *nix after all. The system here is an Ubuntu server but I slightly prefer -but not limited to- a general solution. But keep in mind new directories will be created as time passes and I should be able to follow that, and I want it volatile (so a reboot can(must) reset this).

Thanks in advance for bootstrapping me in any direction.

Improve this question
7
  • 1
    It's not clear why you'd want to do that. Wouldn't counting the number of transactions in your application be easier (and quite possibly more accurate) than anything you could do "from the outside"?
    – Mat
    Nov 1, 2012 at 7:24
  • There are more situation that this one alone where it could be useful to count the number of a certain type of files created. On top of that, I mentioned the fact that it is threaded daemon which makes a general counter a big pain in the neck to build. There is no way for the daemon to know what sqlite engine does. tx
    – Glenn Plas
    Nov 1, 2012 at 9:02
  • The SQL engine doesn't do anything you don't tell it to do. Relying on counting filesystem events is notoriously racy and error-prone, there are very few operations that are guaranteed to be atomic.
    – Mat
    Nov 1, 2012 at 9:05
  • But is it possible to count them somehow ?
    – Glenn Plas
    Nov 1, 2012 at 9:07
  • 2
    inotify would be a place to start. Make sure you read the NOTES section. (There are a few command-line tools available around that interface.)
    – Mat
    Nov 1, 2012 at 9:08

1 Answer 1

Reset to default
0

inotify would be a good candidate to do this sort of counting but as suspected this approach isn't ideal and prone to race conditions, in the end I agree that in fact I'm trying to count the number of transactions with this, so it would be better to do this in the application itself and keep count.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .