0

I have a 10Mb file filled by null bytes. A program is accessing it and changes zeros to specific strings up to the end of the file.

I've tried to use tail -F | grep wanted_text | grep -v "unwanted_text", but it does not monitor changes. It only works for usual text files, but not for files filled by zeros.

All null bytes are replaced by lines separated by a new line character till the end of the file. After the file is filled up it is being renamed and the new one is created instead.

So how could I monitor changes of a file filled by null bytes with ability to filter output?

8
  • 2
    By "zeros", do you mean "nul bytes"? (bytes whose value is zero, rather than ASCII 48, "0")
    – Kusalananda
    Nov 25, 2019 at 14:35
  • What about hash functions? Nov 25, 2019 at 14:51
  • yes, null bytes. cat displays only text, not zeros, thank you!
    – Gryu
    Nov 25, 2019 at 15:00
  • How exactly this other program is writing zeros to this file? If you just need to monitor when a file was changed and not what was changed on it, then inotifywait from inotify-tools may be a solution for this problem... I've tested the command inotifywait --format=%w -q -m -e modify /home/myuser/file.txt while writing nullbytes to the file with dd if=/dev/zero bs=1 count=1 >> /home/myuser/file.txt and it detected when the file was modified (not what was modified or appended) Nov 25, 2019 at 17:04
  • 1
    man tail is explicit that it will report "appended data as the file grows". It just remembers the current file size at each output, and after the delay it seeks to that point and reads to EOF. It is not equipped to scan the whole 10MB and see what is different to last time in any location. Nov 26, 2019 at 0:32

5 Answers 5

1

This is the script for Reader, which should be close to what you need to fake a tail command for a NUL-filled file. It checks for changes in the file (by comparing the whole ls -l output, which includes a timestamp down to nanoseconds), and reports any additions in a batch. It does not report lines that are already in the file when it starts up, only additions while it is running.

It runs at two speeds to avoid wasted checks. If it detects any additions, it tries again after 1.0 seconds. If a cycle sees no additions, it tries again after 5 seconds (this 5 is an argument to the process).

#! /bin/bash
#: Reader: tail -f a file which is pre-formatted with many trailing NUL characters.

#### Implement the User Requirement.

function Reader {

    local RUN="${1:-60}" SLEEP="${2:-5}" FILE="${3:-/dev/null}"

    local AWK='''
BEGIN { NUL = "\000"; }
function Tick (Local, cmd, ts) {
    cmd = "date \047+%s\047";
    cmd | getline ts; close (cmd); return (ts);
}
function TS (Local, cmd, ts) {
    cmd = "date \047+%H:%M:%S.%N\047";
    cmd | getline ts; close (cmd); return (ts);
}
function Wait (secs) {
    system (sprintf ("sleep %s", secs));
}
function isChange (Local, cmd, tx) {
    cmd = sprintf ("ls 2>&1 -l --full-time \047%s\047", Fn);
    cmd | getline tx; close (cmd);
    if (tsFile == tx) return (0);
    tsFile = tx;
    if (index (tx, "\047")) {
        if (fSt != "B") { fSt = "B"; printf ("%s: No file: %s\n", TS( ), Fn); }
    } else {
        if (fSt != "G") { fSt = "G"; printf ("%s: Reading: %s\n", TS( ), Fn); }
    }
    return (1);
}
function atNul (buf, Local, j) {
    j = index (buf, NUL);
    return ((j > 0) ? j : 1 + length (buf)); 
}
function List (tx, Local, ts, X, j) {
    sub ("\012$", "", tx); split (tx, X, "\012");
    ts = TS( );
    for (j = 1; j in X; ++j) {
        printf ("%s %3d :%s:\n", ts, length (X[j]), X[j]);
    }
}
function Monitor (Local, rs, tk, Buf, Now, End) {
    printf ("%s: READER Begins\n", TS( ));
    tk = Tick( ); Expired = tk + Run;
    Now = -1;
    while (Tick( ) <= Expired) {
        if (! isChange( )) { Wait( Sleep); continue; }
        rs = RS; RS = "\000";
        Buf = ""; getline Buf < Fn; close (Fn);
        RS = rs;
        if (Now < 0) Now = atNul( Buf);
        End = atNul( Buf);
        List( substr (Buf, Now, End - Now));
        Now = End;
        Wait( 1.0);
    }
    printf ("%s: READER Exits\n", TS( ));
}
NR == 1 { Run = $0; next; }
NR == 2 { Sleep = $0; next; }
NR == 3 { Fn = $0; }
END { Monitor( Fn); }
'''
    {
        echo "${RUN}";
        echo "${SLEEP}";
        echo "${FILE}";

    } | awk -f <( echo "${AWK}" )
}

#### Script Body Starts Here.

    Reader 40 5 "./myNullFile"
4
  • Great! It works! Thank you very much! Does 40 at the end of the script means script working time in seconds?
    – Gryu
    Dec 2, 2019 at 23:30
  • Yes, it is. I've checked it by changing it to 4000 and script have not been exited since a short period of time like it was previously. Thank you!
    – Gryu
    Dec 2, 2019 at 23:33
  • 1
    I found a bug since I posted. If you click the "edited xx mins ago" under the post of Reader, it will show you the exact change. The effect was that it does not see when the file is unchanged, so it rechecks every time. It works just the same, but is less efficient. The line is just tsFile == tx; (not the if just above it) and it should be tsFile = tx;. Dec 3, 2019 at 0:14
  • 1
    @gryu Glad it worked. If you need more info, or to automate the switch to next file, or with startup conditions, just comment with my tag to make sure I get a message. Dec 3, 2019 at 10:51
1

Problems with the whole concept.

  1. Does the writer only ever replace NUL bytes with other strings, or can it write new strings over old strings, possibly with incomplete overlaps? Will strings always have at least one NUL separator between them?

  2. Can it write over strings with fresh NULs to erase parts of the file, too?

  3. Is the original file really 10MB of NUL, or is it initially a sparse file?

  4. Given that we can only find strings by reading the whole file, how often are you prepared to do this?

  5. Is there any way to lock the file while it is being written, to prevent race conditions?

  6. Can the file size ever change during the whole operation?

awk (at least, GNU/awk) can deal with NUL characters and long lines. It could keep a list of ranges that were NUL (initially just [0,10485760]), and check for new fragmentation in those regions. That would not detect over-writes though. But it would be able to report all the additions without any extra processes.

GNU/awk has a built-in patsplit() function, which cuts up a string according to a RE separator, making an array of fields and an array of separators. So the RE /[\000]+/ should put all the strings in one array, and all the NUL repeats in the other array, and you can length() them all cumulatively to find the total offset in the file for each string. That looks like an excellent candidate for investigation.

The cat command does display NUL characters, by the way. You can see them in a file using the od command. The reason they don't show up in a terminal is that the terminal driver ignores them.

As Romeo suggests, keeping a cksum of the previous file would tell you if it had changed, but not where. So it might be a helpful optimisation, depending on the frequency of updates.

2
  • I'm sorry. Your answer is useful, but I could not understand how to use it. I'm not a programmer, so It will require more time to understand how to use it in my case.
    – Gryu
    Nov 27, 2019 at 14:33
  • 1
    Sorry, I mentioned awk and patsplit() as interesting tools to be investigated, not expecting you to use directly. I currently have (A) a simulation that creates and writes to a 10MB file of nulls. At random intervals, it picks a random row from a text file, and writes it into the file at a random place. This is about 50 lines of script (bash and awk). I wrote this because I don't write code that I can't test. I also have (B) a reader for the file, which picks out all the strings, compares them with what it got last time, and lists deletes and inserts. That is 80 lines of bash and awk. Nov 27, 2019 at 22:03
1

I have done enough to verify that my concept of using GNU/awk with patsplit() is feasible. Setting up a fake Writer took about 70% of the development time. I found a set of dd options that lets me set up a 10MB file, and then write strings in random places in it periodically.

I have a Reader that drags the whole thing into memory as one long string, and separates the nulls into one array and the strings into another. It takes 0.044 secs to read the 10MB, 0.989 to split the string into the arrays, and 0.138 secs to report the start, length and content of the 20 strings I placed. So about 1.2 seconds to do a file snapshot.

All timings done on my 8-year-old cheapo Laptop. I think that, as it has to parse the whole 10MB anyway, having many more strings won't affect performance that badly. Next step is to confirm that.

I believe that keeping an old and new hash table of the strings, and finding the changes, will be simple and efficient.

Is any more known about the addition of strings to the data here? If it was always contiguous with previous data, it would be easy to emulate tail by looking just after the previous string. If it was infrequent, we could check the timestamp before reading the file. If it was writing an index into the first part of the file, then we could check this first. The whole concept of this file makes it hard to see what use it is to the rest of the system anyway -- it is just a hostile way to use storage.

Is this question still of interest? I don't see any response from the OP to my previous questions, but seems to me that string overlaps and so on are just going to show as updates and length changes.

6
  • Thank you! Yes, this question still of interest. I was looking for a program that could be used for this purpose, but could not find it. So, it looks like there's no such program - only scripting could help.
    – Gryu
    Nov 27, 2019 at 15:52
  • 1
    We need to consider synchronisation with the Writer. If that runs always, and the Reader is restarted, you will get all existing messages on the first scan. We could skip reporting on the first scan, and just show additions after that. Or we could save current known messages when we exit, and report all messages since then on restart. We also want to know average arrival rate of messages, to set timers optimally. Also, are these all text messages? Any binary would contain NUL chars that would look like multiple messages. Is the space well organised? Any overlaps would be confusing to read. Nov 29, 2019 at 0:42
  • There are only text messages. All text messages divided by hexadecimal 0A that looks like a dot symbol in mcview. But null bytes, that look like a dot also and have 00 code are located after the last text string ended with 0A hexadecimal code.
    – Gryu
    Nov 29, 2019 at 20:21
  • 1
    Hold on there. Are you saying the Writer adds all messages one after the other, starting at byte 0 of the file and packing them all closely, and using 0x0A (which is newline) to separate them? That simplifies the problem beyond belief. I've spent about 4 days looking for strings placed anywhere in the file, and it sounds like I can just look immediately after the last message I found. My only issue would be: what happens when the file is full? Does it start over from the front, or switch to a new file? I still can't figure why anybody would design this way though. Nov 30, 2019 at 0:16
  • 1
    OK, I have the code for this situation too -- both a Writer test package and a Reader. What you have introduced is another issue: you want this to follow newer versions of the file as they are created? If you don't do that, it will just hang on a file that is no longer being written to, and you will need to restart it with a different filename as parameter. Dec 1, 2019 at 19:43
1

I have tested code for a Writer (to generate a test) and a Reader (the script you asked for), and will post both later. This is a 30-second test run to prove this works. The Writer makes a 10MB file when it starts using dd, and dumps it using od, and reports each output as Write Pos. The data is a random line from the script file. Reader just shows the length and content (each length is 1 shorter because the write includes NL and the read discards NL). We need to fix the changeover between files when you decide how and when this happens.

Paul--) ~/wdog -w 1m; ./Reader & sleep 3; ./Writer & jobs; wait
[1] 25030
21:40:00.032622643: READER Begins
21:40:00.059864870: No file: ./myNullFile
[2] 25066
[1]-  Running                 ./Reader &
[2]+  Running                 ./Writer &
1+0 records in
1+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0421849 s, 249 MB/s
0000000  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
        nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
10485760
21:40:03.155837791: WRITER Begins
21:40:03.166173417: Reading: ./myNullFile
Initial NUL is at 0
21:40:04.053667668: Write Pos          0 Lth  32 Txt :print Tx[Row] > Wk; close (Wk);:
21:40:04.221015418  31 :print Tx[Row] > Wk; close (Wk);:
21:40:07.453403168: Write Pos         32 Lth  26 Txt :Row = int (nTx * rand());:
21:40:08.320950327: Write Pos         58 Lth  26 Txt :Row = int (nTx * rand());:
21:40:08.331308212  25 :Row = int (nTx * rand());:
21:40:08.331308212  25 :Row = int (nTx * rand());:
21:40:11.526810010: Write Pos         84 Lth  42 Txt :n = patsplit (buf, Str, /[^\000]+/, Sep);:
21:40:12.444730213  41 :n = patsplit (buf, Str, /[^\000]+/, Sep);:
21:40:12.998876406: Write Pos        126 Lth  51 Txt :function Monitor (Local, rs, tk, Buf, Now, P, Q) {:
21:40:13.474633987  50 :function Monitor (Local, rs, tk, Buf, Now, P, Q) {:
21:40:14.604161200: Write Pos        177 Lth  43 Txt :Parser( Q, Buf); Differ( Q, P); Now = "P";:
21:40:15.529375510  42 :Parser( Q, Buf); Differ( Q, P); Now = "P";:
21:40:18.381688129: Write Pos        220 Lth  44 Txt :cmd | getline ts; close (cmd); return (ts);:
21:40:18.611992430  43 :cmd | getline ts; close (cmd); return (ts);:
21:40:19.583131365: Write Pos        264 Lth  19 Txt :split ("", X, FS);:
21:40:19.642066181  18 :split ("", X, FS);:
21:40:20.670745471: Write Pos        283 Lth   2 Txt :}:
21:40:21.701154684   1 :}:
21:40:23.060019472: Write Pos        285 Lth   5 Txt :done:
21:40:23.749500189   4 :done:
21:40:25.956196125: Write Pos        290 Lth   2 Txt :}:
21:40:26.832889643   1 :}:
21:40:27.098544055: Write Pos        292 Lth  19 Txt :split ("", X, FS);:
21:40:27.861066600  18 :split ("", X, FS);:
21:40:30.707892127: Write Pos        311 Lth  32 Txt :print Tx[Row] > Wk; close (Wk);:
21:40:30.939979744  31 :print Tx[Row] > Wk; close (Wk);:
21:40:33.705042808: Write Pos        343 Lth  15 Txt :Reader 30 10 &:
21:40:34.010378554  14 :Reader 30 10 &:
21:40:35.103897039: Write Pos        358 Lth   2 Txt :}:
21:40:36.063585547   1 :}:
21:40:38.661280841: Write Pos        360 Lth  49 Txt :printf ("%s: Write Pos %10d Lth %3d Txt :%s:\n",:
21:40:38.674634657: WRITER Exits
21:40:39.144828936  48 :printf ("%s: Write Pos %10d Lth %3d Txt :%s:\n",:
21:40:41.188952035: READER Exits
[1]-  Done                    ./Reader
[2]+  Done                    ./Writer
1

This is the script for Writer. It uses the dd command to create the initial file in one shot (if it does not exist), and then uses dd to stuff random lines from a script file into the file. It used to do this at random positions, but it now places each one after the previous one. It adds lines at random intervals averaged around a given argument (2 seconds in this version). It exits after a specific time limit, or if the file is full.

#! /bin/bash

#.. Declare the shared file.
FILE="./myNullFile"
SIZE=$(( 10 * 1024 * 1024 ))

#.. Using script as source of the strings.
TEXT="./isNulls"
WORK="./myLine"

#### Simulate the file writer defined in the question.

function Writer {

    local RUN="${1:-60}" SLEEP="${2:-5}"

    local AWK='''
BEGIN { printf ("%s: WRITER Begins\n", TS( )); }
function Begin (Local) {
    Pos = getNull( );
    printf ("Initial NUL is at %d\n", Pos);
    if (Pos < 0) exit;
}
function TS (Local, cmd, ts) {
    cmd = "date \047+%H:%M:%S.%N\047";
    cmd | getline ts; close (cmd); return (ts);
}
function Wait (secs) {
    system (sprintf ("sleep %s", secs));
}
function getNull (Local, rs, Buf) {
    rs = RS; RS = "^$";
    getline Buf < Fn; close (Fn);
    RS = rs;
    return (-1 + index (Buf, "\000"));
}
function Str (Local, Row, Lth) {
    Row = int (nTx * rand());
    Lth = length (Tx[Row]);
    if (Pos + Lth >= Sz) {
        printf ("%s is full: Pos %d, Lth %d, Sz %d\n", Fn, Pos, Lth, Sz);
        exit;
    }
    printf ("%s: Write Pos %10d Lth %3d Txt :%s:\n",
        TS( ), Pos, 1 + Lth, Tx[Row]);
    print Tx[Row] "\n" > Wk; close (Wk);
    system (sprintf (Fmt, Pos, 1 + Lth, Wk, Fn, Wk));
    Pos += 1 + Lth;
}
NR == 1 { Fmt = $0; srand (); next; }
NR == 2 { Fn = $0; next; }
NR == 3 { Sz = $0; next; }
NR == 4 { Wk = $0; Begin( ); next; }
NF { sub (/^[ \011]+/, ""); Tx[nTx++] = $0; next; }
{ Str( ); }
END { printf ("%s: WRITER Exits\n", TS( )); }
'''
    local EXPIRED=$(( SECONDS + RUN ))
    local AWK_WT='BEGIN { srand(); } { print 0.1 + 2 * $1 * rand(); }'
    {
        DD_OPT='status=none conv=notrunc bs=1 seek="%s" count="%s"'
        DD_FNS='if="%s" of="%s" && rm -f "%s"'

        echo "dd ${DD_OPT} ${DD_FNS}"
        echo "${FILE}"; echo "${SIZE}"; echo "${WORK}"
        awk NF "${TEXT}"
        while (( SECONDS <= EXPIRED )); do
            sleep "$( echo "${SLEEP}" | awk "${AWK_WT}" )"
            echo ''
        done
    } | awk -f <( echo "${AWK}" )
}

#### Script Body Starts Here.

    [[ -r "${FILE}" ]] || {
        dd count=1 bs="${SIZE}" if="/dev/zero" of="${FILE}"
        od -A d -t x1a "${FILE}"
    }

    Writer 32 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .