My file has millions of lines, resides in memory
/dev/shm/tmp.file, is accessed by multiple threads, looks like this
831092,25a1bd66f2eec71aa2f0a8bb3d,/path/to/a/file 4324,8d83c29e4d8c71bd66f1bd66fs,/path/to/another/file ...
and is sorted by the part after the second
sort -t , -k3.
In general each line has the shape
[0-9]*,[0-9a-z]*,.* and file paths can contain any characters except
I need to extract the lines of all files that reside within a given directory as quickly as possible and without making an additional copy. Since the file is sorted that way, the lines I am looking for are an uninterrupted chunk of the file.
Currently I use
grep -F ',<directory>' /dev/shm/tmp.file but I know it would be much quicker to do a binary search for the first hit and then expand the chunk line by line or with another binary search without reading in the entire file for each new line. However, this has to be integrated into a bash script, and I found no way to do something like lseek in the bash.
There is sgrep but it requires the complete lines to be sorted.
How can I extract all matches with
',<directory>' quicker than
Edit: The input
/dev/shm/tmp.file is only there to do this kind of extraction. Hence, pre-processing it in some way to make the job easier is an option.
a.b sorting between
a/b is not an issue since all subdirectories should be included in the chunk.