I have just came to the problem of having to cut some lines from a large (gigabyte) sized file, and being aware of potential CPU hog trying to read it in memory, I wanted to edit it in-place instead... and so came upon these questions:
- How do I remove certain lines (using line numbers) in a file?
- Is there a way to modify a file in-place?
... and further also these:
However, I was wandering about something else: I believe (but I'm not sure) that any filesystem (like ext3
) would have to employ something like a linked list, in order to be able to describe something like fragments of a file that are mapped to areas of disk.
Thus, it should be possible to do something like this - for example, let's say, I have a file bigfile.dat
like this (numbers should indicate byte offset, but it's a bit difficult to align them):
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
L 1\n L 2\n L 3\n L 4\n L 5\n L 6\n
This file could then, in principle, be loaded in a terminal application for browsing - let's imagine we call a tool editsegments bigfile.dat
, and let's say it similar to how less -N bigfile.dat
would display the same file (with line numbers):
1 1 L 1
2 2 L 2 *
3 3 L 3
4 4 L 4 *
5 5 L 5
6 6 L 6
bigfile.dat (END)
Let's say, I could enter a command there (say d
for delete lines), click another key or the mouse where it is indicated above with *
- meaning that everything between lines 2 and 4 should be deleted. The program would then respond with this being shown:
1 1 L 1
2 5 L 5
3 6 L 6
bigfile.dat (END)
Now we can see that leftmost first column shows "new" line number (after cut), second column is "old" line number (before cut) - and then the actual line contents follow.
Now, what I imagine happens after this pseudoapplication editsegments
is exited, is that first and foremost, bigfile.dat
is untouched; however, now there would be also an extra text file in same directory, say bigfile.dat.segments
; with these contents:
d 4:15 # line 2-4
... and additionally, a special file (like a "symlink") - let's call it bigfile.dat.iedit
- would appear.
Now, basically, the result of all this would be, that if I now try to open bigfile.dat.iedit
with something like less -N bigfile.dat.iedit
, I'd want to get the "edited" contents:
1 L 1
2 L 5
3 L 6
bigfile.dat (END)
... which could be achieved, I guess, by somehow instructing the operating system, that when $FILE.iedit
is opened, first $FILE.segments
should be opened and read; the d 4:15
would instruct that bytes 4 to 15 in the original file should be left out - which would result with something like:
0 1 2 3 4 5 6 7 8 9 10 11 12,3,4 15 16 17 18 19 20 21 22 23
L 1\n
L
2
\n
L
3
\n
L
4
\n
L 5\n
L 6\n
0 1 2 3 ------------------------------->16 17 18 19 20 21 22 23
In other words - assuming that in a filesystem concept of a file, each byte of content also contains a "link" to the next byte in the chain - it should be possible to instruct the filesystem to establish a new linked list based on a script, and provide the contents as represented by this modified linked list through a special file (symlink or pipe).
That is what I meant by "scripted" in the title - that the "new" linked list can be controlled by a script file ($FILE.segments
), user-editable in a text editor (or generated by a front end application). What I meant by "multipass" is the fact that bigfile.dat
in this process is not modified at all; so I could edit the first (original) gigabyte today, saving progress in ($FILE.segments
) - then I could edit the second gigabyte tomorrow, again saving progress in ($FILE.segments
) etc. - all the while, the original bigfile.dat
is unchanged.
When all edits are complete, one could probably call a command of sorts (say, editsegments --finalize bigfile.dat
), which would simply permanently encode the new linked list as the contents of bigfile.dat
(and in line with that, remove bigfile.dat.segments
and bigfile.dat.iedit
). Or even easier, one could just do:
cp bigfile.dat.iedit /path/to/somewhere/else/bigfile.modified.dat
Of course, besides a d
elete script command, one could have a r
eplace command as well, say:
r 16:18 AAA
... saying: replace the content between bytes 16 and 18 with the next 18-16+1=3 bytes after the space (that is, the AAA
) - the linked list could in fact "hook" into the script command content itself (the below chart containing also the d
elete):
0 1 2 3 4 5 6 7 8 9 10 11 12,3,4 15 16 17 18 19 20 21 22 23
L 1\n
L
2
\n
L
3
\n
L
4
\n
L
5
\n
L 6\n
0 1 2 3 ------------------------------->| | 19 20 21 22 23
.
.
.
.
.
.
.
\n
r
1
6
:
1
8
AAA \n
.
.
.
.
Now, I guess that programs like hexedit
(as mentioned here) do change files in-place - but I'd just like the benefit of possibility of scripting (even better if it could be regulated by a GUI application, even if a terminal one), and the benefit of not actually having the original file changed, until one confirms all edits are as required.
I'm not sure if something like this is possible at all - and even if it is, I guess it may require a dedicated driver (rather than just a user program)... But I guess it is worth asking anyways - is there anything like this for Linux?
Many thanks in advance for any answers,
Cheers!