23

I have the scenario where lines to be added on begining and end of the huge files.

I have tried as shown below.

  • for the first line:

    sed -i '1i\'"$FirstLine" $Filename
    
  • for the last line:

    sed -i '$ a\'"$Lastline" $Filename  
    

But the issue with this command is that it is appending the first line of the file and traversing entire file. For the last line it's again traversing the entire file and appending a last line. Since its very huge file (14GB) this is taking very long time.

How can I add a line to the beginning and another to the end of a file while only reading the file once?

  • Can you check with commands that I have mentioned. – Rahul Patil Aug 22 '13 at 11:27
19

sed -i uses tempfiles as an implementation detail, which is what you are experiencing; however, prepending data to the beginning of a data stream without overwriting the existing contents requires rewriting the file, there's no way to get around that, even when avoiding sed -i.

If rewriting the file is not an option, you might consider manipulating it when it is read, for example:

{ echo some prepended text ; cat file ; } | command

Also, sed is for editing streams -- a file is not a stream. Use a program that is meant for this purpose, like ed or ex. The -i option to sed is not only not portable, it will also break any symlinks to your file, since it essentially deletes it and recreates it, which is pointless.

You can do this in a single command with ed like so:

ed -s file << 'EOF'
0a
prepend these lines
to the beginning
.
$a
append these lines
to the end
.
w
EOF

Note that depending on your implementation of ed, it may use a paging file, requiring you to have at least that much space available.

  • Hi, ed command which u provided is working very well for huge files. But I have 3 huge files like Test, Test1, Test 2. I gave command like ed -s Tes* << 'EOF' 0a prepend these lines to the beginning . $a append these lines to the end . w EOF But its taking only Test file and adding first/last lines. How can we make changes in same command so that it has to do add first and last line in all files. – UNIXbest Aug 22 '13 at 12:50
  • @UNIXbest - Use a for loop: for file in Tes*; do [command]; done – Chris Down Aug 22 '13 at 12:55
  • Hi Down, i have used below command for file in Tes*; do ed -s Tes* << 'EOF' 0a HEllO HDR . $a Hello TLR . w EOF done But its still writing into first file. – UNIXbest Aug 22 '13 at 13:06
  • Right, because you need to use "$file", not Tes* as the argument to ed. – Chris Down Aug 22 '13 at 13:25
  • 2
    @UNIXbest If your problem has been solved by this answer, you should consider accepting it. – Joseph R. Aug 22 '13 at 13:55
9

Note that if you want to avoid allocating a whole copy of the file on disk, you could do:

sed '
1i\
begin
$a\
end' < file 1<> file

That uses the fact that when its stdin/stdout is a file, sed reads and writes by block. So here, it's OK for it to override the file it is reading as long as the first line you're adding is smaller than sed's block size (should be something like 4k or 8k).

Note though that if for some reason sed fails (killed, machine crash...), you'll end up with the file half processed which will mean some data the size of the first line missing somewhere in the middle.

Also note that unless your sed is the GNU sed, that won't work for binary data (but since you're using -i, you are using GNU sed).

  • this errors for me on Ubuntu 16.04 – Csaba Toth Nov 9 '16 at 19:11
4

Here are some choices (all of which will create a new copy of the file so make sure you have enough space for that):

  • simple echo/cat

    echo "first" > new_file; cat $File >> new_file; \
      echo "last" >> new_file; 
    
  • awk/gawk etc

    gawk 'BEGIN{print "first\n"}{print}END{print "last\n"}' $File > NewFile 
    

    awk and its ilk read files line by line. The BEGIN{} block is executed before the first line and the END{} block after the last line. So, the command above means print "first" at the beginning, then print every line in the file and print "last" at the end.

  • Perl

    perl -ne 'BEGIN{print "first\n"} print;END{print "last\n"}' $File > NewFile
    

    This is essentially the same thing as the gawk above just written in Perl.

  • 1
    Note that in all of these cases, you will require at least 14GB more space for the new file. – Chris Down Aug 22 '13 at 12:11
  • @ChrisDown good point, I edited my answer to make that clear. I assumed that was not a problem since the OP was using sed -i which creates temp files. – terdon Aug 22 '13 at 12:14
3

I prefer the much simpler:

gsed -i '1s/^/foo\n/gm; $s/$/\nbar/gm' filename.txt

This transforms the file:

asdf
qwer

to the file:

foo
asdf
qwer
bar
2

There is no way to insert data at the beginning of a file¹, all you can do is create a new file, write the additional data, and append the old data. So you'll have to rewrite the whole file at least once to insert the first line. You can append the last line without rewriting the file however.

sed -i '1i\'"$FirstLine" $Filename
echo "$LastLine" >>$Filename

Alternatively, you can combine the two commands in one run of sed.

sed -i -e '1i\'"$FirstLine" -e '$ a\'"$Lastline" $Filename

sed -i creates a new output file and then moves it over the old file. This means that while sed is working, there is a second copy of the file using up space. You can avoid this by overwriting the file in place, but with major restrictions: the line you're adding has to be smaller than sed's buffer, and if your system crashes you'll end up with a damaged file and some content lost in the middle, so I strongly recommend against it.

¹ Linux does have a way to insert data into a file, but it can only insert a whole number of filesystem blocks, it can't insert strings of arbitrary lengths. It's useful for some applications, such as databases and virtual machines, but it's useless for text files.

  • Not true. Look at fallocate() with FALLOC_FL_INSERT_RANGE available on XFS and ext4 in modern kernels (4.x.x) man7.org/linux/man-pages/man2/fallocate.2.html – Eric Apr 20 at 12:57
  • @Eric You can only insert whole blocks, though, not arbitrary byte lengths, at least as of Linux 4.15.0 with ext4. Is there a filesystem that can insert arbitrary byte lengths? – Gilles 'SO- stop being evil' Apr 20 at 22:11
  • Right but it still doesn't make your statement correct. You wrote: "There is no way to insert data at the beginning of a file". That's still not true: there is a mechanism to insert extents at the beginning of a file. It comes with caveats, sure, but it's worth mentioning because some users may not care about the block size restrictions by filling with spaces or carriage returns. – Eric Apr 22 at 5:25
1

You can use Vim in Ex mode:

ex -sc '1i|ALFA' -c '$a|BRAVO' -cx file
  1. 1 select first line

  2. i insert text and newline

  3. $ select last line

  4. a append text and newline

  5. x save and close

  • what if we wanted to do this to multiple files? – geoyws Oct 25 '16 at 6:54
  • @geoyws that is not really in scope for this question – Steven Penny Oct 25 '16 at 11:31
0
$ (echo "Some Text" ; cat file1) > file2
  • 4
    Only code answer are not acceptable, please improve your answer – Networker Aug 22 '14 at 21:33
  • Consider expanding your answer to include an explanation of your suggestion, or links to documentation which support your solution. – HalosGhost Aug 22 '14 at 21:49
-1

Modern Linux kernels (higher than 4.1 or 4.2) support inserting data at the beginning of a file via the fallocate() system call with FALLOC_FL_INSERT_RANGE on ext4 and xfs filesystems. In essence this is a logical shifting operation: the data is logically relocated at a higher offset.

A constraint exists regarding the granularity of the range you want to insert at the beginning of the file. But for text files you can probably allocate a little more than required (up to the granularity boundary) and fill with spaces or carriage returns, but that depends on your application

I don't know of any readily available linux utility that manipulates file extents but it isn't difficult to write: get a file descriptor and call fallocate() with the appropriate arguments. For further details, refer to the man page of the fallocate system call: http://man7.org/linux/man-pages/man2/fallocate.2.html

  • A utility isn't the problem (assuming a non-embedded Linux): util-linux contains a fallocate utility. The problem is that a granularity of whole blocks makes this useless for most text files. Another problem is that the range allocation and subsequent modification are not atomic. So this doesn't actually solve the problem here. – Gilles 'SO- stop being evil' Apr 20 at 22:13
  • The granularity is a caveat I've mentionned already and no, it doesn't make it useless, it depend on the application. Where did you see in the question that atomicity is important? I can only see the problem of performances. Even so this syscall seems to be atomic: elixir.bootlin.com/linux/latest/source/fs/open.c#L228 and if atomicity becames important (it isn't, but say it is for the sake of argument) then just use file locking. (point me to the place in the kernel code where fallocate atomicity is broken please, I'm curious) – Eric Apr 22 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.