161

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

If it's any help, I'm using RedHat and an Ubuntu linux systems.

7 Answers 7

246

As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

$ cat t.txt
12
34
56
78
90

sed

$ sed -e '1,3d' < t.txt
78
90

You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

tail

$ tail -n +4 t.txt
78
90

awk

$ awk 'NR > 3 { print }' < t.txt
78
90
11
  • 13
    You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier. May 2, 2012 at 23:46
  • As long as the file is not a symlink or hardlink.
    – jw013
    May 3, 2012 at 0:26
  • 2
    @Svetlana sed -i specifically. Most implementations just delete the file and replace it with a new one, which doesn't work for links since you end up leaving the original at its other location.
    – jw013
    Jan 7, 2014 at 22:21
  • 12
    how about explaining what '1,3d', +4, et.c. means? The question was for n lines, but you didn't tell what n is (as apparently n is 2 in your examples, though it's not obvious for a noob what to change in order to change n) Feb 1, 2015 at 10:29
  • 3
    This uses a temp file so not very useful for a 100% util disk space. Would be interesting to have a solution that does this literally "in-place".
    – Shai
    Sep 2, 2016 at 21:09
41

sed -i '1,3d' file.txt

This deletes first 3 line from file.txt.

3
  • 2
    I need to remove the 50 first lines from a 10GB+ text file. Even if it's supposed to work "in-place", this command still takes a few minutes. Is there any really fast alternative ?
    – Sébastien
    Oct 25, 2019 at 8:09
  • 1
    @Sébastien if you only have to remove the 50 first lines, open the file in a text editor, select the 50 first lines and delete them maybe? It took 17 seconds to remove the 8 473 386 first lines of a 7GB+ text files with this command, I have to admit I find it quite fast.
    – smonff
    Apr 6, 2020 at 9:13
  • If you have a 10GB log file and only ~1GB left of space, this solution won't work as creates a tmp file
    – bk201
    Oct 6, 2021 at 22:00
9

If the tabulated lines are the ones that have a tab character:

grep '␉' <input_file >output_file

( being a literal tab character) or equivalently

sed -n '/␉/p' <input_file >output_file

In a bash/ksh/zsh script, you can write $'\t' for a tab, e.g. grep $'\t' or sed -n $'/\t/p'.

If you want to eliminate 10 lines at the beginning of the file:

tail -n +11 <input_file >output_file

(note that it's +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or

sed '1,10d' <input_file >output_file

On Linux, you can take advantage of GNU sed's -i option to modify files in place:

sed -i -n '/\t/p' *.txt

Or you can use a shell loop and temporary files:

for x in *.txt; do
  tail -n +11 <"$x" >"$x.tmp"
  mv "$x.tmp" "$x"
done

Or if you don't want to modify the files in place, but instead give them a different name:

for x in *.txt; do
  tail -n +11 <"$x" >"${x%.txt}.data"
done
2
  • 3
    "tabulated" usually means "pretty-printed in a table", not "indented with tab characters". May 3, 2012 at 2:14
  • @IgnacioVazquez-Abrams I know. The pretty-printed table sometimes uses tab characters, that's easier to spot than aligned columns. Of course, if Paul gave a sample input, I could give a better matcher. May 3, 2012 at 10:04
4

You can use Vim in Ex mode:

ex -s -c '1d5|x' file
  1. 1 move to first line

  2. 5 select 5 lines

  3. d delete

  4. x save and close

0
1

By percentage

Using bash, to clean up a file using a percentage number instead of an absolute number of lines:

sed -i -e 1,$( printf  "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log

Watch out because that command can be destructive since it deletes content in-place, without creating a copy.

It deletes the first 75% of lines from the mentioned file.

0
# deletes first line
echo "a\nb" | sed '1d' 

# read list.txt and write list.csv without first line
cat list.txt | sed '1d' > list.csv 

Other useful commands:

# finds first character (pipe|)
grep '^|' 

# deletes pipe
sed 's/|//g'  

# deletes space
sed 's/ //g' 
-1

i created a small script which will delete everything from the /var/log/messages except last 4 lines.

# cat remove-range-of-lines.sh

#!/usr/bin/bash

#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`

#exclude last 4 lines
remove_line=`expr $line_count - 4`

#remove everything except last 4 lines
sed -i '1,'$remove_line'd' /var/log/messages

using this script we can always keep latest entry in the /var/log/messages based on our requirement

if you want to keep last 10000 entries in the /var/log/messages below is the modified script.

#!/usr/bin/bash

#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`

#exclude last 10000 lines
remove_line=`expr $line_count - 10000`

#remove everything except last 10000 lines
sed -i '1,'$remove_line'd' /var/log/messages
1
  • 3
    This is not an answer to this question. The issue is to remove the first n lines, not to keep the last n lines.
    – Kusalananda
    Aug 12, 2021 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.