83

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

If it's any help, I'm using RedHat and an Ubuntu linux systems.

130

As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

$ cat t.txt
12
34
56
78
90

sed

$ sed -e '1,3d' < t.txt
78
90

You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

tail

$ tail -n +4 t.txt
78
90

awk

$ awk 'NR > 3 { print }' < t.txt
78
90
  • 8
    You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier. – Yanick Girouard May 2 '12 at 23:46
  • 1
    Thanks @YanickGirouard, @IgnacioVazquezAbrams! You two have just saved me a ton of manual labor on my research! :) – Paul Sep 6 '12 at 5:22
  • 2
    @Svetlana sed -i specifically. Most implementations just delete the file and replace it with a new one, which doesn't work for links since you end up leaving the original at its other location. – jw013 Jan 7 '14 at 22:21
  • 5
    how about explaining what '1,3d', +4, et.c. means? The question was for n lines, but you didn't tell what n is (as apparently n is 2 in your examples, though it's not obvious for a noob what to change in order to change n) – Robin Manoli Feb 1 '15 at 10:29
  • 2
    This uses a temp file so not very useful for a 100% util disk space. Would be interesting to have a solution that does this literally "in-place". – Shai Sep 2 '16 at 21:09
9

sed -i '1,3d' file.txt

This deletes first 3 line from file.txt.

6

If the tabulated lines are the ones that have a tab character:

grep '␉' <input_file >output_file

( being a literal tab character) or equivalently

sed -n '/␉/p' <input_file >output_file

In a bash/ksh/zsh script, you can write $'\t' for a tab, e.g. grep $'\t' or sed -n $'/\t/p'.

If you want to eliminate 10 lines at the beginning of the file:

tail -n +11 <input_file >output_file

(note that it's +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or

sed '1,10d' <input_file >output_file

On Linux, you can take advantage of GNU sed's -i option to modify files in place:

sed -i -n '/\t/p' *.txt

Or you can use a shell loop and temporary files:

for x in *.txt; do
  tail -n +11 <"$x" >"$x.tmp"
  mv "$x.tmp" "$x"
done

Or if you don't want to modify the files in place, but instead give them a different name:

for x in *.txt; do
  tail -n +11 <"$x" >"${x%.txt}.data"
done
  • 3
    "tabulated" usually means "pretty-printed in a table", not "indented with tab characters". – Ignacio Vazquez-Abrams May 3 '12 at 2:14
  • @IgnacioVazquez-Abrams I know. The pretty-printed table sometimes uses tab characters, that's easier to spot than aligned columns. Of course, if Paul gave a sample input, I could give a better matcher. – Gilles May 3 '12 at 10:04
2

You can use Vim in Ex mode:

ex -sc '1d5|x' file
  1. 1 move to first line

  2. 5 select 5 lines

  3. d delete

  4. x save and close

0

echo "a\nb" | sed '1d' # deletes first line

cat list.txt | sed '1d' > list.csv # read list.txt and write list.csv without first line

Other useful commands:

grep '^|' # finds first character (pipe|)

sed 's/|//g' # deletes pipe

sed 's/ //g' # deletes space

0

By percentage

Using bash, to clean up a file using a percentage number instead of an absolute number of lines:

sed -i -e 1,$( printf  "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log

Watch out because that command can be destructive since it deletes content in-place, without creating a copy.

It deletes the first 75% of lines from the mentioned file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.