127

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

If it's any help, I'm using RedHat and an Ubuntu linux systems.

197

As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

$ cat t.txt
12
34
56
78
90

sed

$ sed -e '1,3d' < t.txt
78
90

You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

tail

$ tail -n +4 t.txt
78
90

awk

$ awk 'NR > 3 { print }' < t.txt
78
90
| improve this answer | |
  • 11
    You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier. – Yanick Girouard May 2 '12 at 23:46
  • 1
    Thanks @YanickGirouard, @IgnacioVazquezAbrams! You two have just saved me a ton of manual labor on my research! :) – Paul Sep 6 '12 at 5:22
  • 2
    @Svetlana sed -i specifically. Most implementations just delete the file and replace it with a new one, which doesn't work for links since you end up leaving the original at its other location. – jw013 Jan 7 '14 at 22:21
  • 9
    how about explaining what '1,3d', +4, et.c. means? The question was for n lines, but you didn't tell what n is (as apparently n is 2 in your examples, though it's not obvious for a noob what to change in order to change n) – Robin Manoli Feb 1 '15 at 10:29
  • 3
    This uses a temp file so not very useful for a 100% util disk space. Would be interesting to have a solution that does this literally "in-place". – Shai Sep 2 '16 at 21:09
22

sed -i '1,3d' file.txt

This deletes first 3 line from file.txt.

| improve this answer | |
  • 1
    I need to remove the 50 first lines from a 10GB+ text file. Even if it's supposed to work "in-place", this command still takes a few minutes. Is there any really fast alternative ? – Sébastien Oct 25 '19 at 8:09
  • 1
    @Sébastien if you only have to remove the 50 first lines, open the file in a text editor, select the 50 first lines and delete them maybe? It took 17 seconds to remove the 8 473 386 first lines of a 7GB+ text files with this command, I have to admit I find it quite fast. – smonff Apr 6 at 9:13
6

If the tabulated lines are the ones that have a tab character:

grep '␉' <input_file >output_file

( being a literal tab character) or equivalently

sed -n '/␉/p' <input_file >output_file

In a bash/ksh/zsh script, you can write $'\t' for a tab, e.g. grep $'\t' or sed -n $'/\t/p'.

If you want to eliminate 10 lines at the beginning of the file:

tail -n +11 <input_file >output_file

(note that it's +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or

sed '1,10d' <input_file >output_file

On Linux, you can take advantage of GNU sed's -i option to modify files in place:

sed -i -n '/\t/p' *.txt

Or you can use a shell loop and temporary files:

for x in *.txt; do
  tail -n +11 <"$x" >"$x.tmp"
  mv "$x.tmp" "$x"
done

Or if you don't want to modify the files in place, but instead give them a different name:

for x in *.txt; do
  tail -n +11 <"$x" >"${x%.txt}.data"
done
| improve this answer | |
  • 3
    "tabulated" usually means "pretty-printed in a table", not "indented with tab characters". – Ignacio Vazquez-Abrams May 3 '12 at 2:14
  • @IgnacioVazquez-Abrams I know. The pretty-printed table sometimes uses tab characters, that's easier to spot than aligned columns. Of course, if Paul gave a sample input, I could give a better matcher. – Gilles 'SO- stop being evil' May 3 '12 at 10:04
3

You can use Vim in Ex mode:

ex -sc '1d5|x' file
  1. 1 move to first line

  2. 5 select 5 lines

  3. d delete

  4. x save and close

| improve this answer | |
0

By percentage

Using bash, to clean up a file using a percentage number instead of an absolute number of lines:

sed -i -e 1,$( printf  "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log

Watch out because that command can be destructive since it deletes content in-place, without creating a copy.

It deletes the first 75% of lines from the mentioned file.

| improve this answer | |
0
# deletes first line
echo "a\nb" | sed '1d' 

# read list.txt and write list.csv without first line
cat list.txt | sed '1d' > list.csv 

Other useful commands:

# finds first character (pipe|)
grep '^|' 

# deletes pipe
sed 's/|//g'  

# deletes space
sed 's/ //g' 
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.