4

I have a collection of text files containing more data than I need. Each file's first line contains a comma-separated string that looks like this:

stop_id,stop_code,stop_name,stop_desc,stop_lat,stop_lon,location_type,parent_station,zone_id

Then, below those keys is all the data. I need to extract a subset of that data into a new text file so I can work with the subset (I don't need all the data, it's too much).

I'm using this command to extract the first line:

sed -n '1p' source.txt > destination.txt

I'm also using this command to extract the specific lines I need:

grep "string" source.txt > destination.txt

The challenge is that when I run the two commands in the same script (pretty much as is, separated by a line or &&), the grep output overwrites the sed output. How can I run both in sequence and have the combined output of both?

I noticed a question that seems similar and involves using a more complex grep command to locate the one line, followed by a range of lines. That won't work here because the first line of each of the files I need to extract data from is different.

Ideally, I want to write a function that I can run against each of the files I need to work with but I need to chain these commands and combine their outputs first.

  • 1
    A simpler command to do the job of sed -n 1p would be head -n 1. – David Foerster Dec 30 '17 at 1:05
8

Just change the grep output to append,

grep "string" source.txt >> destination.txt

  • Ugh! I knew it would be something simple. Thank you! – Paul Jacobson Dec 29 '17 at 8:26
10

sed can do both jobs (print first line and all lines containing string):

sed -n '1p; /string/p' source.txt > destination.txt

or longer version:

sed -n -e '1p' -e '/string/p' source.txt > destination.txt
  • 3
    Might be worth just adding one or two lines to explain what that does / why it works, to help people in future see if it applies to their problem. – EightBitTony Dec 29 '17 at 8:27
  • If string is multiple words, I take I wrap those in ""? So something like this? sed -n '1p; /"string words"/p' source.txt > destination.txt – Paul Jacobson Dec 29 '17 at 8:36
  • 1
    Yes. It's a regex. – Cyrus Dec 29 '17 at 8:39
  • 4
    @PaulJacobson no, no need for quotes, those will be considered part of the pattern. So /"foo"/ will only match "foo" not foo. If you have a space, include the space: sed -n '1p; /string words/p' or, if you can have more than one, sed -n '1p; /string *words/p' – terdon Dec 29 '17 at 8:40
  • 1
    @PaulJacobson: I missed the quotes. Get rid of them. terdon: Thank you. – Cyrus Dec 29 '17 at 8:42
6

There are many ways of doing this. You can use a single command to get both lines, as @Cyrus's sed solution does. Here are a couple of other tools that can do this:

awk 'NR==1 || /string/' source.txt > destination.txt
perl -ne 'print if /string/ || $. ==1' source.txt > destination.txt

You can of course also run both commands you were running and simply change the second to append to the file as BANJOSA suggested. Alternatively, you could group the two commands in a subshel and redirect the subshell's output to a file:

(sed -n '1p' file; grep string file) source.txt > destination.txt

or

{ sed -n '1p' file; grep string file; } source.txt > destination.txt

So if what you want to do is make a function out of these, it would be as simple as adding this to your shell's initialization file (e.g. ~/.bashrc):

foo(){ sed -n '1p' file; grep string file; } 

And now you can run the function foo to do what you want:

foo source.txt > destination.txt
  • +1 but the commands in the function should use "$1" (or "$@") as the file argument, not hard-coded to file. – cas Dec 30 '17 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.