I'm taking my Linux+ midterm and I'm stuck on a question where I'm supposed to copy the head section including the tags and output it to a file. My midterm is due on the 5th so it would really help if someone showed me the correct code to use when greping for the head section and tags. I tried: curl www.hackthissite.org: grep "<head> > ~/data/public/myfirstname\ mylastname/head.txt but all it would say when I cat the file is<head>, doesn't include the head section or tag. So what should I type to get the correct output to the file?


It's a bit more complex than what you're trying to do.

First, there are some syntax issues on your command. This curl www.hackthissite.org: grep "<head> > ~/data/public/myfirstname\ mylastname/head.txt should be:

curl www.hackthissite.org | grep "<head>" > ~/data/public/myfirstname\ mylastname/head.txt

But even though it wouldn't do what you want, because you're only greping the head opening tag, but not what comes in between it and the closing tag.

I came up with this:

curl www.hackthissite.org > TEMPORARYFILE.txt; grep -A $(($(grep -n "</head>" TEMPORARYFILE.txt | cut -d: -f1) - $(grep -n "<head>" TEMPORARYFILE.txt | cut -d: -f1))) "<head>" TEMPORARYFILE.txt > ~/data/public/myfirstname\ mylastname/head.txt; rm TEMPORARYFILE.txt

So, by parts:

grep -n "</head>" TEMPORARYFILE.txt | cut -d: -f1

This will get the number of the line where the closing tag is. The same applies to grep -n "<head>" TEMPORARYFILE.txt | cut -d: -f1, but for the opening tag.

Then we have $(($(grep -n "</head>" TEMPORARYFILE.txt | cut -d: -f1) - $(grep -n "<head>" TEMPORARYFILE.txt | cut -d: -f1))), which is supposed to calculate how many lines there are between the opening tag and the closing tag.

This is used with the -A option of grep, which gives us control of how many lines after the match we want printed. So it will search for the opening head tag and print all the lines between it and the closing tag.

  • It give me a syntax error for "-", says file.txt has no file or directory, ":" is an invalid argument for "--directories" and gives me a list of valid arguments, and it gives me different errors everytime I try to use your code. – Omar Nov 4 '16 at 3:03
  • Oh, my bad, I used a file to test the code (instead of downloading the page every time) and forgot to change it so it only used the output of curl. I'll fix it. – IanC Nov 4 '16 at 3:17
  • I fixed it. It should now create a temporary file on the directory you run it into with the contents of the curl request and delete it after it's done (so even though it's unlikely, be sure you run it in a folder that don't have a file named TEMPORARYFILE.txt). – IanC Nov 4 '16 at 3:33
  • It works for a few seconds, then errors out and says invalid arguments "-f1' for "--directories" valid arguments are read, recurse, and skip. Usage grep [OPTION]... PATTERN [FILE]... Try grep --help for more information. bash: -; syntax error: operand expected (error token is "- "). That's the only error I get now. – Omar Nov 4 '16 at 13:39
  • I've fixed your code, -f1 is supposed to be -f 1, after that, it worked! Thanks so much for your help, I'm sure to get an A on my midterm now! – Omar Nov 4 '16 at 15:46

Using gawk:

curl -s www.hackthissite.org | awk '/<\<head>/,/<\/head>/ {print}'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.