I have a file in which each line is like this

"372"^""^"2015-09-03 06:59:44.475"^"NEW"^"N/A"^""^0^"105592"^"https://example-url.com"^"example-domain < MEN'S ULTRA < UltraSeriesViewAll (18)"^"New"^"MERCHANT_PROVIDED"

I want to extract the urls in the file -- https://example-url.com

I tried these regex using sed command -- sed -n '/"^"http/,/"^"/p'

But it didn't solve my problem.


You could use this

sed -n 's!^.*\^"\(http[^^]*\)"^.*!\1!p'

The potential gotcha for a beginner to REs is that ^ is an indicator for start of line, so you have to ensure you escape it \^ if you want a literal up arrow at the start of your RE.

The RE pattern match can be explained as follows

  • ^.*\^" -- Match from start of line until we see the last possible up-arrow double-quote ^" that satisfies the rest of the pattern
  • \( -- Start a substitution block that can be substituted as \1
  • http[^^]* -- Match http followed by as many characters that are not ^ as possible
  • \) -- End the substitution block "^.* -- Match double-quote and up-arrow "^, then as much as possible (until end of line)

This entire match is replaced by \1, which is the pattern block starting http


Try this:

echo "372"^""^"2015-09-03 06:59:44.475"^"NEW"^"N/A"^""^0^"105592"^"https://example-url.com"^"example-domain < MEN'S ULTRA < UltraSeriesViewAll (18)"^"New"^"MERCHANT_PROVIDED" | cut -f9 -d^
  • I think you meant to wrap your echo command in single quotes '. Otherwise you lose the double quotes when you echo. – Centimane Sep 4 '15 at 12:21
  • Given statement is also equipped with single quote '. Hence I avoided that. – SHW Sep 4 '15 at 12:23
  • But without the single quote fields like f 10 become arrays. It doesn't impact the url itself really, but it's easy enough to echo the result of this to get rid of the double quotes if needed. – Centimane Sep 4 '15 at 12:29
  • The cut is useful but doesn't remove the double quotes surrounding the extracted field. tr -d '"' maybe? – roaima Sep 4 '15 at 13:20

If your version of grep supports PCRE mode, you could try

grep -Po '(?<="\^")http.+?(?="\^")'

If your URLs always start with http and end with a quote, you can just search for the string http and everything up to the next quote:

  1. grep

    $ grep -o 'https*://[^"]*' file
  2. sed

    $ sed -n 's#.*\(https*://[^"]*\).*#\1#;p' file
  3. Perl

    $ perl -ne 's#.*(https*://[^"]*).*#\1# && print' file
  4. With a slightly different approach, you can use awk as well. Just use -F to set the field delimiter to " and print any field starting with hhtp:

    $ awk -F\" '{for(i=1;i<NF;i++){if($i~/^http/){print $i}}}' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.