1

I have tried every possible (to my poor knowledge) combination of grep and sed commands, but fail to extract URLs with the following pattern (Google Alert e-mails in plaintext):

"url": "https://www.google.com/url?rct=3Dj\u0026sa=3Dt\u0026url=3Dhtt=
p://abcnews.go.com/US/wireStory/judge-orders-forfeiture-cartel-money-launde=
ring-case-44765120\u0026ct=3Dga\u0026cd=3DCAEYACoTNzAxNDE5ODc4MzMzMTc5OTA4O=
TIaYjdkMGIxMjNmMjc0YWM4ODpjb206ZW46VVM\u0026usg=3DAFQjCNHKeTb3brU2sr0qOpXXJ=
fuW9Nfntg"

Obviously, what I want to extract is:

http://abcnews.go.com/US/wireStory/judge-orders-forfeiture-cartel-money-laundering-case-44765120

So I need to extract what is between "url=3D" and "\".

I have tried all kinds of grep and sed variations, but nothing works.

I would be very grateful if someone could help me figure this out.

PS: I know that once the URLs extracted I'll have to deal with the = characters, but one problem at a time :)

  • 1
    Maybe you should deal with "=\n" first, to join it into a single line, and then you can sed it (or do that joining in the sed program). – Ralph Rönnquist Feb 7 '17 at 21:54
  • thanks, the problem is that when performing on a file with many such patterns, it doesn't work (it works only if I put this pattern only in a single file or as an echo input) – serge Feb 8 '17 at 19:46
  • Hmm, apparently your sed doesn't have -z, then. If it had, you wouldn't end up with any lines ending in '='. You may need to highlight that in your question. Check man sed and sed --version – Ralph Rönnquist Feb 8 '17 at 22:04
  • sed version is sed (GNU sed) 4.2.2 and I do have -z (just checked it with man) – serge Feb 8 '17 at 22:13
  • Due to the complexity of the file formatting, I think what I should be able to do is: print all lines that contains url=3D BUT print only what comes AFTER url=3D, AND print UNTIL the line that contains u0026ct BUT print only what comes before u0026ct. This way I may be able to escape the problem caused by the formatting, which causes sed and grep to stop at each end of line? But I have no idea of how to do it. As of now, I managed to print everything it matches the line containing u0026ct: awk '{print} /u0026ct/ {exit}' INBOX > output.txt ... – serge Feb 8 '17 at 22:23
1

You can use a command line like the following for the processing:

cat INBOX | sed -z -e 's/=\n//g' | \
   sed -e 's/.*u0026url=3D//;t a;d;:a' -e 's/\\u0026ct=3D.*//'

The first sed step is for joining the lines the ends with "=" with their succeeding line, and thus in particular making one-liners of the interesting lines.

The second sed step firstly both reduces interesting lines by removing their head part, and discards any lines without that head, and secondly removes the tail parts of the target lines.

  • did you remember the -z to the first sed? – Ralph Rönnquist Feb 8 '17 at 21:35
  • yes, I did exactly how you put it. – serge Feb 8 '17 at 21:54
  • 1
    Peculiar; so doing the first sed only result in lines ending with =? I.e. doesn't join? Maybe your line ends are \r then (you are using a mac?) Try replacing \n with 'r'... or you're using windows? replace '\n with \r\n... – Ralph Rönnquist Feb 8 '17 at 22:18
  • I'm using Linux – serge Feb 8 '17 at 22:26
  • Yes, but the file might have, say, Window's line endings. You might replace \n with \r\?\n so as to treat both window's and linux line endings as such. – Ralph Rönnquist Feb 8 '17 at 22:29
0

can you try with this command

awk -F"3D" '{print $4}' input.txt | sed "s/\\\u.*//"
0

I'm not certain how you are getting the alerts but I will provide an example of how to do it if the alerts were in a simple text file. I would deal with the "=" first by using tr then I would use Pearl lookarounds with grep as follows ...

cat input.txt | tr --delete '=\n'| grep -oP '(?<=url3D).*?(?=\\u0026)' input.txt

The output using your sample is

http://abcnews.go.com/US/wireStory/judge-orders-forfeiture-cartel-money-laundering-case-44765120
  • I perform the extractions on a single INBOX file (thunderbird). Thanks for your suggestion, it works flawlessly if performed on a file containing ONLY this sample, however when performed on the INBOX file (with many such URLs), it doesn't work at all (output is the same as the input). I really don't know what's wrong. As I have already mentioned, the pattern is always the same (URLs are located between u0026url=3D and \u0026ct=) – serge Feb 8 '17 at 19:52
  • the weird thing is that until it encounters another instance of "url": , the command you suggest works as intended and ignores all non-relevant strings (lot of html code and useless text), but whenever it encounters the next "url":, it breaks (stops extracting and just puts the rest of the INBOX file content after the first extracted URL) – serge Feb 8 '17 at 21:13
  • Well we are part of the way there then. :) If you would like to edit your question and provide a more complete example (with 2+ urls) I'd be happy to take a look – rcjohnson Feb 9 '17 at 8:41
0

Question solved using the suggestion of Ralph Rönnquist

cat INBOX | sed -z -e 's/=\r\?\n//g' | \ sed -e 's/.*u0026url=3D//;t a;d;:a' -e 's/\\u0026ct=3D.*//' > output.txt

It takes a long time to compute, but it does extract the URLs correctly.

Thanks a lot everybody for your assistance!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.