0

I have a massive text file with lots of text and URLs in this format:

https://www.website.com/BLAH-BLAH/XX/123567890?@=@81%@38&magic=1&

The last part of the URL "1234567890" is actually a random string of characters, not the same numbers every time.

I'd like to strip all text except the URLs in this format. I'd also like to remove everything after .../XX/1234567890

To clarify, in the end I'd like a single file with the URLs that look like:

https://www.website.com/BLAH-BLAH/XX/123567890

  • You've partially described the URI component 1234567890. Your example shows digits but your wording says characters and numbers. Which is it, and are there always ten of them? What about the BLAH-BLAH/XX: is that invariant? What about the website name? – roaima Feb 1 '17 at 18:53
0

Assuming that your URLs are on new lines, you'll want to use grep to pull them out.

grep "https"

(This searches for and returns all lines containing https. You can expand the search string in the quotes to be longer if the random text contains https in lines not including URLs of interest.)

Edit (misread what you wanted it to end up with): Edit 2: To strip the URLs of anything 10 digits after the first '/XX/' you can use awk:

awk -F'/XX/' '{print $1 "/XX/" substr($2,0,10)}'

This splits the given input by the delimiter ('/XX/' in this case) and returns the first value, I.e. everything before the first '/XX/', and then /XX/, then a substring 10, characters long (starting with first letter) of what's after the /XX/.

You can join these together using pipes as follows (assume your URLs are in a text file called urls.txt):

cat urls.txt | grep "https" | awk -F'?' '{print $1 "/XX/" substr($2,0,10)}'
  • The issue is that I dont know what character will come after the random 10-digit string on the end. It could be a $ or a & or a ? or anything else except a letter or number. – MagicMan23 Feb 1 '17 at 17:28
  • Edited my answer to use a different delimiter, and then a substring of fixed length. If the XX is general if you question (instead of a literal 'XX'), you could expand my answer to use '.com' for instance, and extend the substring value appropriately. – Thonners Feb 1 '17 at 17:40
0
grep -Po 'https?://\S+/\w\w/\w{1,10}\b'   bigfile.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.