2

I've found only examples on extracting substrings from formatted text like HTML files, but in my case I need to output a list of URLs for example:

... 
https://twitter.com/user1/status/xyza 
https://twitter.com/user1/status/xyzb
https://twitter.com/user1/status/xyzc
https://twitter.com/user2/status/xyza
https://twitter.com/user2/status/xyzb
...

from a unstructured and very large file (+100 MB) this is what my input looks like:

n          3\\n        \\n      \\n  \\n    \\n      \\n      Retweeted\\n    \\n      \\n        \\n          3\\n        \\n      \\n  \\n\\n      \\n  \\n    \\n      \\n        \\n      \\n      Like\\n    \\n      \\n        \\n          5\\n        \\n      \\n  \\n    \\n      \\n        \\n      \\n      Liked\\n    \\n      \\n        \\n          5\\n        \\n      \\n  \\n\\n      \\n\\n        \\n    \\n  \\n      \\n        \\n        More\\n      \\n  \\n  \\n  \\n    \\n    \\n  \\n  \\n    \\n      \\n        Copy link to Tweet\\n      \\n      \\n        Embed Tweet\\n      \\n        \\n  \\n\\n\\n\\n\\n  \\n\\n    \\n\\n      \\n\\n      \\n        \\n  \\n    \\n      \\n  \\n\\n      \\n    \\n\\n  \\n\\n\\n      \\n\\n\\n    \\n      \\n          \\n\\n    \\n        \\n          \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n        \\n        \\n  \\n    \\n  \\n      \\n\\n    \\n        \\n\\n    \\n\\n          Back to top ↑\\n\\n  \\n\\n\\n    \\n  \\n    \\n  \\n\\n\\n  \\n\\n\\n    \\n  \\n    Loading seems to be taking a while.\\n    \\n      Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.\\n    \\n  \\n\\n\\n\\n      \\n    \\n  \\n\\n      \\n    \\n\\n\\n\\n\\n\\n  \\n\\n\\n  \\n    \\n      Suggested by Twitter\\n      \\n        \\n      \\n    \\n   \\n\\n    \\n  \\n    \\n  \\n    \\n    false\\n  \\n  \\n    \\n    \\n  \\n\\n  \\n\\n\\n\\n  \\n      \\n  \\n    \\n      \\n        © 2015 Twitter\\n        About\\n        Help\\n        Terms\\n        Privacy\\n        Cookies\\n        Ads info\\n      \\n    \\n  \\n\\n\\n  \\n\\n\\n\\n      \\n    \\n  \\n\\n\\n    \\n  \\n  \\n\\n\\n\\n    \\n    \\n  \\n\\n  \\n\\n  \\n\\n    \\n  \\n\\n  \\n    \\n\\n\",\"meta_tags\":[{},{\"content\":\"0; URL=https://mobile.twitter.com/i/nojs_router?path=%2FTerriBauman%2Fstatus%2F680996161843380224\"},{\"name\":\"robots\",\"content\":\"NOODP\"},{\"name\":\"msapplication-TileImage\",\"content\":\"//abs.twimg.com/favicons/win8-tile-144.png\"},{\"name\":\"msapplication-TileColor\",\"content\":\"#00aced\"},{\"name\":\"swift-page-name\",\"content\":\"permalink\"},{\"content\":\"article\"},{\"content\":\"https://twitter.com/TerriBauman/status/680996161843380224\"},{\"content\":\"Terri Bauman on Twitter\"},{\"content\":\"https://pbs.twimg.com/media/BcaVtMKCEAAyz9f.jpg:large\"},{\"content\":\"true\"},{\"content\":\"“Social Media Jobs: https://t.co/NDDK4WaRA4 Please Retweet to spread words #OnlineJobs #Jobs”\"},{\"content\":\"Twitter\"},{\"content\":\"2231777543\"}],\"links\":[\"https://twitter.com/\",\"https://twitter.com/about\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/#supported_languages\",\"https://twitter.com/?lang=id\",\"https://twitter.com/?lang=msa\",\"https://twitter.com/?lang=cs\",\"https://twitter.com/?lang=da\",\"https://twitter.com/?lang=de\",\"https://twitter.com/?lang=en-gb\",\"https://twitter.com/?lang=es\",\"https://twitter.com/?lang=fil\",\"https://twitter.com/?lang=fr\",\"https://twitter.com/?lang=it\",\"https://twitter.com/?lang=hu\",\"https://twitter.com/?lang=nl\",\"https://twitter.com/?lang=no\",\"https://twitter.com/?lang=pl\",\"https://twitter.com/?lang=pt\",\"https://twitter.com/?lang=ro\",\"https://twitter.com/?lang=fi\",\"https://twitter.com/?lang=sv\",\"https://twitter.com/?lang=vi\",\"https://twitter.com/?lang=tr\",\"https://twitter.com/?lang=el\",\"https://twitter.com/?lang=ru\",\"https://twitter.com/?lang=uk\",\"https://twitter.com/?lang=he\",\"https://twitter.com/?lang=ar\",\"https://twitter.com/?lang=fa\",\"https://twitter.com/?lang=mr\",\"https://twitter.com/?lang=hi\",\"https://twitter.com/?lang=bn\",\"https://twitter.com/?lang=gu\",\"https://twitter.com/?lang=ta\",\"https://twitter.com/?lang=kn\",\"https://twitter.com/?lang=th\",\"https://twitter.com/?lang=ko\",\"https://twitter.com/?lang=ja\",\"https://twitter.com/?lang=zh-cn\",\"https://twitter.com/?lang=zh-tw\",\"https://twitter.com/login\",\"https://twitter.com/account/begin_password_reset\",\"https://twitter.com/signup\",\"https://twitter.com/TerriBauman\",\"https://pbs.twimg.com/profile_images/598412523734310913/t3ettYkj.jpg\",\"https://pbs.twimg.com/profile_images/598412523734310913/t3ettYkj.jpg\",\"https://twitter.com/TerriBauman\",\"https://twitter.com/TerriBauman\",\"https://twitter.com/TerriBauman\",\"https://twitter.com/TerriBauman\",\"https://twitter.com/hashtag/Entrepreneur?src=hash\",\"https://twitter.com/hashtag/SocialMediaExpert?src=hash\",\"https://twitter.com/hashtag/SocialMediaMarketer?src=hash\",\"https://twitter.com/hashtag/BusinessOwner?src=hash\",\"https://twitter.com/hashtag/InternetMarketer?src=hash\",\"https://twitter.com/hashtag/SocialMediaJobs?src=hash\",\"https://t.co/ZciT91kZwP\",\"https://twitter.com/about\",\"http:////support.twitter.com\",\"https://twitter.com/tos\",\"https://twitter.com/privacy\",\"http:////support.twitter.com/articles/20170514\",\"http:////support.twitter.com/articles/20170451\",\"https://twitter.com/#\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"http://support.twitter.com/forums/26810/entries/78525\",\"http:////dev.twitter.com/docs/embedded-tweets\",\"http:////dev.twitter.com/docs/embedded-tweets\",\"https://twitter.com/account/begin_password_reset\",\"https://twitter.com/signup\",\"https://twitter.com/signup\",\"https://twitter.com/login\",\"http://support.twitter.com/articles/14226-how-to-find-your-twitter-short-code-or-long-code\",\"https://twitter.com/TerriBauman/status/680996164058001408\",\"https://twitter.com/TerriBauman/status/680977383365578752\",\"https://twitter.com/TerriBauman\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://t.co/NDDK4WaRA4\",\"https://twitter.com/hashtag/OnlineJobs?src=hash\",\"https://twitter.com/hashtag/Jobs?src=hash\",\"https://t.co/SJvkM1yWUI\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/TerriBauman/status/680996161843380224\",\"https://twitter.com/cakafete\",\"https://twitter.com/KassemAlYateem\",\"https://twitter.com/Worldspacetech1\",\"https://twitter.com/ElisaBW\",\"https://twitter.com/patrickarrelle\",\"https://twitter.com/AcousticsPro1\",\"https://twitter.com/#\",\"http://status.twitter.com\",\"https://twitter.com/about\",\"http:////support.twitter.com\",\"https://twitter.com/tos\",\"https://twitter.com/privacy\",\"http:////support.twitter.com/articles/20170514\",\"http:////support.twitter.com/articles/20170451\"]}"},{"url":"http://status.twitter.com/page/2","result":"{\"date_crawled\":\"2015-12-27T10:01:58Z\",\"title\":\"Twitter Status\",\"lossyHTML\":\"\\n\\n\\r\\n\\r\\n    \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n            \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n        \\r\\n                \\r\\n        \\r\\n\\r\\n        \\r\\n        Twitter Status\\r\\n        \\n\\r\\n        \\r\\n         \\r\\n\\r\\n        \\r\\n\\r\\n    \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\r\\n    \\r\\n\\r\\n\\r\\n\\r\\n\\r\\n        \\r\\n\\r\\n\\r\\n\\r\\n    \\r\\n    \\r\\n        \\r\\n            \\r\\n                \\r\\n                    Updates on the status of the Twitter service.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nRelated Links\\r\\nOfficial Company Blog\\r\\n\\r\\nOfficial Help Documents\\r\\n\\r\\nDeveloper Community\\r\\n\\r\\n\\r\\n\\r\\n                    Archive\\r\\n\\r\\n\\r\\n\\r\\n \\r\\n                    Powered by Tumblr\\r\\n                \\r\\n\\r\\n                \\r\\n            \\r\\n            \\r\\n\\r\\n\\r\\n            \\r\\n                \\r\\n                    \\r\\n       

I've been trying to do:

grep 'https://' input.txt | grep 'status' >> output.txt

I've seen examples of usage of sed and awk but in addition of being extremely hard to understand they almost always are based on column selection which in my case is not posible.

  • 1
    Maybe it's just me, but I don't see URLs in your input file? – Jeff Schaller Dec 27 '15 at 16:25
  • scroll to the middle of the text, you'll see them – J. Bend Dec 27 '15 at 16:39
3

Try this with GNU grep to get URLs with two slashes:

grep -o 'http[s]*://[^/][^\\]*' file

URLs with two and more slashes:

grep -o 'http[s]*://[^\\]*' file

Recommended reading: The Stack Overflow Regular Expressions FAQ

[s]*: the star quantifier (*) means that the preceding expression can match zero or more times. Here the preceding expression can be any character from the character class (marked with brackets) which only contains a s. It is easier to use s*.

[^\\]*: matches any character except a backslash zero or more times. I escaped the backslash with a backslash to prevent escaping ].

  • Nice! Thank you. Now I want to understand: the [s]* says optionally include the character s, right? and the [^\]* I have no idea what it does, can you explain me please? – J. Bend Dec 27 '15 at 16:47
  • I've updated my answer. – Cyrus Dec 27 '15 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.