0

I have a file with multiple lines of URLs. There are URLs I am not interested in processing or viewing and want them to be ignored. I only want anything else but them to show as an output.

Here is the command I have so far:

grep 'http://' data.txt | sed 's/.*\(http:.*\)\".*/\1/'

I want to exclude urls including the following:

http://schemas.openxmlformats.org...

I am very new to this and appreciate your help sincerely.

Update: Here is the file I am processing:

Relationships Xmlns             : http://schemas.openxmlformats.org/package/2006/relationships
Relationships Relationship Type : http://schemas.openxmlformats.org/officeDocument/2006/relationships/extended-properties
Style Sheet Xmlns               : http://schemas.openxmlformats.org/spreadsheetml/2006/main
Relationships Xmlns             : http://schemas.openxmlformats.org/package/2006/relationships
Relationships Relationship Type : http://schemas.openxmlformats.org/officeDocument/2006/relationships/sharedStrings
Workbook Xmlns                  : http://schemas.openxmlformats.org/spreadsheetml/2006/main
Relationships Xmlns             : http://schemas.openxmlformats.org/package/2006/relationships
Relationships Relationship Type : http://schemas.openxmlformats.org/officeDocument/2006/relationships/hyperlink
Relationships Relationship Target: http://www.yahoo.com/
Worksheet Xmlns                 : http://schemas.openxmlformats.org/spreadsheetml/2006/main
Sst Xmlns                       : http://schemas.openxmlformats.org/spreadsheetml/2006/main
Types Xmlns                     : http://schemas.openxmlformats.org/package/2006/content-types
Properties Xmlns                : http://schemas.openxmlformats.org/officeDocument/2006/extended-properties

I want the http://www.yahoo.com to be extracted alone and rest ignored because they contain schemas.openxmlformats

  • It would be nice to have sample input and expected output to test. – Olorin Mar 8 '18 at 2:31
  • @Olorin Added the data above for your reference. – ksa_coder Mar 8 '18 at 2:35
1

I think this can be done with sed alone:

sed -n '\,http://schemas.openxmlformats.org,!s/.*\(http:.*\).*/\1/p'
  • -n disables automatic printing of text, so only selected lines can be printed.
  • \,http://schemas.openxmlformats.org,! runs the following command only on lines not matching (hence ! at the end) http://schemas.openxmlformats.org. I have used , instead of / as the regex delimiter here, hence the \, at the start. This reduces the need for escaping \ in the pattern.
  • The s command is the same as yours, but I use a p after it to print the line, which now contains just the URL.

I am assuming just one URL per line.

Removing the extra quote gives me the output correctly:

$ sed -n '\,http://schemas.openxmlformats.org,!s/.*\(http:.*\).*/\1/p' inpu-file
http://www.yahoo.com/
  • I tried this along with adding cat data.txt | before it but nothing showed. – ksa_coder Mar 8 '18 at 2:36
  • 1
    @ksa_coder your s command in sed doesn't match the input provided. It has " being matched, when there's no quotes in your input text. – muru Mar 8 '18 at 2:58
1

Using grep with -v option give you the ability to select non-matching lines. for example, giving a file file.txt with this content:

first line
second line
third line
fourth text 

Using this command:

grep "line" file.txt | grep -v "second"

The result will be:

first line
third line

If you want to exclude multiple words at the same time you can use a regex like that:

grep "line" file.txt | grep -vE "(second|first)"

The result will be:

    third line

After question Update:

For this case you can use either of these methods:

  1. grep 'http://www.yahoo' data.txt | sed 's/.*\(http:.*\)/\1/'
  2. grep 'http://' data.txt | sed 's/.*\(http:.*\)/\1/' | grep yahoo

The first way will give you only www.yahoo.

The second one will give you all the URL that contains yahoo as a word in it.

For extracting all URLs but some:

grep 'http://' data.txt | sed 's/.*\(http:.*\)/\1/' | grep -vE "(openxmlformats|<Another URL to exclude>)"
  • Yes this did it partially because I am getting the whole line instead of only the URL. How can I remove the Relationships Relationship Target part of the line in this case? Of course keeping in mind there might be more than one record matching – ksa_coder Mar 8 '18 at 2:42
  • This will only work if knowing the value of URL to be extracted. It could be yahoo.com or it could be any other domain as long as its not the excluded value. – ksa_coder Mar 8 '18 at 2:50
  • @ksa_coder your example was not so clear, check the edit again please – Wissam Roujoulah Mar 8 '18 at 2:53
  • I added multiple lines in the file that I want to be extracted and not excluded. It got the right number of urls (3 in total) but all of them have the same value (other values are being discarded). Any idea why? – ksa_coder Mar 8 '18 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.