0

I have a file that contains various different information in an unformatted way. I have been asked to create a list of all .com , .org and .net .de domain names referenced in the text. Each entry should only contain the domain.extension (stackexchange.com NOT http://www.stackexchange.com/index) and should only appear once in the list.

Here is a sample of the data:

230 User astalavista.com logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls -la
227 Entering Passive Mode (212,254,194,163,2,188)
150 Opening BINARY mode data connection for 'file list'.
dr-x------   1 root users         4096 Jun  4 06:13 astalavista.com
226 Transfer complete.
ftp> cd astalavista.com
250 CWD command successful.
ftp> ls -la
227 Entering Passive Mode (212,254,194,163,2,189)
150 Opening BINARY mode data connection for 'file list'.
-rw-rw-rw-   1 astalavista.com users     23410936878 Apr 29 22:10 
09-04-28-astacom_full.tar
-rw-rw-rw-   1 astalavista.com users     20617651590 Apr 29 14:18 
09-04-28-astacom_full.tar.bz2
-rw-rw-rw-   1 astalavista.com users        88287111 Apr 29 15:57 
09-04-29-astacom_sql_full.sql.tar.bz2
-rw-rw-rw-   1 astalavista.com users     26413034040 May  2 00:21 
09-05-01-astacom-Public_HTML.tar
rw-rw-rw-   1 astalavista.com users       277843549 May  1 17:29 
09-05-01-astacom-SQL_Dump.tar
[snip]
226 Transfer complete.
ftp> mdelete *
ftp> ls -la
227 Entering Passive Mode (212,254,194,163,2,193)
150 Opening BINARY mode data connection for 'file list'.
226 Transfer complete.
ftp>

So far I have used egrep to find all the occurences of .com , .net , .org , .de .

egrep '\.\de|\.\com|\.\org|\.\net' hacklog1 

I've been trying to use sed to remove the rest of the unwanted data to no avail. Could someone suggest a command syntax to help me.

  • have you tried egrep -o '\S*(\.de|\.com|\.org|\.net)' hacklog1? – DopeGhoti Nov 29 '16 at 22:26
  • That still leaves the unwanted area at the start for example it leaves me with results such as <title>ASTALAVISTA.com - <link>astalavista.com - <generator>Astalavista.com - milw0rm.com – Joe Nov 29 '16 at 22:28
  • 1
    Your false positive doesn't even appear in your sample input, so I don't see how that's possible. Even so: egrep -o '[a-zA-Z0-9\.]+(\.com|\.net|\.de|\.org)' input – DopeGhoti Nov 29 '16 at 22:32
  • Apologies, I just took a random sample from a much larger file, there are URLs in many different formats throughout the file. – Joe Nov 29 '16 at 22:34
1

Try something like:

egrep -o '[0-9A-Za-z_-]+\.(com|org|net|de)\b'

It should be fairly easy to tune the characters allowed before the TLD and the list of recognized TLDs to your liking.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.