2

I have a problem that i can not think how to solve. I have a directory that contains 1000's of files which each contain a URL with a URL parameter:

File 1:

example.com/car/?wheel=tyre

File 2:

example.com/car/?wheel=rim

File 3:

test.com/food/?fruit=apple

File 4:

test.com/food/?fruit=banana

File 5:

test.co.uk/car/interior=chair

File 6:

example.co.uk/car/interior=chair

I want to remove duplicate files that contain the same domain name and parameter but keep the first. The item inside the parameter eg. tyre, rim, apple, bananna and chair should be ignored. so the desired output is below:

File 1:

example.com/car/?wheel=tyre

File 3:

test.com/food/?fruit=apple

File 5:

test.co.uk/car/interior=chair

File 6:

example.co.uk/car/interior=chair

File 2 has been removed because it is the same domain name and URL parameter name as File 1. File 4 has been removed because it is the same domain name and URL parameter name as File 3. File 5 and 6 are kept, because they use different domains even though they use the same URL parameter.

I am thinking i need to first iterate through all files in the directory:

FILES=data/urls/*
for i in $FILES
do
  cat $i | *Check for matching domain, then matching parameter*
else
 rm $i
fi
done

or maybe search for a domain name using a for loop, then a nested for loop to check the parameter using uniq. However i am not sure how to do this.

Any ideas how i can do this?

EDIT: I have added all url's to one file to make them easier to work with using:

sed -i -e '$a\' *.txt
cat * > all.txt

Edit 2: Some URL's contain multiple folders in the path eg.

www.example.com/1/2/3/4/5/?l=123
www.example.com/1/2/3/?1=AAA
  • This is not a very efficient way to store such things. Nor is it easy to work with them. I'd rather keep them in a database. – Tomasz Jan 20 at 18:57
  • @Tomasz I agree, however i have to deal with the data as it is, i may pipe all files into 1 file to make it easier. – jonny b Jan 20 at 19:02
  • Files 5 and 6 miss ? that introduces the parameter. Just for the record. – Tomasz Jan 20 at 19:58
  • Yeah that makes it more difficult because they all have a different pattern – jonny b Jan 20 at 20:07
  • If the files have different content then they're not duplicates. – roaima Jan 20 at 20:38
2

A little Bash script like this should do:

#!/bin/bash

declare -A a                             # declare associative array 
for i in *; do                           # loop through files in directory

  read -r url < "$i"                     # read the url from the file
  domain=${url%%/*}                      # delete longest match from back
  param=${url##*/}                       # delete longest match from start
  param=${param%%=*}                     # delete "=value" from param
  k=$domain$param                        # construct key

  if [[ -n $k && -z ${a[$k]} ]]; then    # check its prior use as key k
    a[$k]=1                              # new key: mark it as such
  else                                   # known/faulty key: delete the file
    echo Delete "$i"                     # Replace with: rm "$i" if satisfied
  fi
done
  • This is giving me the following error: ./test.sh: line 6: a: bad array subscript ./test.sh: line 7: a[$k]: bad array subscript – jonny b Jan 20 at 20:29
  • @jonnyb Can you insert an "echo $k" above the if-statement (below the k-assignment) to see what the bad key is? I don't have a bad key for the files you showed in your post. – ozzy Jan 20 at 20:31
  • Its not giving me the output of echo $k for some reason, its just giving me the same output as before – jonny b Jan 20 at 20:36
  • actually , its showing a new empty line in the output so maybe its a space? – jonny b Jan 20 at 20:37
  • @jonnyb Maybe it does. Maybe you have an empty file. Can you insert an "echo "$i" before the k-assignment? And then check to see if that file actually matches your data specification... – ozzy Jan 20 at 20:38
1

With all lines in one file, you can use this Perl code:

perl -nle '($d,$p)=m{^(\S+?)/\S+/\??(\w+)=}; $a{"$d$p"}=$_ if "$d$p";
END {print $a{$_} for (keys %a)}' input

input is the data file, but you can pipe into this Perl command instead. The working is simple. Regex is first used to isolate the domain and the first param of each entry. Then the whole line is written to a dictionary, where the earlier isolated parts are the key. Any earlier value is overwritten. In the end the hash is printed.

It produces this output:

test.co.uk/car/interior=chair
www.example.com/1/1/?l=spanish
example.com/car/?wheel=rim
www.example.com/1/1/?selectedTab=live&selectedStream=1
example.co.uk/car/interior=chair
test.com/food/?fruit=banana

From this input:

example.com/car/?wheel=tyre
example.com/car/?wheel=rim
test.com/food/?fruit=apple
test.com/food/?fruit=banana
test.co.uk/car/interior=chair
example.co.uk/car/interior=chair
www.example.com/1/1/?l=thai 
www.example.com/1/1/?l=thai
www.example.com/1/1/?l=ukrainian
www.example.com/1/1/?l=turkish
www.example.com/1/1/?selectedTab=live&selectedStream=1
www.example.com/1/1/?l=ukrainian
www.example.com/1/1/?l=turkish
www.example.com/store/?l=turkish
www.example.com/1/1/storecommon.js?v=aHpFwmVHuavD&l=dutch
www.example.com/1/1/2/0/?l=english
www.example.com/1/1/9/0/?l=english
www.example.com/1/1/?l=spanish

There is one glitch with ?v=aHpFwmVHuavD&l=dutch, which comprises of two parameters. They possibly ought to be broken down and processed one by one, which makes things more complex. Make sure this is not the case. Also note that domain.com and www.domain.com are not effectively the same.

  • Thanks, this nearly works however it doesnt delete some entries that it should – jonny b Jan 20 at 20:30
  • @jonnyb Can you give an example? – Tomasz Jan 20 at 20:40
  • Here is some of the examples that remain: www.example.com/1/1/?l=thai www.example.com/1/1/?l=thai www.example.com/1/1/?l=ukrainian www.example.com/1/1/?l=turkish www.example.com/1/1/?selectedTab=live&selectedStream=1 www.example.com/1/1/?l=ukrainian www.example.com/1/1/?l=turkish www.example.com/store/?l=turkish www.example.com/1/1/storecommon.js?v=aHpFwmVHuavD&l=dutch www.example.com/1/1/2/0/?l=english www.example.com/1/1/9/0/?l=english www.example.com/1/1/?l=spanish – jonny b Jan 20 at 20:57
  • I think your script works as i asked, is my error in commuicating that some URL's have more then one folder eg. example.com/../../../../../ i will modify the question – jonny b Jan 20 at 21:05
  • @jonnyb Your list includes multiple parameters as in www.example.com/1/1/storecommon.js?v=aHpFwmVHuavD&l=dutch, so it's not going to be easy until you simplify their handling. – Tomasz Jan 20 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.