I have a problem that i can not think how to solve. I have a directory that contains 1000's of files which each contain a URL with a URL parameter:
I want to remove duplicate files that contain the same domain name and parameter but keep the first. The item inside the parameter eg. tyre, rim, apple, bananna and chair should be ignored. so the desired output is below:
File 2 has been removed because it is the same domain name and URL parameter name as File 1. File 4 has been removed because it is the same domain name and URL parameter name as File 3. File 5 and 6 are kept, because they use different domains even though they use the same URL parameter.
I am thinking i need to first iterate through all files in the directory:
FILES=data/urls/* for i in $FILES do cat $i | *Check for matching domain, then matching parameter* else rm $i fi done
or maybe search for a domain name using a for loop, then a nested for loop to check the parameter using uniq. However i am not sure how to do this.
Any ideas how i can do this?
EDIT: I have added all url's to one file to make them easier to work with using:
sed -i -e '$a\' *.txt cat * > all.txt
Edit 2: Some URL's contain multiple folders in the path eg.