Using Linux,

  1. I want to sort and identify duplicate addresses from the file which contains addresses (one address in one single line) but no specific delimiter.

  2. Redirect list of each duplicate address in a file with filename as the duplicate address itself.

I tried all using grep -i "AX.*BY.*CZ" from the address criteria, sed, awk, agrep

but the problem is that every time it searches only the presence of string in the file but not in the sequential way like AX and then BY and then CZ. That's why lot of anomaly in the output.

Please let me know how to do it?

closed as unclear what you're asking by Jeff Schaller, steve, RomanPerekhrest, Archemar, Romeo Ninov Dec 23 '17 at 16:19

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    kindly please insert INPUT/OUTPUT in order to help you. – αԋɱҽԃ αмєяιcαη Dec 23 '17 at 8:35
  • 1
    please insert a small sample of your input and expected output into your question. Linking to a google drive document (or other external resource) is NOT the same as inserting it into your question - it is problematic for most of the same reasons that posting images is problematic (see meta.unix.stackexchange.com/questions/4086/…) – cas Dec 23 '17 at 9:58
  • 4
    You should anonymise your list too - that .csv file you posted looks like it contains the addresses for about 20,000 people and businesses. you've just compromised their privacy and helped to enable identify theft. – cas Dec 23 '17 at 10:16
  • Are the lines with the same address identical or can the delimiter be different? – Hauke Laging Dec 23 '17 at 14:41

Looking at your address-list.csv file, this appears to be a case of Garbage In, Garbage Out.

I doubt if you can do what you want with grep or sed. In fact, I doubt if there's any tool or scripting language that you can use.

You may be able to do something vaguely useful with awk or perl or similarly capable language but there doesn't even seem to be a consistent format to the addresses - most have what looks like a six-digit postal code, but not all lines do. Many seem to have the city name and the country as the 2nd-last and last fields (e.g. "Kolkata INDIA") but some skip the country, some have the city name before the postal code, some have a town name followed by the street address, and some have neither city nor country. Some have the personal/family/business name before the street address, some have it after, some don't have a name.

In short, the entire file is a mess that can't be processed automatically without a huge set of street names, city/town/suburb names, postal codes, state names, etc....and even if you had such a list, trying to match all of those against substrings of each line would be horrendously complex.

Processing this requires intelligence (human or artificial), not a regular expression or other formula.

This is the problem with trying to process free-form text that doesn't even pretend to have any structure. With some of the oddities in the file, I'd say that this .csv file goes way beyond merely free-form and into "bizarre-form".

And this is why you never let users enter addresses in free-form text fields. You give them a standard set of fields for the various parts that make up an address - name, street number, street name, suburb, city, post code, country, etc. Some users will still manage to screw it up somehow but the vast majority of your data will be in a usable form...and the remainder can either be fixed by hand or just discarded.

IMO, your best option is to manually edit it with a text editor into some standardised form that can be easily processed with line-oriented tools like grep. vi is particularly good at such repetitive editing tasks - modal editors excel at such tasks, and a skilled person can finish in a tiny fraction of the time that it would take using a simple, modeless editor like nano.

I would suggest something like:

name:street address:postal code:city:country:comments

NOTE: there aren't many : characters in the .csv file and those that are there can be removed or replaced with . characters. Also, country may be optional (it seems to be a list of entirely Indian addresses) but it doesn't hurt to have it - your code could default to "India" if the country field is empty.

Alternatively, maybe you can try to find an address list that isn't garbage...but i guess you're trying to create one because this is all you have.

  • 1
    +1 for such a detailed, thoughtful answer to a question that is basically hopeless. :) – B Layer Dec 23 '17 at 10:54
  • it might be possible to train an AI on a list of well-formatted addresses (like a phone book) and then get it to make sense of this essentially-junk file. it's probably easier (and with a more certain result) to just edit the file and fix it. or pay someone to do it. – cas Dec 23 '17 at 10:58
  • I imagine it is possible with AI. And every smart business person in the world would hire some dude to do it manually, instead. ;) – B Layer Dec 23 '17 at 11:02
  • at first, I thank cas and B Layer for such quick response. The data was provided by one of my friend who worked on it for months to gether this much information (address) from various resources and very hopeful while giving it to me (thought being a linux enthusiast). I too agree that the data was purely raw and without any specific pattern but I must believe that there are many tasks which seem difficult and provide a different taste of satisfaction when we do it successfuly. I have performed the following operations on this file to get result upto maximum of what I can. – ESSPEE Dec 24 '17 at 3:10
  • At first I made a copy of this file cp address-list.csv file.csv Then replaced all non-alphanumeric characters with space within file using command sed -i "s/[^A-Z,^0-9]/ /Ig" file.csv Then removed all leading and trailing spaces sed -i "s/^[ \t]*//;s/[ \t]*$//" file.csv Then replaced instances of multiple spaces to single space sed -i "s/[[:space:]]\+/ /g" file.csv Now, I selected first 8 field space delemited and redirected to a new file called file2.csv cat file.csv | cut -d" " -f1-8 > file2.csv – ESSPEE Dec 24 '17 at 8:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.