I have a file full of long SQL queries, one per line. I need to create a list of unique queries, but most of the queries include parameter values that make using an exact matching tool like
uniq impossible. Is there a way to find unique lines "fuzzily", like
If the queries are predictable enough, maybe you could simply
sed out the parameter values--e.g. if many queries contain equality comparison with numbers,
sed 's/=[[:digit:]]+//g' would remove all the actual numbers, leaving only the column names.
Otherwise, the only really general solutions I can think of are pattern recognition techniques like k-nearest neighbors, which can classify arbitrary lists of strings into clusters based on similarity.