I have a file full of long SQL queries, one per line. I need to create a list of unique queries, but most of the queries include parameter values that make using an exact matching tool like
uniq impossible. Is there a way to find unique lines "fuzzily", like
If the queries are predictable enough, maybe you could simply
sed out the parameter values--e.g. if many queries contain equality comparison with numbers,
sed 's/=[[:digit:]]+//g' would remove all the actual numbers, leaving only the column names.
Otherwise, the only really general solutions I can think of are pattern recognition techniques like k-nearest neighbors, which can classify arbitrary lists of strings into clusters based on similarity.
You might have luck running every query through EXPLAIN ANALYZE and finding unique results in the query plans.