I have a file full of long SQL queries, one per line. I need to create a list of unique queries, but most of the queries include parameter values that make using an exact matching tool like uniq
impossible. Is there a way to find unique lines "fuzzily", like agrep
?
2 Answers
If the queries are predictable enough, maybe you could simply sed
out the parameter values--e.g. if many queries contain equality comparison with numbers, sed 's/=[[:digit:]]+//g'
would remove all the actual numbers, leaving only the column names.
Otherwise, the only really general solutions I can think of are pattern recognition techniques like k-nearest neighbors, which can classify arbitrary lists of strings into clusters based on similarity.
You might have luck running every query through EXPLAIN ANALYZE and finding unique results in the query plans.
uniq
return?). It would definitely help to have a sample input and expected output.uniq
,agrep
, etc. since there are 2 paths to take for you, there seems to be 2 potential Q's. Just don't copy/paste this verbatim there or you'll likely get them both closed as cross-posts which isn't discouraged on SE sites. Also you might want to show some of these queries so we can better assist you.