I explained a similar situation with plain text files on Grep huge number of patterns from huge file. Many people there said I should, so now I'm migrating my data to a sqlite database:
I have a file from which I extract about 10,000 patterns. Then I check if the database doesn't contain such patterns. If it doesn't, I need to save them externally in
file for further processing:
for id in $(grep ^[0-9] keys); do if [[ -z $(sqlite3 db.sqlite "select id from main where id = $id") ]]; then echo $id >>file fi done
Since I'm new to SQL, I couldn't find a simple way to do this. Also, this loop is useless as it is 20 times slower than what I achieved with
awk on the mentioned URL.
Since the database is huge, keeps growing, and I run this loop very frequently, is it possible to make this faster?