I have two files and I want to generate a new file with the lines found in file2 but not in file1. The original files are not sorted.

Now, here is the tricky part. Some of the lines in the files follow the following formats:

./$ARTIST/$ALBUM/$N1 - $N2 - $TITLE.(flac|mp3)
./$ARTIST/$ALBUM/$N1 - $TITLE.(flac|mp3)
./$ARTIST/$ALBUM/$N1. $TITLE.(flac|mp3)

Where $N1 and $N2 are numbers of 1-3 digit length, e.g. 01, 12, 132

$ARTIST, $ALBUM and $TITLE can be made up of any sequence of alphanumeric characters including . and - but not /.

I want lines ending with flac or mp3 to be considered a match if $ARTIST, $ALBUM and $TITLE are the same and the corresponding line to be excluded.

Example file1

./The BEATLES/white album/01. Revo-lution.flac
./123-123/234-234/01 - 02 - 3769.flac
./THE BEATLES/abbey road/cover.jpg

Example file2

./The BEATLES/white album/01. Notinfile1.flac
./123-123/234-234/03. 3769.flac
./THE BEATLES/abbey road/cover.jpg

Output file

./The BEATLES/white album/01. Notinfile1.flac


I have thought about the problem and have come up with a solution but need someone to help me code it.

For each line of both files a key can be generated using the following method. Subsequent comparison of the keys between the files can be used to output the lines in file2 whose keys do not match those in file1

Key generation as follows:

If line does not end with flac or mp3, key is the line itself.

Else, check for the first occurrence of . or - after the last /.

If . is the case, then key is $ARTIST/$ALBUM/$TITLE where $TITLE is whatever comes after the dot.

If - is the case, then count the number of dashes in the part of the line after the last /.

If one is found then key is $ARTIST/$ALBUM/$TITLE where $TITLE is whatever comes after the dash.

If more than or equal to two dashes found, then check if what comes between the first and second dash is purely numbers and spaces. If it is, then $TITLE is after the second dash. If it isn't, then $TITLE is after the first dash.

Will someone please help me code this?

  • Ok. I think the edits were removed because I added information while it was out of sync. The blank lines are not part of the files. I have tried reading up on awk but the learning curve is too steep for me to master it in a short time. – Ong Kuan Yang Apr 14 '16 at 10:40
  • Please try to restore some of the things the edit did (while keeping the additional info you added). Especially put your code/file contents of the example files into code blocks. You can also remove greetings and "thanks" and "please help" and the likes. They are not needed on SE because they do not add info to a question. I will assume that you want to write a shell script with basic Unix tools if you mention awk. – Lucas Apr 14 '16 at 10:46
  • I am replying to Lucas's answer here as I do not have sufficient reputation to comment. Unfortunately, the sed script does not work. All it does is add ./ in front of every line of file2.reduced. The error in console is this repeated multiple times: grep: invalid option -- '�' Usage: grep [OPTION]... PATTERN [FILE]... The weird character probably came from my data as I forgot to mention that Unicode characters of other languages are used in $ARTIST, $ALBUM and $TITLE. Any solutions? – ongkuanyang Apr 15 '16 at 4:01
  • Sorry I missed an s as a first char in the sed script. See my edit. – Lucas Apr 15 '16 at 4:10
  • It works although the cryptic errors are still present. Thanks. – ongkuanyang Apr 15 '16 at 4:22

It is often usefull to have files in sorted order when you want to compare them. The -u makes every line in the output unique by removing any duplicates.

sort -u file1 > file1.sorted
sort -u file2 > file2.sorted

comm can compare sorted files but it can only compare literal text. So this will reduce the problem but can only filter out exact matches. -1 removes lines that are unique to the first file and -3 removes lines that are common to both file arguments. That leaves us with lines that are unique to the second file.

comm -1 -3 file1.sorted file2.sorted > file2.reduced

Now we need to do the complicated work only for this (hopefully) smaller file.

sed 's#^\./##' file2.reduced | while read line; do
  title=$(echo "$filename" | sed 's/^[0-9]\{1,3\}\. //;t;s/^[0-9]\{1,3\} - [0-9]\{1,3\} - //;t;s/^[0-9]\{1,3\} - //')
  # We use fixed strings in case there are special chars in the file name
  # If the file names are "regex-save" we can use one grep instead:
  # ! grep -q -E "^\./$artist_album/.*$title\.(mp3|flac)\$" file1.sorted
  if ! grep -F "./$artist_album/" file1.sorted | grep -F -e "$title." | grep -q -E '(mp3|flac)$'; then
    echo "./$line"
done > results

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.