Maintain parallel lossy and lossless music collections

Question

I have a large music collection. Some of it is lossless files, and some is lossy.

I would like to maintain a copy of the collection that consists of the original collection's lossy files, and lossy transcodes of the original collections lossless files.

Some assumptions:

• I know how to use ffmpeg to convert flac to opus files
• I only have flac files that need to be converted, no wav or alac codecs
• The lossy files can be opus, vorbis, or mp3

I want to:

• Use minimal storage for the the new music collection. I.e. it would link back to the original lossy files where appropriate.
• Keep the collection up to date as I add more lossy and lossless files to the original, or update the metadata.
• Not have to re-transcode lossless files that have not been modified.

I imagine I will need to use some custom scripting to accomplish this, but if anyone has recommendations or tips before I sink lots of time into this, I would be eternally grateful.

Answer 1

I don't like Makefiles (I might concur with this guy); however, make does what you want, out of the box:

You define a rule, for example, that you want an .opus file for every source .flac file:

Makefile, from the top of my head

TARGETDIR=/path/to/compressed/library
%.opus: %.flac
    ffmpeg -ffmpegflags -and -stuff -i "$<" -o "$@"
$(TARGETDIR)/%.opus: %.opus
    cp --reflink=always "$<" "$@"

That could convert all your FLACs into OPUSes in-tree. And it will only do that if the .opus file isn't there yet, or older than the last change in the FLAC.

I don't like it, because it happens in-tree, i.e. you don't end up with a clean "originals only" directory. At least use a cp that supports reflinks, on a file system that does, too, so that your copies are shallow (and don't actually need any space). It also doesn't deal with subdirectories gracefully, I think, you'll find

Then, honestly, make's functionality here is really just:

For each wildcard source file (%.flac), check whether the result (same file .opus) has been built already, and if not (or the build is older than the source file), do the build.

That's a bit backwards, and also not complicated enough to depend on Make. So, shell scripting. I use zsh. And while I don't test what I write, I try to comment it:

#!/usr/bin/zsh
# Copyright 2022 Marcus Müller
# SPDX-License-Identifier: BSD-3-Clause
# Find the license text under https://spdx.org/licenses/BSD-3-Clause.html

# set options:
setopt null_glob    # Don't fail if there's no file matching a pattern
setopt no_case_glob # Don't care about case in matching

TARGET_DIR=../compressed_library

make_containing_dir() {
  target_dir="${1:h}"
  if [[ ! -d "${target_dir}" ]] ; then
    logger -p user.debug "Creating directory ${target_dir}"
    mkdir -p "${target_dir}" || logger -p user.err "can't mkdir ${target_dir}"
}

for compressed_source in **/*.{mp3,opus,vorbis,mp4} ; do
  if [[ -d "${compressed_source}" ]]; then
    continue # skip directories that happen to have a matching suffix
  fi

  logger -p user.debug "dealing with compressed source ${compressed_source}"
  
  target_file="${TARGET_DIR}/${compressed_source}"
  make_containing_dir "${target_file}"

  # -h : check whether target exists and is symlink
  if [[ ! -h "${target_file}" ]] ; then
   ln -s "$(pwd)/${compressed_source}" "${target_file}" \
     || logger -p user.err "copying ${compressed_source} failed"
  fi

done

for uncompressed_source in **/*.flac ; do
  if [[ -d "${uncompressed_source}" ]]; then
    continue # skip directories that happen to have a matching suffix
  fi

  logger -p user.debug "dealing with uncompressed source ${compressed_source}"

  target_file="${TARGET_DIR}/${uncompressed_source%%.flac}.opus"
  #                                               ^ strip the .flac suffix
  make_containing_dir "${target_file}"

  #         /-- compare source file for "older than"
  #         |   target file; this returns !=0 if the source file
  #         |   is newer, or the target file nonexisting
  #         \--------------------\
  #                              |
  if [[ "${uncompressed_source}" -ot "${target_file}" ]]; then
    ffmpeg -loglevel=fatal \
           -i "${uncompressed_source}" \
           -b:a 96k \
           "${target_file}" \ 
      || logger -p user.err "transcoding ${uncompressed_source} failed"
  fi

done

This is very untested, but at least it logs to syslog (journalctl -xef is your friend)