3

I am looking for a unix command sequence, which is able to perform a quite complex, conditional deleting action.

All files with the extensions "[NAME].cut" and "[NAME].cut.bak" shall be deleted, IF there exists no corresponding file named "[NAME].rec" or "[NAME].mpg" in the same folder.

The command shall be run on an integrated device with some minimal kernel, and therefore not use "special tools". e.g. 'ls' and 'rm' can be used, 'find' is not present (i.e. only via an external tool, which I would prefer not to depend on)

If possible, the command shall delete recursively in subdirectories as well...

Background is following: I am developing a tool for a Linux-based PVR, that allows cutting of recorded programmes. For each recording (extension .rec or .mpg) the segment markers are stored in a .cut-file. When some recording gets moved/renamed/deleted, then the corresponding cut-file remains on the disk useless. I already have implemented the removing of those useless cut-files in C. But I am wondering, whether there may be a (simple) system based solution. In this case it could be run via 'system' and & in the background, which would make my application more responsive...

Here are the commands, which are included within the PVR's firmware:

affinity, arp, awk, basename, bash, busybox, cat, chgrp, chmod, chown, cmp, cp, cramfsck, cut, date, dd, df, dirname, dmesg, du, echo, egrep, env, expr, false, fgrep, flash_erase, flash_eraseall, flash_info, free, getty, grep, head, hexdump, hostname, id, ifconfig, install, kill, killall, ln, logger, login, ls, md5sum, mkcramfs, mkdir, mknod, mktemp, more, mount, mv, nice, norcleanmark, od, passwd, pidof, ping, pmtest, portmap, printenv, printf, ps, pwd, readlink, renice, rm, rmdir, route, sed, seq, sh, shutdown, sleep, sort, strings, stty, sync, tail, tee, telnet, tftp, time, tinylogin, touch, tr, true, tty, twin, umount, uname, uniq, usleep, vi, wc, which, whoami, xargs, yes

Do you have any tips for me?

  • Which are the extensions to be deleted? [NAME].cut and [NAME].cut.bak or .cut and .cut.bak? Also if there exists no corresponding file with extension [NAME].rec or [NAME.mpg]? – kos Jun 28 '15 at 12:12
  • The extensions are '.cut' and '.cut.bak'. For example, if there are the files 'a.mpg', 'a.cut', 'b.cut', 'c.cut.bak', 'd.cut.bak', 'd.rec', then only 'b.cut' and 'c.cut.bak' shall be deleted. – chris86 Jun 28 '15 at 12:15
  • What is the actual use case? What is the embedded device? (How much RAM, what kind of "disk": SD cards?) How would that command be started? Thru some web interface? How are by what programs are the .cut & .rec files created? It might be some XY problem so you really should edit your question to improve it (and give motivations and more context) – Basile Starynkevitch Jun 28 '15 at 12:34
4

I couldn't keep this simpler; this works but it assumes there are no files whose filename contains newlines in the target directory; first test the command using this:

find . -type f \( -name "*.cut" -o -name "*.cut.bak" \) -exec bash -c '[ -f "$(<<< "{}" sed "s/\(.*\/[^.]*\).*/\1/").rec" -o -f "$(<<< "{}" sed "s/\(.*\/[^.]*\).*/\1/").mpg" ] && echo "{}"' \;

If the files listed are those expected to be deleted, you can go ahead and run this:

find . -type f \( -name "*.cut" -o -name "*.cut.bak" \) -exec bash -c '[ -f "$(<<< "{}" sed "s/\(.*\/[^.]*\).*/\1/").rec" -o -f "$(<<< "{}" sed "s/\(.*\/[^.]*\).*/\1/").mpg" ] && rm "{}"' \;

Test on a directory hierarchy created ad hoc:

~/tmp$ tree
.
└── dir1
    ├── file1.cut
    ├── file1.cut.bak
    ├── file1.rec
    ├── file2.cut
    ├── file2.cut.bak
    ├── file2.mpg
    ├── file3.cut
    ├── file3.cut.bak
    └── subdir1
        ├── file1.cut
        ├── file1.cut.bak
        ├── file1.rec
        ├── file2.cut
        ├── file2.cut.bak
        ├── file2.mpg
        ├── file3.cut
        └── file3.cut.bak

2 directories, 16 files
~/tmp$ find . -type f \( -name "*.cut" -o -name "*.cut.bak" \) -exec bash -c 'if [ -f "$(<<< "{}" sed "s/\(.*\/[^.]*\).*/\1/").rec" -o -f "$(<<< "{}" sed "s/\(.*\/[^.]*\).*/\1/").mpg" ]; then rm "{}"; fi' \;
~/tmp$ tree
.
└── dir1
    ├── file1.rec
    ├── file2.mpg
    ├── file3.cut
    ├── file3.cut.bak
    └── subdir1
        ├── file1.rec
        ├── file2.mpg
        ├── file3.cut
        └── file3.cut.bak

2 directories, 8 files

As you can see, all files with extension .cut or .bak for which a file with the same name and extension .rec or .mpg exists are deleted recursively (file1.cut and file1.cut.bak are deleted because of file1.rec, file2.cut and file2.cut.bak are deleted because of file2.mpg; file3.cut and file3.cut.bak are not deleted because there's no file3.rec or file3.mpg in the same directory)

  • Just two examples, that go wrong: './-Das Korallendreieck 1/Das Korallendreieck 1.cut' and 'Forrest Gump.cut' shall be removed, but in both cases the corresponding '.rec' file is present. – chris86 Jul 7 '15 at 21:59
  • @chris86 I think you have to wait some minutes for the global rep to update network-wise, however let me get this straight, those are the only two files that are not working? Or it's a sample? If the first, are you sure they're named exactly the same but the extension? Try to run ls file1 | hexdump -C and ls file2 | hexdump -C, see if they match in everything but the extensions and if the extensions don't contain extra characters – kos Jul 7 '15 at 22:07
  • Got the code to work now. But there is a problem, when a file name contains a '.' (dot) before the extension. Those files will be falsely deleted... – chris86 Jul 8 '15 at 16:48
  • @chris86 What was wrong with it? Anyway the dot problem should be easily solvable, try switching both seds' inner command to this one: s/\(.*\/.*\)\..*/\1/ – kos Jul 8 '15 at 17:00
  • I explained in the chat: '-e' does not find any files that are larger than 4 GB... To the dot problem: No. With the latest modification of sed, it always removes the '.cut.bak' file, even if there is a '.rec' present. -> I don't understand this cryptical sed-syntax - but maybe a fix encoding of "name ends with the string '.cut' or '.cut.bak' would be better? – chris86 Jul 8 '15 at 17:13
0

If the command is to be run on some small embedded device with very limited commands and shells (e.g. missing find, minimal shell à la sash), a possible approach would be to write some tiny C (or perhaps C++) program doing that, and compile that program to an executable (it would be nice to make that program a free software).

You'll use nftw to recursively traverse the file tree, and build an array (or a vector, or a list) of paths to be removed. Then a second pass would effectively remove the files. Relevant functions are nftw(3) (to traverse the file tree), snprintf(3) (to construct file names), access(2) (to test existence of file), stat(3) (to get type, size and other meta-data about a file), remove(3) (to remove a file), basename(3), etc.

If you have GNU find, you might make a shell script with it (but details could depend upon your actual shell, especially if it is small enough to not be POSIX compliant). Collect the filenames to be deleted in some temporary file. Then remove them later.

If you insist on using shell solutions, and assuming that GNU find is available, you might have several shell scripts:

First, a shell script (invoked from outside), let's name it clean-the-mess.sh which would create a temporary file, then invoke another internal shell script consider-cleaning.sh for every .cut & .cut.bak file:

#!/bin/sh
## untested clean-the-mess.sh script
## a temporary file listing files to remove
cleanlistfile=$(tempfile -s .cleanlist)
## on exit, or SIGQUIT, SIGTERM, or SIGINT signal, remove it
trap "rm $cleanlistfile" EXIT QUIT TERM INT
## find every file name .cut | .cut.bak; run consider-cleaning.sh on it
find -type f \( -name '*.cut' -o 'name '*.cut.bak' \) \
    -exec consider-cleaning.sh '{}' $cleanlistfile \;
## remove the collected file list
for f in $(cat $cleanlistfile) ; do rm $f ; done

That won't work if some files have spaces in their name. Perhaps consider xargs(1)

Then you need to code the internal consider-cleaning.sh script, which adds file paths to the list of files to be cleaned. It could be

#!/bin/sh
# internal untested consider-cleaning.sh script
file=$1
collectlist=$2
case $file in 
 *.cut) 
   bas=$(basename $file .cut)
   if [ ! -f "$bas.rec" -a ! -f "$bas.mpg" ]; then
      echo $file >> $collectlist
   fi
 ;;
 *.cur.bak)
   bas=$(basename $file .cut.bak)
   if [ ! -f "$bas.rec" -a ! -f "$bas.mpg" ]; then
      echo $file >> $collectlist
   fi
 ;;
 esac

My scripts are not proven, untested, so are probably wrong. Put #!/bin/sh -vx as their first line to test & debug them (you'll get some execution trace)

Alternatively, embed a tiny Lua interpreter with the few basic primitive file related operations, and write your script in Lua. Lua is very easily embeddable, and can be made to be a tiny interpreter.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.