I have a recursive directory called 'dir'. I am writing to list of files from all subdirectories to a CSV file with the following command in linux on the terminal.

dir$ find . -type f -printf '%f\n' > old_names.csv

I am using a detox code to change filenames. And I am making a new list using

dir $ find . -type f -printf '%f\n' > new_names.csv

I would like to join this to lists together and make a new list with two columns something like this;

enter image description here

To do that I read both csv files into pandas data frame and join them on index as follows in python3 script

 import pandas as pd
 import csv


The problem is I am getting something like this, wrong file pairs;

enter image description here

When I open the new_names.csv I see that file list is written in a different order than old_names list so joining on index resulting in wrong pairs. How can I solve this problem?

  • There's no particular reason that the two finds must produce output in the same order, so the whole exercise may be flawed, but if you're in a situation where they do, do you need to be using Python to join them or would paste suffice? – Michael Homer Jan 21 '19 at 19:11
  • Hi, thanks! I have many folders like that so i wanted to do join them in a for loop over directories in python. – kutlus Jan 21 '19 at 19:15
  • There's just no reason that this would work except under pretty controlled conditions (specific filesystems in use, possibly control of other simultaneous operations on the system). detox will tell you what changes it's making and you'd be better off to use that information instead, I think, rather than trying to reverse-engineer it. – Michael Homer Jan 21 '19 at 19:18

The find command just outputs in the order the filesystem gives its directory entries in, without any sorting or processing. Depending on the filesystem you're using and other factors, renaming even a single file could change the iteration order, but changing all of them is quite likely to do so. Without a tightly-controlled environment there's no particular reason that two finds should give the same order like that.

For example, many modern filesystems store names in a hash table, and iterate in the order entries appear there. A tiny filename change may be much earlier or later in the table than the original, or even cause total re-hashing of the entire directory so that everything moves. There's no realistic way to put the pieces back together in that case.

It's possible that sorting the filenames might help, if they each have a unique unchanged prefix, but that's the only realistic sort of post-processing you could do and carry on with two separate files from two find runs. I don't recommend even trying that.

However, detox does have a -v option that prints out the changes it is making (and -n to print out what it would do). You could use that to produce your CSV file, or directly from Python using subprocess.run.

detox -v ... | sed -e 's/ -> /,/' > names.csv

would produce a CSV file at least as well as one of your finds, with the old and new names automatically matched up. For the basenames (like %f did) you'll need to postprocess, which you can do in Python if necessary, or in the shell.

  • Thank you Michael this was helpful, I am convinced that i can`t get around this. The detox code not the detox package but just some functions I defined them to replace some characters with others, and called them detox functions. Now,I have decided convert first list into data frame, call each row with index and apply the detox functions on each row to create the new name list. – kutlus Jan 21 '19 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.