0

I have millions of xml files in a folder. The name of the files follow a specific pattern:

ABC_20190101011030931_6049414.xml

In this I am interested only in the last set of digits before xml 6049414. I have a list of around 8000 such numbers in a text file. The details in the text file is as follows - a number in a line:

104638
222885
108880071

I am using the following code to move the files from the folder that matches the number given in the text file:

#folder where the xml files are stored  
cd /home/iris/filesToExtract  
SECONDS=0

#This line reads each number in the hdpvr.txt file and if a match is found moves that file to another folder called xmlfiles.  
nn=($(cat /home/iris/hdpvr.txt));for x in "${nn[@]}";do ls *.xml| grep "$x"| xargs -I '{}' cp {} /home/iris/xmlfiles;done  

#this line deletes all the other xml files from filesToExtract folder
find . -name "*.xml" -delete  
echo $SECONDS

I am facing two issues. 1 Some of the files are not getting moved despite there is a match and 2. Even if the match is found in the middle part of the file name for example

from this ABC_20190101011030931_6049414.xml -> this 20190101011030931  

if a match is found it still moves....how can I get the exact matches and move the files.

  • the numbers in your text file are only used to match the _XXXXX.xml in your xml filenames ? – darxmurf Apr 30 at 5:41
  • @darxmurf yes.. – Apricot Apr 30 at 5:43
1

Would something like this make the job ?

pushd /home/iris/filesToExtract
for i in $(</home/iris/hdpvr.txt); do find . -mindepth 1 -maxdepth 1 -type f -name "*_$i.xml" -print0 | xargs -r -0 -i mv "{}" /home/iris/xmlfiles; done
find . -mindepth 1 -maxdepth 1 -type f -name "*.xml" -delete  
popd
  • pushd will move you in the specified directory
  • for+find line will get the ID from your text file, find files ending like _ID.xml and move them in the /home/iris/xmlfiles folder
  • the last find like will delete the non moved files but only in this folder and not sub ones
  • popd will put you back in your original directory

You can also do it the brutal way with mv but it will throw errors if a file is not found

pushd /home/iris/filesToExtract
for i in $(</home/iris/hdpvr.txt); do mv "*_$i.xml" /home/iris/xmlfiles; done
find . -mindepth 1 -maxdepth 1 -type f -name "*.xml" -delete  
popd
  • I kicked out your SECONDS variable as you did not use it in your example – darxmurf Apr 30 at 6:03
  • Instead of using print0 and piping to xargs, let find do the work: -exec mv -t $destdir {} + – glenn jackman Apr 30 at 13:19
  • works too but I would "protect" your exec with double quotes like "{}" – darxmurf Apr 30 at 14:01
  • find doesn't need the quotes: the shell removes them before launching find. And bash doesn't care about empty braces. – glenn jackman Apr 30 at 15:57
2

Another solution, thanks to glenn jackmann!

#!/bin/bash

# folder where the xml files are stored
xmldir=/home/iris/filesToExtract

# xml backup folder
backupdir=/home/iris/xmlfiles

while read -r line; do
    mv -t "$backupdir" *_*_${line}.xml 2>/dev/null
done <"$xmldir/hdpvr.txt"
rm -i *.xml

Pattern *_*_${line}.xml is used to find the files in the directory.

Replace rm -i *.xml with rm *.xml if you want to delete the remaining xml files immediately.

  • Might be faster, instead of looping over the files and moving the one-by-one, to do mv -t "$backupdir" *_*_"$i".xml – glenn jackman Apr 30 at 13:15
  • Also, should not use for to read files: while read -r i; do ... done < file.txt – glenn jackman Apr 30 at 13:16
  • @glennjackman For this specific filename pattern a dirty for-loop should do, but you're right! – Freddy Apr 30 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.