0

I'm trying to write a script that will copy all file listed in a text file, around 3 million lines, which contains two columns, the source and the destination with a new filename:

path/to/source/directory/filename.pdf path/to/destination/directory/Newfilename.pdf
path/to/source/directory/filename2.pdf path/to/destination/directory/Newfilename2.pdf
path/to/source/directory/filename3.pdf path/to/destination/directory/Newfilename3.pdf
...

All files are PDF format, where Newfilename.pdf is the new filename for the same source PDF file.

ADDITIONALLY, I would like to copy the file and add information to its destination filename, i.e.:

From:
Newfilename.pdf

To:
Newfilename_yyyyMMddHHmmss.pdf (e.g. Newfilename_20200225095823.pdf)

Where yyyyMMddHHmmss is the actual copy executing date and time for each file and this is the same format for all, causing the destination file to be copied with its complemented name:

path/to/destination/directory/Newfilename_20200225095823.pdf
path/to/destination/directory/Newfilename2_20200225095824.pdf
path/to/destination/directory/Newfilename3_20200225095830.pdf
...

I do not have enough knowledge to handle commands, an idea of ​​what I was researching is the following:

#!/bin/bash
filename=$1

while read -r source destination; do
# reading each value
cp -p source destination
done < $filename

However, I read some similar publications, for performance, the while loop and read are tremendously slow when reading from a file or a pipe, because the read shell built-in reads one character at a time. Reference here.

How it could be done with a better solution?

I will greatly appreciate your help.

  • 2
    Please edit your question and add a reference where you found the claim that the while loop and read are tremendously slow when reading from a file or a pipe – Bodo Feb 25 at 15:06
  • Thanks for your recomendation @Bodo. – msarcom Feb 25 at 15:27
  • 1
    If you read the referenced answers in detail you can see bad examples that read input line by line and call programs (e.g. cut) to process a single line instead of passing the whole file to the program. Or a manipulation of the input data implemented in shell script code instead of using specialized programs. In your case you have to run cp for every combination of source and destination file name, so I don't see anything wrong with your loop. Please specify if you want to have the same time stamp for all destination files or individual time stamps when copying every single file started. – Bodo Feb 25 at 15:45
  • Thanks to your observation @Bodo I made just the clarification that the date and time should be for each file that should be copied. Regarding your performance comment, I greatly appreciate your criteria and I will do the corresponding tests to analyze the time it takes for the process with the code proposed by AdminBee. – msarcom Feb 25 at 16:14
  • 1
    I would think that the time spent copying 3 million files will dwarf the time spent by the shell to read a text file. Are you optimizing prematurely? – glenn jackman Feb 25 at 17:54
1

Leaving the performance aspect aside, the first part of your question can be solved using bash's variable manipulation methods:

timestamp="$(date +%Y%m%d%H%M%S)"

while read -r source destination; do
  newname="${destination/%.pdf/_$timestamp.pdf}"
  cp -p "$source" "$newname"
done < "$filename"

If the timestamp is to be the "moment of copying" rather than that of calling the script, the call to date must be placed inside the loop:

while read -r source destination; do
  timestamp="$(date +%Y%m%d%H%M%S)"
  newname="${destination/%.pdf/_$timestamp.pdf}"
  cp -p "$source" "$newname"
done < "$filename"

Update: As pointed out by @Jetchisel, bash from v4.2 upwards has builtin functionality to format dates using the printf command, which would make the call to the external date command unnecessary:

while read -r source destination; do
  printf -v timestamp '%(%Y%m%d%H%M%S)T'
  newname="${destination/%.pdf/_$timestamp.pdf}"
  cp -p "$source" "$newname"
done < "$filename"
| improve this answer | |
  • I think it should be cp -p "$source" "$newname". As the example in the question shows different time stamps for the individual files, it might be necessary to call date inside the loop. (This is not clearly specified in the question.) – Bodo Feb 25 at 15:39
  • @Bodo good catch, thanks! – AdminBee Feb 25 at 15:50
  • Thank you very much for your answer @AdminBee, it's a great help. – msarcom Feb 25 at 15:55
  • 1
    The presumed performance aspect is hard to work around anyway, as long as one is limited to using the standard cp or such an external tool. Since the filenames are unique, it just can't be done without launching a new cp for each file, and that's going to dwarf anything related to reading some lines from a file. (Then there's the I/O itself, which of course depends on the hardware.) – ilkkachu Feb 25 at 17:17
  • 1
    Instead of date you can use printf in bash, well if your bash has that builtin that can format date. printf '%(%Y%m%d%H%M%S)T' should give the same result as that date format, that way date is not called/run every line, just my two cents. – Jetchisel Mar 4 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.