2

I have a shell script which makes some analysis and prepares (i.e.: writes) some commands to run in a separated file.

So I have something like that:

echo my_command_to_run >> /tmp/file_command_to_run.txt

I have the feeling that the program is slower and slower. Is it possible that the program takes longer when the file is bigger (~3M of lines)?

I am also storing some stuff in memory, so this is also probably a source of my problem, but I just want to know if I need to redirect the output in different files. (e.g.: write several files of 2000 lines)


EDIT: My script is preparing the move of ~64M (millions) files into a much better architecture. So I go through all the different "structured" folders, and prepare the move.

I have such array in memory:

topic1 -> /path/to/my/folder
topic1_number_of_files -> nb

so my array is also getting bigger because I have several entries (max ~ 4'000).

Otherwise this is always the same OPs which are run. Only my array and my file are getting bigger.


EDIT2: Below is my script

Note:

  • I have several folders containing maximum 100'000 files inside.
  • I can have : folder1 -> (source1__description1, source1__description2, source2__description3)

Goal: have something like that:

  • source1/folder1 -> (source1__description1, source1__description2)
  • source2/folder1 -> (source2__description3, ...)

Current performances:

~900'000 lines inserted in 14 hours <=> this will take around 40 days to prepare all the move commands

#!/bin/bash

argument=$1

if [[ -n "$argument" ]] && [[ -e $argument ]]; then
    html_folder=$argument
    echo "We will move [folder]/files from your parameters: '$html_folder'"
else
    html_folder="/var/files/html_files/"
    echo "NO PARAMETER (or folder does not exist) - We will move [folder]/files from $html_folder"
fi

######################## create the list ########################
filename="/var/files/html_files/list_folder.txt" # list generated with  ls -1 -f (this doesn't take everything in memory)
ls -1 -fp $html_folder |  grep '/$' |  grep 'folder'> "$filename"
#################### END create the list ########################


echo " "


# --------------------------------------------------------------
# -------------- Global variables for moving part --------------
# --------------------------------------------------------------
    # Variables for storing the folder/files tree
    declare -A folder_array         # array of folder '/files/publisher_html/10.3390' => 4 (i.e.: 4 folders for mdpi)
    declare -A folder_files_array   # array of files in last folder '/files/publisher_html/10.3390' => 51 (i.e.: 51 files in the 4th folders for mdpi)
    storageFolder="/files/publisher_html/"
    nb_limit=100000 # max number of file per folder
    file_nb=0
    current_folder=""
# --------------------------------------------------------------
# --------------------------------------------------------------
# --------------------------------------------------------------


# --------------------------------------------------------------
# -------------- Global functions for moving part --------------
# --------------------------------------------------------------

    countNumberOfFilesPerFolder () {
        nb=0
        if [[ -e $1 ]]; then        
            nb=$(ls -1fp $1 |  grep -v '/$' | wc -l )        
        fi
        echo $nb
    }

    createFolderIfNeeded () {
        # $1  # first arg (/path/to/htmlfiles/10.3390)
        tmp_folder=""
        nb_folder=1
        nb_files=0

        if [[ ! -e $1 ]]; then # if folder doesn't exist
            sudo mkdir -p "$1/folder$nb_folder" ; # create the folders if don't exist
        else
            #echo "THE FOLDER $tmp_folder ALREADY EXISTED...BE AWARE!!!"
            if [[ -e ${folder_array[$1]} ]]; then
                nb_folder=${folder_array[$1]} # take the value from memory if available
            else
                nb_folder=$(ls -1f $1 | grep folder | wc -l )
            fi

            if (($nb_folder==0)); then # if no subfolder for the publisher folder
                nb_folder=1
                nb_files=0
                sudo mkdir -p "$1/folder$nb_folder" # simply create the first folder
            else
                # if [[ -e ${folder_files_array[$1]} ]]; then
                if [[ ${folder_files_array[$1]} ]]; then
                    nb_files=${folder_files_array[$1]}  # value from memory
                    #echo "value from MEEEEEM: $1 => $nb_files"
                else
                    nb_files=`countNumberOfFilesPerFolder "$1/folder$nb_folder"`
                    #echo "value from COOOOOOUNT: $1 => $nb_files"
                fi

                if (($nb_files >= $nb_limit)); then # create a new folder + reset memory value
                    ((nb_folder++))
                    nb_files=0
                    sudo mkdir -p "$1/folder$nb_folder"
                    #`createFolderIfNeeded "$1/folder$nb_folder"` # NO CORRECT -> will create a subfolder
                fi            
            fi       
        fi

        #((nb_files++))
        folder_files_array[$1]=$nb_files
        folder_array[$1]=$nb_folder

        current_folder="$1/folder$nb_folder" # change the global variable
    }

    extractPrefix() {
        whotest[0]='test' || (echo 'Failure: arrays not supported in this version of
    bash.' && exit 2)
        array=(${1//__/ })
        prefix=${array[0]}
        echo $prefix
    }
# --------------------------------------------------------------
# --------------------------------------------------------------
# --------------------------------------------------------------


toMoveFolder=$html_folder"toMove/"
toMoveFileIndex=1
toMoveCmdNumber=0
maxCmdInFile=2000
if [[ ! -e $toMoveFolder ]]; then # if folder doesn't exist
    sudo mkdir -p $toMoveFolder ; # create the folders
fi

cd $html_folder

while read -r folder # for each folder
do
    if [[ -e $folder ]]; then        
        echo "Will manage folder: $folder"


# ---------------------------------------------------------------------------------------------------
# -------------------------------------- MOVE INDIVIDUAL FILES --------------------------------------
# ---------------------------------------------------------------------------------------------------

    argument=$html_folder$folder
    cpt=0
    #argument=$1

    if [[ -n "$argument" ]] && [[ -e $argument ]]; then
        html_files_folder=$argument
    else
        html_files_folder="/var/files/html_files/html_files/"
    fi

    ######################## create the list ########################

    htmlList="/var/files/html_files/list_html.txt" # list generated with  ls -1 -f (this doesn't take everything in memory)
    ls -1f $html_files_folder > "$htmlList" # no need to exclude the "." and ".." (we exclude from the foreach)
    #################### END create the list ########################

    echo " "


    current_folder=$storageFolder # probably useless

    while read -r line
    do
        name=$line

        if [[ $name != "." ]] && [[ $name != ".." ]]; then # don't take the folder itself
            prefix=`extractPrefix $name`
            if [ -n $prefix ]; then
                # change the global $current_folder
                # + create new subfolder if needed
                # + increment nb of files in folder
                createFolderIfNeeded $storageFolder$prefix
                ((cpt++))

                if(( $toMoveCmdNumber >= $maxCmdInFile )); then
                    toMoveCmdNumber=0
                    ((toMoveFileIndex++))
                fi

        echo "sudo mv $html_files_folder$name $current_folder/$name" | sed -r 's/[\(\)]+/\\&/g' >> $toMoveFolder"command_"$toMoveFileIndex".txt"
                ((toMoveCmdNumber++))

                ((folder_files_array[$storageFolder$prefix]++))

                if (( $cpt % 50 == 0 ));then
                    echo ""
                    echo "Remind: folder -> $current_folder/"
                    echo "${#folder_array[@]} publishers in memory!"
                fi

                echo "#$cpt - $name (${folder_files_array[$storageFolder$prefix]} files)"
            else
                echo "ERROR -> $name has not been moved as expected"
            fi        
       fi

    # >> $toMoveFolder"file"$toMoveFileIndex".txt" # <== does not take the toMoveFileIndex variation in consideration
    done < "$htmlList" # useful if we use the while

    echo "Folder $html_files_folder has been processed"
    echo " "


# ---------------------------------------------------------------------------------------------------
# ---------------------------------------------------------------------------------------------------
# ---------------------------------------------------------------------------------------------------


    else  # END  if [[ -e $folder ]]; then
        echo " "; echo "ERROR -> folder $folder does NOT exist!"; echo " "
        continue
    fi

done < "$filename" # useful if we use the while

echo "The script to prepare the move of the html files FROM FOLDER in other folders finished!"
echo " "
echo " "
echo "FOLDER ARRAY AT THE END: "
    for i in "${!folder_array[@]}"; do echo "folder  : $i => nb_folder: ${folder_array[$i]} / nb__file in last folder: ${folder_files_array[$i]}"; done

echo " "
echo " "
echo "This is the end of the script"

And the partitions:

$df -h
/dev/sdb1         2.0T  370G  1.7T  19% /var/files
X.X.X.X:/files     11T  2.8T  7.2T  28% /files

LAST EDIT:

After further analysis, I found that /var/files/html_files/ was a symlink to /files/html_files/ So the source and destination were actually the same (remote) server. I placed my script to run on the "remote" server, and it seems to be much faster.

Thanks for your help and interesting comments!

14
  • It goes without asking. One way of burning CPU cycles and that I do avoid here is to have big log files. I/O is always costly. Dec 28 '15 at 7:20
  • @RuiFRibeiro, thanks. So would the solution I proposed (write in different files) be less costly?
    – Bast
    Dec 28 '15 at 7:24
  • Any solution that you come up with of having none or smaller log files will always be less costly. Dec 28 '15 at 7:26
  • 1
    Consider that you really don't know if the program is running slower, and if it is, you don't know if the output is causing it. Premature optimization is not advised. First do timing tests to find out if it really is running slower. Then test to find out where the time is going. You can spend too much effort optimizing the wrong area and not get desired results and waste your labor.
    – RobertL
    Dec 28 '15 at 7:41
  • 1
    What is 3M lines? Million?
    – RobertL
    Dec 28 '15 at 7:42
3

I just want to know if I need to redirect the output in different files. (e.g.: write several files of 2000 lines)

Splitting into a larger number of files will not necessarily equal faster execution. Three simple test cases illustrate this. These three cases print 3M lines each. These are listed in the order of execution speed, fastest to slowest.

  1. One Redirection Outside of Loop

    for i in $(seq $((3000000/2000))); do seq 2000; done > file
    
  2. Appending to the Same File, Inside the Loop

    for i in $(seq $((3000000/2000))); do seq 2000 >> file; done
    
  3. Splitting Output to Multiple Files

    for i in $(seq $((3000000/2000))); do seq 2000 > file$i; done
    

The latter commands consistently take more user and system time than the former commands.

From this we can conclude that splitting into a larger number of files does not guarantee performance increase in this simple case. The opposite is true.

Number of I/O Operations

The performance depends not only on the size of the file but also the number of IO operations. When appending (>>) even more I/O calls take place in order to seek to the end of the file.

This first script performs the i/o operations (>>) outside the for loop:

$ cat outloop.sh
#!/bin/sh
>file
for i in $(seq 1 ${1:?})
do
    echo $i
done >> file

This script, on the other hand, performs the i/o operations (>>) on each iteration, inside the for loop:

$ cat inloop.sh
#!/bin/sh
>file
for i in $(seq 1 ${1:?})
do
    echo $i >> file
done

Run and compare, see how the location of the >> operator affects performance:

$ x=500000; time sh outloop.sh $x; time sh inloop.sh $x; 

real    0m1.227s
user    0m0.389s
sys     0m0.859s

real    0m2.996s
user    0m0.809s
sys     0m2.197s

Placing the redirection operator outside the loop doubles the performance when writing 500000 lines (on my system).

8
  • Thanks a lot for this very interesting demonstration. That's even more interesting, because I do echo in the for loop...But how would it behave if I only write all the content after 64 millions of lines? -- keep everything in memory?
    – Bast
    Dec 28 '15 at 11:04
  • are you sure that "done >> /my/file.txt" will reduce the number of I/O?? Because even if I make a really simple script, it seems that this is also inserting the text directly: ---- for i in $(seq 1 10); do echo $i - date; sleep 1; done >> /tmp/output.txt
    – Bast
    Dec 28 '15 at 13:06
  • Interesting argument. Not sure your first conclusion is correct. Maybe the penalty for moving the redirect to inside the loop is because the file handles need to be set up multiple times.
    – joepd
    Dec 28 '15 at 13:17
  • @joepd That's the point. That would also be true in the script of the original question. There is another case, that of appending to the same file inside the loop, which should cover more cases and be more similar to the original script. I've added that as case #2 in the answer.
    – RobertL
    Dec 28 '15 at 17:11
  • @Bast, Re: everything in memory: You would need to setup another test case to simulate holding the data in memory. Because we don't know the structure of your unseen script, it's difficult to tell. My suggestion would be to have your unseen script write all to standard output, and redirect the output of the script to a file on the command line at execution time. But that might not work for your script for multiple reasons, but again, not enough information is given to determine that. In general, streaming the data out as it is processed is better than holding everything in memory.
    – RobertL
    Dec 28 '15 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.