I have a list of files in /tmp/drop directory. I have taken the essential element of my question into test script zz.sh. I have taken a part of the script and put it here. These are the files and the lines listed - and when it given to while loop - how does the variable file gets assigned only the file name and not the various other strings in those lines?

Any help would be appreciated and would be helpful to others too.

machine001:/home/aaa999999> ls -l /tmp/drop | grep "2017-01-29" | egrep  '\.gz$' | tail -10
+ ls -F -l /tmp/drop
+ tail -10
+ grep 2017-01-29
+ egrep \.gz$
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-21:33:13.888.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-21:48:14.632.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:03:19.098.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:18:19.416.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:33:19.878.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:48:25.636.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:03:26.515.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:18:28.279.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:33:33.059.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:48:33.841.complete.gz
machine001:/home/aaa999999> cat zz.sh
+ cat zz.sh
set -x

# move from drop dir
ls -1 ${DROP_DIR} | grep ${YESTERDAY} | egrep '\.gz$' | while read file; do
    mv  ${DROP_DIR}/$file ${ARCHIVE_DIR}
        echo 'File name is ' ${file}
  • What do you want to do? Looks to me like mv /tmp/drop/*2017-01-29*.gz /tmp/arch? – pfnuesel Jan 31 '18 at 0:53
  • 4
    In the first case, you have ls -l (letter ell) while in the script it appears to be ls -1 (digit one) - different options, different behaviors. Regardless, please see Why not parse ls? – steeldriver Jan 31 '18 at 1:01
  • @steeldriver Good catch! – G-Man Jan 31 '18 at 4:04
  • 2
    for file in "$DROP_DIR/*$YESTERDAY*.gz" ; do ... ; done - parsing ls is not only potentially dangerous, it isn't even necessary. Also, that while read loop is in a pipe subshell, so can't affect the environment or variables of the parent script (see unix.stackexchange.com/questions/9954/… for why). – cas Jan 31 '18 at 4:58
  • @cas drop those double quotes surrounding the wildcard globs – roaima Jan 31 '18 at 19:18

ls -1 prints just the filenames. ls -l prints a bunch of other stuff, too. The other is the number one, the other a lowercase letter el.

Though the -1 is unnecessary here, since if the output is redirected to a pipe, the "one-column" mode is implied.

But really, you don't need the ls and the pipe here, at all. The loop can be replaced with just:

for file in "$DROP_DIR/"*"$YESTERDAY"*.gz ; do 
    mv "$file" "$ARCHIVE_DIR"
    echo "File name is ${file#$DROP_DIR/}"

See also: ParsingLs on Greg's Bash wiki.


Suggestion using GNU date:

yesterday=$( date -d yesterday +%F )

mkdir -p "$archive_dir"

mv -f "$drop_dir"/*"$yesterday"*.gz "$archive_dir"

Here, "$drop_dir"/*"$yesterday"*.gz will expand to all files matching the given pattern. This will work as long as the pattern expand to several thousands of characters, in which case the shell will complain with "Argument list too long".

Or, if you really want to do looping to output each filename as they are moved (only the mv command from above is changed to the following):

for name in "$drop_dir"/*"$yesterday"*.gz; do
    [ -f "$name" ] || continue
    mv "$name" "$archive_dir"
    printf 'Moved "%s"\n' "$name"

Here, the for loop will iterate over the same names as was used in the previous variation, but one by one. Each name will be held in $name (which includes the directory name). If there were no files matching the pattern, the pattern will remain unexpanded. This is why we do an explicit test to make sure that $name really corresponds to an existing file before moving it.

Or, with find:

find "$drop_dir" -maxdepth 1 -type f -name "*$yesterday*.gz" \
    -print -exec mv {} "$archive_dir" ';'

Here, find will pick up the names from $drop_dir that corresponds to regular files (-type f) and that matches the pattern given to -name. For each such name, mv is called to move the file into tho destination folder. Before calling mv though, -print will list the filename (including path) on the terminal.

The -maxdepth 1 option stops find from recursing into subfolders of $drop_dir.


Your question isn't too clear to me, but I think what you seem to be looking for is the -o option to your egrep statement:

 egrep -o "[^[:space:]]+\.gz$"

The -o option (--only-matching) causes egrep to only output the matching portion of matching lines. The [^[:space:]] matches anything not a space character.

  • that would print just .gz, which isn't likely to be very useful. – ilkkachu Jan 31 '18 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.