1

I have a program that takes data from five government sources and merges them into one large database for my company. I use wget to retrieve the files. However I have discovered that one of the sources changes the name every time it is updated.

For example, the last time I got the file it was called myfile150727.flatfile. Today when I tried to run my program I got exit status 8 no such file. When I manually got into the ftp I found that the file is now called myfile150914.flatfile. So obviously the filename is changing based upon the date it was last updated.

Can I modify my script to take this fact into account and still automatically download the file?

2
  • 1
    why then use wget if you have ftp ? if ftp is working from unix plateform you should use it, it is quiet scriptable and allow wildcard get (e.g. get myfile*.fileflat) – Archemar Sep 17 '15 at 11:53
  • I'm not a bash scripter. I'm a c++ programmer, so I'm just picking up the bash I need along the way. Please, tell me more about this ftp wildcard get. – nonremovable Sep 17 '15 at 11:55
5

Yes, but the details depend on how the file's name changes. If it is always today's date, just tell your script to get that:

filename=myfile"$(date +%y%m%d)".flatfile
wget ftp://example.com/"$file"

Or, if it is not updated daily and there is only one file called myfileWHATEVER.flatfile, get that:

wget "ftp://example.com/myfile*.flatfile"

If you can have many files with similar names, you could download all of them and then keep only the newest:

wget -N "ftp://example.com/myfile*.flatfile"
## Find the newest file
for file in myfile*.flatfile; do
    [[ "$file" -nt "$newest" ]] && newest="$file";
done
## Delete the rest
for file in myfile*.flatfile; do
    [[ "$file" != "$newest" ]] && rm "$file"
done

Alternatively, you can extract the date from the file name instead:

wget -N "ftp://example.com/myfile*.flatfile"
for file in myfile*.flatfile; do
    fdate=$(basename "${file//myfile}" .flatfile)
    [[ "$fdate" -gt $(basename "${nfile//myfile}" .flatfile) ]] && nfile="$file"
done
for file in myfile*.flatfile; do
    [[ "$file" = "$nfile" ]] || rm "$file"
done

Note that the above will keep multiple files if more than one have the same modification date.

5
  • Your code based on the modification time is more complex than what's needed here: the file names are ordered by time. Using file names is potentially more robust in case an old data dump is updated for some reason. – Gilles 'SO- stop being evil' Sep 17 '15 at 23:14
  • @Gilles stat -c %Y compares modification, not creation times. The -c is a GNU flag which is equivallent to --format. It has nothing to do with creation. Your change to stat -m breaks the script since there's no -m switch, at least on GNU stat. As for the modification times, yes, I could compare them but I think it is likelier to have a file uploaded with the wrong name than to have an old one modified. Plus, I happened to have a similar script lying about already :) – terdon Sep 18 '15 at 11:48
  • Woops, sorry, brain fart. P.S. You could use the -nt operator of test instead of parsing times, it would make your script portable outside Linux and even to dash (test -nt is one of its few extensions to POSIX). Regarding the use of the name, it's likely that the file name is generated by the data generation process whereas the file mtime indicates when the file was produced or copied, so the file name is far more likely to be reliable. – Gilles 'SO- stop being evil' Sep 18 '15 at 11:53
  • @Gilles ooh, nice, I didn't know about -nt, thanks! I also added a version based on the file's name so the OP can choose. – terdon Sep 18 '15 at 12:16
  • This worked perfectly. Thank you! My situation was actually the third of the four cases you described. I appreciate your help. – nonremovable Sep 18 '15 at 14:49
0

something like

for site in site1.com site2.com
do
   mkdir -p $site
   ftp -in <<EOF
   connect $site
   user archemar foo
   lcd $site
   cd /path/to/remote/file
   mget myfile*.fileflat
   quit
EOF
done
  • replace user archemar foo by your username and password, or user ftp if annonymous ftp is allowed.
  • replace cd /path/to/remote/file by the obvious
  • this will place misc *.flatfile in corresponding site dir. If name are unique, remove mkdir $site and lcd $site.
  • according to your update process, you may wish to rm before transfer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.