1

I've been trying to make a script merge all csv files to a montly file. The script below does not work, it gives the following error:

awk: fatal: cannot open file `/ibv_ftp/merged/folder/file201910*.csv' for reading (No such file or directory)

This is the example of the script that does not work correctly. it looks for the file "file201910*.csv" instead of all "file20191030.csv" files. each day has its own file.

To be clear. The issue i am encountering is the asterix not functioning as a wildcard. But instead its being used in the filename. It seems to not function when called after a variable. Even when using the variable in different ways, with/without quotes or instead of the asterix using [01-31] or question marks. they all show as a part of the file name.

#  Defines the current year     
y=$(date +'%Y')
# Choose which months to be merged
for i in {06..10}
  do
  # Defines year and month
   ym=$(date +'%Y')"$i"
   # monthly
   awk 'FNR==NR||FNR>2' /ibv_ftp/merged/folder/file"$ym"*.csv > /ibv_ftp/merged/monthly/file"$ym".csv
done

The scripts calls /ibv_ftp/merged/folder/file201910*.csv or file201910[01-31].csv or file201910??.csv A previous script like this did work. it used different files and looked like this:

today=$(date +'%Y%m%d')
echo $today
# daily

awk 'FNR==NR||FNR>2' /ibv_ftp/folder/iv1_"$today"_*.csv > /ibv_ftp/merged/daily/folder/file"$today".csv

In this example the files are called iv1_20191030_111500.csv (last part is the timestamp)

  • 1
    Did you specify the correct directory for the input CSV files? The code that you say works writes to .../merged/daily/folder, do you want to read these files? You currently look in .../merged/folder. – Kusalananda Oct 30 at 10:46
  • I see you are looping over a range of months. Does the error occur for every value of the month part? I only ask because the error message you show happens to refer to the last month in your range, and in my tests, such an error would occur if no file with that name part exists. – AdminBee Oct 30 at 10:46
  • Yes, every file it has to edit returns the error path not found. @AdminBee – koen Oct 30 at 12:22
  • Do you possibly have set -f or set -o noglob futher up the script? Does ls -ld -- /ibv_ftp/merged/folder/file201910*.csv at the prompt of an interactive shell show the list. Does the user running the script have permission to read the contents of the /ibv_ftp/merged/folder directory? – Stéphane Chazelas Oct 30 at 14:23
0

The error is telling you that nothing matches the pattern /ibv_ftp/merged/folder/file201910*.csv. (It does this because when the shell cannot expand a wildcard character such as * it usually leaves it as a literal.)

Looking at the code you included that you say does work, the source pattern starts with /ibv_ftp/folder/ where as your non-working code uses /ibv_ftp/merged/folder/.

  • its a different script. that one places the files into the merged folder. 15 minute data > merged daily > merged monthly. The issue is with the monthly script. – koen Oct 30 at 12:07
  • @koen in that case you need to make this quite clear in your question. We can only work on the information you give us. – roaima Oct 30 at 13:04
  • Sorry, this is the first time for me posting here, if you need any information or if something is unclear please ask me to further elaborate. I find it a very weird question in which i know where the problem is, but its rather difficult to make myself clear. – koen Oct 30 at 14:06
  • @koen my question still answers your edited question. The * is used as a literal because it doesn't match anything. I suggested that was because your source path is wrong, and you haven't answered or addressed that. – roaima Oct 30 at 14:40
  • Thank you for thinking with me, I've noticed there was indeed an error with the file path. I went looking in the wrong place to resolve this error. – koen Oct 30 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.