1

I know this is a bloody beginner question. But maybe someone has a short answer on this one. In my folder I have several files that end on .SAFE. There are files with different endings though aswell... I would like to create a .txt file in which each line is one of those files. After that I'd like to run a programm that uses this .txt file as input. And all that I'd like to do in one bash-script. More or less like:

programm.sh
####
1. get all files with .SAFE-ending
2. make a file called files.txt
3. run <some_program> 

My folder looks like this

files.txt
S2A_MSIL1C_20200418T101031_N0209_R022_T32UPB_20200418T122607.SAFE
S2A_MSIL1C_20200418T101031_N0209_R022_T32UQB_20200418T122607.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UPB_20200421T111956.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UPB_20200421T122440.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UQB_20200421T111956.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UQB_20200421T122440.SAFE
S2A_MSIL1C_20200424T103021_N0209_R108_T32UPB_20200424T124310.SAFE
S2A_MSIL1C_20200428T101031_N0209_R022_T32UPB_20200428T111417.SAFE
S2A_MSIL1C_20200428T101031_N0209_R022_T32UQB_20200428T111417.SAFE
S2A_MSIL2A_20200418T101031_N9999_R022_T32UPB_20200430T165258.SAFE

And the files.txt looks like this: (just that I created it with python)

2A_MSIL1C_20200418T101031_N0209_R022_T32UPB_20200418T122607.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UPB_20200421T122440.SAFE
S2B_MSIL1C_20200426T101549_N0209_R065_T32UPB_20200426T131809.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UQB_20200421T111956.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UPB_20200421T111956.SAFE
S2B_MSIL1C_20200423T100549_N0209_R022_T32UQB_20200423T135558.SAFE
S2B_MSIL1C_20200423T100549_N0209_R022_T32UPB_20200423T135558.SAFE
S2A_MSIL1C_20200428T101031_N0209_R022_T32UQB_20200428T111417.SAFE
S2B_MSIL1C_20200426T101549_N0209_R065_T32UQB_20200426T131809.SAFE
S2A_MSIL1C_20200421T102021_N0209_R065_T32UQB_20200421T122440.SAFE
S2A_MSIL1C_20200424T103021_N0209_R108_T32UPB_20200424T124310.SAFE
S2B_MSIL1C_20200419T102549_N0209_R108_T32UPB_20200419T124956.SAFE
S2A_MSIL1C_20200428T101031_N0209_R022_T32UPB_20200428T111417.SAFE
S2A_MSIL1C_20200418T101031_N0209_R022_T32UQB_20200418T122607.SAFE                                                                                                                                                  

Afterwards I run parallel -j4 <some_programm> :::: files.txt on the command line.

This all works, but I'd like to make all this in one bash script. One of the reasons is, that I'd like to learn some shell-scripting.

I'm thankful for any advice:)

2
  • So what should the script do exactly? Why don't you just run parallel -j4 <some_programm> ::: *SAFE?
    – terdon
    Apr 30, 2020 at 17:34
  • yes I did this. I just wanted to know how to do it in bash. Apr 30, 2020 at 19:22

1 Answer 1

4

If you want to use parallel and run a command on all the files ending in .SAFE, all you need is:

parallel -j4 <some_program> ::: *.SAFE

Or, if your file names can contain \n, use -0:

parallel -j4 -0 <some_program> ::: *.SAFE

Saving filenames in a file is rarely a good approach. For one thing, it will break if the filenames can contain \n (newline) characters unless you make the file null-separated. You usually want to use a glob instead. Still, here are a few ways of getting filenames into a file:

  1. Simple, but can't deal with newlines in file names:

    printf '%s\n' *.SAFE > filenames.txt
    
  2. More complex, but can deal with arbitrary file names:

    find . -type f -name '*.SAFE' -print0 > filenames
    

    That creates a NULL-separated file, so you need a tool that can handle such input. parallel can do it with -0:

    parallel -0 -j4 <some_program> :::: filenames
    

So with this approach, your script could be:

#!/bin/sh
find . -type f -name '*.SAFE' -print0 > filenames
parallel -0 -j4 <some_program> :::: filenames

But, if your file names don't have spaces or newlines or any other strangeness, just run parallel -j4 -0 <some_program> ::: *.SAFE and be done with it.

3
  • parallel -j4 <some_program> ::: *.SAFE deals correctly with any strangeness except \n and \0 (in particular space, ', ", and * are absolutely fine). parallel -j4 -0 <some_program> ::: *.SAFE deals correctly with any strangeness except \0.
    – Ole Tange
    May 5, 2020 at 22:59
  • @OleTange ah, thanks, I thought ::: could deal with newlines without the -0 since it was passed a glob. Answer edited.
    – terdon
    May 6, 2020 at 8:10
  • Yeah, it is one of those corner cases that may be changed in future versions, because it is not what you would expect.
    – Ole Tange
    May 6, 2020 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.