2

I need this to be more efficient

Right now it takes up to 20 hrs depending on the line (these are fairly large MCS datasets).

  • Split large data file into its "shots"
  • Creates a list of each shot name to be used in for loop
  • Loops through each shot and performs the same processes
  • Appends each shot to a new data file, so that you have the same line aa before, but processed. In this case i am filtering the data repeatedly, which is why I think this can be run in parallel.

You can ignore all of the SU commands and as well as everything in the for loop, I just need to know how to run this in parallel (say 32 nodes). This is a relatively new topic for me, so an in depth explanation would be appreciated!

script:

#! /bin/bash    
# Split the input file into one file for each shot. NB mustclose each o/p file at the earliest opportunity otherwise it will crash!
susplit <$1 key=fldr stem=fldr_ verbose=1 close=1

# Create a list of shot files
ls fldr* > LIST

# Loop over each shot file; suppress direct wave; write to new concatenated output file
for i in `cat LIST`; do
    echo $i
    suchw key1=tstat key2=tstat a=200 < $i | suwind key=tracf min=10 max=400 tmin=0 tmax=6 | suweight a=0 | suresamp rf=4 | sustatic hdrs=1 sign=-1 | sureduce rv=1.52 | sumedian median=1 xshift=0 tshift=0 nmed=41 | suflip flip=3 | sureduce rv=1.52 | suflip flip=3 | suresamp rf=0.25 | suweight inv=1 a=0 | sustatic hdrs=1 sign=1 >> $2
done

# Tidy up files by removing single shot gathers and LIST
rm -f fldr* LIST &
0

1 Answer 1

4

I assume it is the for loop you want parallelized:

#! /bin/bash    
# Split the input file into one file for each shot. NB mustclose each o/p file at the earliest opportunity otherwise it will crash!
susplit <$1 key=fldr stem=fldr_ verbose=1 close=1

sucit() {
    i=$1
    echo $i
    suchw key1=tstat key2=tstat a=200 < $i | suwind key=tracf min=10 max=400 tmin=0 tmax=6 | suweight a=0 | suresamp rf=4 | sustatic hdrs=1 sign=-1 | sureduce rv=1.52 | sumedian median=1 xshift=0 tshift=0 nmed=41 | suflip flip=3 | sureduce rv=1.52 | suflip flip=3 | suresamp rf=0.25 | suweight inv=1 a=0 | sustatic hdrs=1 sign=1
}
export -f sucit

parallel sucit ::: fldr* > $2

# Tidy up files by removing single shot gathers and LIST
rm -f fldr* LIST &

Depending on what susplit does you can make it even faster. If a shot in "large_data_file" starts with <shot>\n and ends with </shot>\n then something like this may work:

sucpipe() {
    suchw key1=tstat key2=tstat a=200 | suwind key=tracf min=10 max=400 tmin=0 tmax=6 | suweight a=0 | suresamp rf=4 | sustatic hdrs=1 sign=-1 | sureduce rv=1.52 | sumedian median=1 xshift=0 tshift=0 nmed=41 | suflip flip=3 | sureduce rv=1.52 | suflip flip=3 | suresamp rf=0.25 | suweight inv=1 a=0 | sustatic hdrs=1 sign=1
}
export -f sucpipe

parallel --block -1 --recstart '<shot>\n' --recend '</shot>\n' --pipepart -a $1 sucpipe > $2

It will try to split bigfile into n blocks, where n=number of cores. The splitting is done on the fly so it will not write temporary files first. Then GNU Parallel will pass each block to a sucpipe.

If bigfile is binary (i.e. not text) with a header of 3200 bytes and a recordlength of 1000 bytes, then this might work:

parallel -a bigfile  --pipepart --recend '' --block 1000 --header '.{3200}' ...

For more details walk through the tutorial: man parallel_tutorial Your command line will love you for it.

8
  • Sweet, I will give this a shot and let you know how it works. Thank you very much!
    – acquisto2
    Commented Mar 24, 2017 at 1:56
  • BTW, susplit creates a separate file for each shot, which is a collection of traces (data) that have a header value of that same shot number. These files are only used to create a list to run the for loop, and are deleted later.
    – acquisto2
    Commented Mar 24, 2017 at 1:59
  • @acquisto2 If you update your question with two example shots from bigfile, it may be easier to show what the second solution should look like.
    – Ole Tange
    Commented Mar 24, 2017 at 7:30
  • @acquisto2 GNU Parallel can handle your 32 nodes. Read about --sshlogin in the man page or give some more information. Your echo $i looks like you want to monitor how it's going and you can use --etaor --joblog (with tail -f) for that.
    – hschou
    Commented Mar 24, 2017 at 8:37
  • I guess I should have specified, but I am not running GNU. Our server is Unix and does not have the commands above...
    – acquisto2
    Commented Mar 25, 2017 at 17:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .