2

I need this to be more efficient

Right now it takes up to 20 hrs depending on the line (these are fairly large MCS datasets).

  • Split large data file into its "shots"
  • Creates a list of each shot name to be used in for loop
  • Loops through each shot and performs the same processes
  • Appends each shot to a new data file, so that you have the same line aa before, but processed. In this case i am filtering the data repeatedly, which is why I think this can be run in parallel.

You can ignore all of the SU commands and as well as everything in the for loop, I just need to know how to run this in parallel (say 32 nodes). This is a relatively new topic for me, so an in depth explanation would be appreciated!

script:

#! /bin/bash    
# Split the input file into one file for each shot. NB mustclose each o/p file at the earliest opportunity otherwise it will crash!
susplit <$1 key=fldr stem=fldr_ verbose=1 close=1

# Create a list of shot files
ls fldr* > LIST

# Loop over each shot file; suppress direct wave; write to new concatenated output file
for i in `cat LIST`; do
    echo $i
    suchw key1=tstat key2=tstat a=200 < $i | suwind key=tracf min=10 max=400 tmin=0 tmax=6 | suweight a=0 | suresamp rf=4 | sustatic hdrs=1 sign=-1 | sureduce rv=1.52 | sumedian median=1 xshift=0 tshift=0 nmed=41 | suflip flip=3 | sureduce rv=1.52 | suflip flip=3 | suresamp rf=0.25 | suweight inv=1 a=0 | sustatic hdrs=1 sign=1 >> $2
done

# Tidy up files by removing single shot gathers and LIST
rm -f fldr* LIST &
4

I assume it is the for loop you want parallelized:

#! /bin/bash    
# Split the input file into one file for each shot. NB mustclose each o/p file at the earliest opportunity otherwise it will crash!
susplit <$1 key=fldr stem=fldr_ verbose=1 close=1

sucit() {
    i=$1
    echo $i
    suchw key1=tstat key2=tstat a=200 < $i | suwind key=tracf min=10 max=400 tmin=0 tmax=6 | suweight a=0 | suresamp rf=4 | sustatic hdrs=1 sign=-1 | sureduce rv=1.52 | sumedian median=1 xshift=0 tshift=0 nmed=41 | suflip flip=3 | sureduce rv=1.52 | suflip flip=3 | suresamp rf=0.25 | suweight inv=1 a=0 | sustatic hdrs=1 sign=1
}
export -f sucit

parallel sucit ::: fldr* > $2

# Tidy up files by removing single shot gathers and LIST
rm -f fldr* LIST &

Depending on what susplit does you can make it even faster. If a shot in "large_data_file" starts with <shot>\n and ends with </shot>\n then something like this may work:

sucpipe() {
    suchw key1=tstat key2=tstat a=200 | suwind key=tracf min=10 max=400 tmin=0 tmax=6 | suweight a=0 | suresamp rf=4 | sustatic hdrs=1 sign=-1 | sureduce rv=1.52 | sumedian median=1 xshift=0 tshift=0 nmed=41 | suflip flip=3 | sureduce rv=1.52 | suflip flip=3 | suresamp rf=0.25 | suweight inv=1 a=0 | sustatic hdrs=1 sign=1
}
export -f sucpipe

parallel --block -1 --recstart '<shot>\n' --recend '</shot>\n' --pipepart -a $1 sucpipe > $2

It will try to split bigfile into n blocks, where n=number of cores. The splitting is done on the fly so it will not write temporary files first. Then GNU Parallel will pass each block to a sucpipe.

If bigfile is binary (i.e. not text) with a header of 3200 bytes and a recordlength of 1000 bytes, then this might work:

parallel -a bigfile  --pipepart --recend '' --block 1000 --header '.{3200}' ...

For more details walk through the tutorial: man parallel_tutorial Your command line will love you for it.

  • Sweet, I will give this a shot and let you know how it works. Thank you very much! – acquisto2 Mar 24 '17 at 1:56
  • BTW, susplit creates a separate file for each shot, which is a collection of traces (data) that have a header value of that same shot number. These files are only used to create a list to run the for loop, and are deleted later. – acquisto2 Mar 24 '17 at 1:59
  • @acquisto2 If you update your question with two example shots from bigfile, it may be easier to show what the second solution should look like. – Ole Tange Mar 24 '17 at 7:30
  • @acquisto2 GNU Parallel can handle your 32 nodes. Read about --sshlogin in the man page or give some more information. Your echo $i looks like you want to monitor how it's going and you can use --etaor --joblog (with tail -f) for that. – hschou Mar 24 '17 at 8:37
  • I guess I should have specified, but I am not running GNU. Our server is Unix and does not have the commands above... – acquisto2 Mar 25 '17 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.