1

I am having a blonde moment.

This is my data from myfile.csv

1429829254,e,SE,StckXchg,HDCU3000623,d,scan,253,47968,94,1420824420,JSSE-213,199,BS,KIT,*OUT*
1429897704,e,SE,StckXchg,HDCU3000623,d,scan,253,68496104,103,1420923818,SP-TMX6BP-101-112Z,*OUT*
1429897718,e,SE,StckXchg,HDCU3000623,d,scan,253,68510090,104,1420923832,02132,*OUT*
1429897767,e,SE,StckXchg,HDCU3000623,d,scan,253,68559085,105,1420923878,02132,*OUT*
1429912759,e,SE,StckXchg,HDCU3000623,d,scan,255,47966,107,1420924040,JSF-1,4"MEG0004,*OUT*
1429912766,e,SE,StckXchg,HDCU3000623,d,scan,255,54955,108,1420924047,02134,*OUT*
1429982863,e,SE,StckXchg,HDCU3000623,d,scan,256,47965,120,1420987942,JSDL-JA120901S,*OUT*
1429982870,e,SE,StckXchg,HDCU3000623,d,scan,256,54953,121,1420987961,02132,*OUT*
1429982877,e,SE,StckXchg,HDCU3000623,d,scan,256,61954,109,1420971277,02134,*OUT*

I currently put this data into an array and extract each of the elements individually.

But our input data has changed and now there the elements with a line can vary- and can vary by a lot. I was curious to know how I can extract data from the above sample, from the 12th element up until the last element.

With my code this is what I do:

cat $out_file_name |  while read line
do
    awk_var=`echo "$line" | awk -F"," '{print $1, $5, $10, $12, $13, $14}'`         #Get the appropriate arguments
    awk_var_array=($awk_var)

    timestamp=${awk_var_array[0]}
    container==${awk_var_array[1]}
    scan_id=${awk_var_array[2]}
    part_no=${awk_var_array[3]}
    direction=${awk_var_array[4]}

    #further processing and send the above to a mysql table
done

but now this obviously doesn't work. So I need to find a way of storing all the element from the 12th element till the last element (in/out).

would it be done with a for loop? Something like this perhaps?

for element_count in $(seq 13 $({#awk_var_array[@]}-1))
do
    var_name=my_var_''$element_count
    $var_name=${awk_var_array[element_count]
done
  • It would help to see the loop you have around your posted code, and whether (as it seems) you throw away the first line of data while processing the second line, etc. - I suppose you might want something like: while IFS=, read timestamp x x x container x x x x scan_id x part_no direction rest ; do ...some_processing... ; done < file.csv. – Janis Apr 27 '15 at 5:34
  • @Janis, Could you elaborate? I'm a strong programmer -.- ! – 3kstc Apr 27 '15 at 5:41
  • Your code seems to have a loop that you did not post; how else would "$line" be else retrieved from your csv file? - So you'd not need all that echo, pipes, awk, arrays, etc.; rather just a loop over the csv file entries and a read to "unique identifiers" for your processing. The code would get quite simple and performant. - But to be sure that would require that you post the surrounding loop code (how you read the csv file) and explain the further processing of your variables (it doesn't seem you need them stored in arrays). - Note: the 'x' in my code are ignored fields. – Janis Apr 27 '15 at 5:45
  • I just did - it's a while read line from a csv file. I process the data - concatenate some identifiers then shoot it off over to a mysql table... – 3kstc Apr 27 '15 at 5:50
  • Okay, so my assumption seems to have been correct. I posted an answer based on that. – Janis Apr 27 '15 at 5:55
1

You can use bash parameter expansion ${parameter:offset:length} form with array:

$ array=($(seq 20))
$ printf "%s " "${a[@]: -12}"
9 10 11 12 13 14 15 16 17 18 19 20
  • Perfect! But how can I assign a unique identifier to each of the values? – 3kstc Apr 27 '15 at 5:35
  • @3kstc: I'm not sure what do you mean unique identifier here? You can store that sub array into another array for manipulation. Or you can get each element starting from 12th element. Example ${a[@]: -12:1} will get the 12th element, ${a[@]: -12:2} will get the 13th element and so on. – cuonglm Apr 27 '15 at 5:45
0

Based on the further information we got from the various comments, I think all you need is this code:

while IFS=, read timestamp x x x container x x x x scan_id x part_no direction rest
do
   # compose SQL command using "${timestamp}" "${container}" ... "${direction}" "${rest}" and send it to mysql database
done < "${out_file_name}"

Note: The x in the read command represent dummy fields that will be skipped (and not assigned an own unique identifier), and the variable rest contains all the fields in one variable beyond the ones explicitly enumerated in the read argument list.

  • Thanks Janis - one last question - how would I modify the code if I wanted to get for example from part_no field all the way to the n-1th field and than also the nth field. Where n is the total number of fileds – 3kstc Apr 27 '15 at 6:09
  • @3kst; You mean you are not interested in details of part_no and subsequent fields, and it suffices for you to have them collected all in one variable? - Then reduce the read variable list: read timestamp x x x container x x x x scan_id x rest. - The point is; increase the list with as many variables as you are interested to distingush individually, and subsume the rest in a final variable rest. If there are less fields in the data than the read command has then the respective fields will just be blank, if there are more fields, the remaining are in the final variable; no harm. – Janis Apr 27 '15 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.