1

Assume the data consists of byte offset which is not fixed i.e. the distance of two subsequent file headers varies. The point of this thread is to go through each size of events separately in arrays.

Example data

fafafafa
00005e58
da1e5afe
00000000
*
fafafafa
00005e58
da1e5afe
00000000
*
00000001
ffffffff
555eea72
00000000
*
00000004
fafafafa
01da1300
*
00000004
02991c00
fafafafa
01da1300
fafafafa
01da1300
fafafafa
01da1300

where the field deliminator is fafafafa.

My proposal

#!/bin/bash
# http://stackoverflow.com/a/10383546/54964

# http://unix.stackexchange.com/a/209789/16920
myarr = ($( cat 25.6.2015_test.txt | awk -F 'fafafafa' '$1~/^[a-z0-9*]+$/ {print $1}') )

# http://stackoverflow.com/a/15105237/54964
# Now access elements of an array (change "1" to whatever you want)
echo ${myarr[1]}

# Or loop through every element in the array
for i in "${myarr[@]}"
do
   :
  echo $i
done

Script run as a whole

Output

awk2array.sh: line 5: syntax error near unexpected token `('
awk2array.sh: line 5: `myarr = ($( cat 25.6.2015_test.txt | awk -F 'fafafafa' '$1~/^[a-z0-9*]+$/ {print $1}') ) '

which I do not understand, since even brackets. I would like to get the output into an array or store each event into a file named arithmetically (0.txt, 1.text, ..., n.txt). I now describe some of the commands separately and some parts of the codes about which I am uncertain.

AWK command run separately

The AWK command when run separately omits the field deliminator, giving

00005e58
da1e5afe
00000000
*
00005e58
da1e5afe
00000000
*
00000001
ffffffff
555eea72
00000000
*
00000004
01da1300
*
00000004
02991c00
01da1300
01da1300
01da1300

Wanted output is to have all data in array where the field separator is fafafafa such that fafafafa should be included in the cell, for instance

Value of first cell in array
----------------------------
fafafafa
00005e58
da1e5afe
00000000
*

Value of second cell
--------------------
fafafafa
00005e58
da1e5afe
00000000
*
00000001
ffffffff
555eea72
00000000
*
00000004

3rd cell
--------
01da1300
*
00000004
02991c00

4th cell
--------
fafafafa
01da1300

5th cell
--------
fafafafa
01da1300

6th cell
--------
fafafafa
01da1300

How can you store big data into N array by AWK? You can also store each event into file after reading it without starting to read the file again and continuing from the point where left.

3

Problem

So many things wrong here

#!/bin/bash

myarr = (

has got a space between it meaning nothing is assigned if it even runs at all.

cat 25.6.2015_test.txt | awk

Awk can open its own files no need for cat

-F 'fafafafa' '$1~/^[a-z0-9*]+$/

-F is the field separator not record, so all this is doing is removing the text fafafafa, it's still reading each line as a record so your next condition is entirely pointless.

myarr = ($( cat 25.6.2015_test.txt | awk -F 'fafafafa' '$1~/^[a-z0-9*]+$/ {print $1}') )

This will print multiple lines which will all be separate elements in the array as they are split on newlines and have no visibility of what is a record in awk(if you had actually split on records instead of fields).

echo ${myarr[1]}
echo $i

Quote these unless you want to see all the files in your directory everytime you echo (due to the * in the records)

 :

Why ?


Solution

# Create an array
myarr=()
# Save the number of different blocks to be saved, notice the 
# `-vRS` which sets the field separator
blocks=$(awk -vRS='fafafafa' '$1~/^[a-z0-9*]+$/{x++}END{print x}' file)

# While the the counter is less than the number of blocks.
while [[ $x -le $blocks ]] ;do
    # Increase the counter
    ((x++))
    # Add the value for that block to the array, notice the quotes around
    # `$()`, they are important in keeping all the block as one array 
    # element. The awk also increments its own counter for each 
    # occurrence of 'fafafafa' and your condition for '$1'. When both
    # counters match the block is saved to the array.
    myarr+=("$(awk -vRS='fafafafa' -vN="$x" '$1~/^[a-z0-9*]+$/{x++}
                                             x==N{print RS$0}' test)")

done
  • 1
    can you put your final code all in one piece ? – Archemar Jun 25 '15 at 8:49
  • @Archemar How come ? – 123 Jun 25 '15 at 8:55
  • Yes, it would be easier to read your code as a single block at the end of your answer. – Léo Léopold Hertz 준영 Jun 25 '15 at 10:00
3
while read -d '&' -r data
do 
    myarr[${#myarr[@]}]="$data" 
done < <(sed '1! s/fafafafa/\&&/' 25.6.2015_test.txt) 

will put all your data from file 25.6.2015_test.txt into array myarr separated by fafafafa altogether with it. sed is used to put delimeter character & (you can use any which is not expected in the text) before fafafafa except first line (in opposite case we receive empty 1st memeber of the array). read puts portion of text separated by & into intermediate variable data. ${#myarr[@]} produce number of elements in array myarr. Since the numeration starts from 0 thus we can receive index of the next element of the array:

  • array empty, number of elements is 0, so first element has index == 0
  • array has 1 element with index 0, so number of elements is 1, next index == 1
  • array has 2 elements with indexes 0,1, so number of elements is 2, next index == 2
  • I like this approach a lot! You can include by it for instance 100 events in one file, for instance something like gsed 's/fafafafa/\&&/100g'. – Léo Léopold Hertz 준영 Jun 25 '15 at 8:20
  • 1
    Can you add an explanation please :) – 123 Jun 25 '15 at 8:27
  • I get these errors in running your code in OSX 10.10.3 Yosemite. 25.6.2015_costa.sh: line 7: syntax error near unexpected token <'; 25.6.2015_costa.sh: line 7: done < <(sed '1! s/fafafafa/\&&/' 25.6.2015_test.txt) '. What do you get? I will try next Redhat. – Léo Léopold Hertz 준영 Jun 25 '15 at 9:38
  • I get in Redhat Linux also similar errors 25.6.2015_costa.sh: line 4: syntax error near unexpected token <'; 25.6.2015_costa.sh: line 4: done < <(sed '1! s/fafafafa/\&&/' 25.6.2015_test.txt) '. What is your output? – Léo Léopold Hertz 준영 Jun 25 '15 at 9:57
1

The line

myarr = ($( cat 25.6.2015_test.txt | awk -F 'fafafafa' '$1~/^[a-z0-9*]+$/ {print $1}') 

is wrong. Use the line below:

myarr=$(awk -F 'fafafafa' '$1~/^[a-z0-9*]+$/ {print $1}' 25.6.2015_test.txt) 

And you should use ":

echo "${myarr[1]}"

and

echo "$i"

And you could use this awk command

  • with fafafafa:

    awk '{if ($1 ~ /^fafafafa$/) {line+=1; print ""; print "cell "line;print "--------"; print $1} else {print $1}}' 25.6.2015_test.txt
    
  • without fafafafa

    awk '{if ($1 ~ /^fafafafa$/) {line+=1; print ""; print "cell "line;print "--------";} else {print $1}}' 25.6.2015_test.txt
    

Example output without fafafafa

cell 1
--------
00005e58
da1e5afe
00000000
*

cell 2
--------
00005e58
da1e5afe
00000000
*
00000001
ffffffff
555eea72
00000000
*
00000004

cell 3
--------
01da1300
*
00000004
02991c00

cell 4
--------
01da1300

cell 5
--------
01da1300

cell 6
--------
01da1300
  • This does just print data, OP want to store data into awk (I supposed to search cell containing XY, or put in separated file). – Archemar Jun 25 '15 at 8:01
  • I edited the body such that you can also store each event into file system directly without starting reading file again. I think this approach of storing each event into file system is much better approach than reading the file long time and lastly making the array. The array approach should work such that each new event is added to the end of the array. – Léo Léopold Hertz 준영 Jun 25 '15 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.