2

I'm working on a script that chews on some data it sucks in from a CSV file. I've already read the data into several arrays (one for each column in the file); I now need to actually work with all of the data in sequence.

Currently, I am doing this:

# Read in the data:
declare -a DATACOL1 DATACOL2 RAWDATA
RAWDATA=($( sed '1d' /path/to/data.csv )) # Remove the header line
for line in ${RAWDATA[@]}; do
   declare -a LINEDATA LINE
   LINE=$( echo "$line" | sed 's/,/ /g' )
   for field in LINE; do
       LINEDATA+=("${field}")
   done
   DATACOL1+=(${LINEDATA[0]})
   DATACOL2+=(${LINEDATA[1]})
done


# Work on the data:
for i in $( seq 0 $[${#DATACOL1[@]}-1}; do
   stuff and things with ${DATACOL1[i]} and ${DATACOL2[i]}
done

My (quite possibly interrelated) questions are twofold:

  • Is there a more elegant way to later work in the data than for i in $( seq 0 $[${#DATACOL1[@]}-1} for iterating over them? It works, but is ugly.

  • Is there a more elegant way to suck in the CSV data?

This is on bash 3, so I do not have associative arrays.

6
  • Can we write it in perl? (typo: seed -> sed)
    – JJoao
    Nov 17, 2015 at 19:08
  • This would probably be a lot easier to do in perl/python, but it's preferred in this instance to stick with bash, as it's one cog in a fairly extensive bash Rube Goldberg machine, and I'd just as soon keep all the cogs speaking the same language.
    – DopeGhoti
    Nov 17, 2015 at 19:09
  • 1
    Your method of reading the file will break if there is any whitespace. Nov 17, 2015 at 19:28
  • 1
    there's nothing elegant about shell arrays. they are large, ponderous, and wasteful.
    – mikeserv
    Nov 17, 2015 at 20:33
  • 1
    I didn't say elegant, I said more elegant; it's a quality I'm striving for even if it can't actually be reached.
    – DopeGhoti
    Nov 17, 2015 at 20:42

1 Answer 1

4

I would write:

mapfile -t rawdata < <(sed 1d /path/to/data.csv)
datacol1=()
datacol2=()

for line in "${rawdata[@]}"; do
    IFS=, read -ra fields <<< "$line"
    datacol1+=( "${fields[0]}" )
    datacol2+=( "${fields[1]}" )
done

for ((i=0; i < "${#datacol1[@]}"; i++)); do
    stuff with "${datacol1[i]}" and "${datacol2[i]}"
done
  • use mapfile to read the lines of a file into an array
  • use IFS and read to read comma-separated fields from a line
    • will break with any commas inside quoted strings: use a real CSV parser.
  • use the C-like form of for to avoid calling out to seq

Of course, you don't need the 2nd loop or the datacol* variables

for line in "${rawdata[@]}"; do
    IFS=, read -a fields <<< "$line"
    stuff with "${fields[0]}" and "${fields[1]}"
done

On bash 3 you won't have mapfile, so use a while-loop

datacol1=()
datacol2=()
while IFS=, read -ra fields; do
    datacol1+=("${fields[0]}")
    datacol2+=("${fields[1]}")
done < <(sed 1d /path/to/data.csv)
2
  • Thanks, mucking about with IFS is obvious in retrospect and is exactly the answer I needed. And the for loop is more verbose, but a lit less ugly than seq 0 $[${#DATACOL1[@]}-1}. Thanks for both.
    – DopeGhoti
    Nov 17, 2015 at 20:10
  • This doesn't seem to want to run in parallel for me, the last example. May 12, 2021 at 18:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .