6

I have file.txt that I need to read into a Bash array. Then I need to remove spaces, double quotes and all but the first comma in every entry. Here's how far I've gotten:

$ cat file.txt
10,this
2 0 , i s
30,"all"
40,I
50,n,e,e,d,2
60",s e,e"

$ cat script.sh
#!/bin/bash
readarray -t ARRAY<$1
ARRAY=( "${ARRAY[@]// /}" )
ARRAY=( "${ARRAY[@]//\"/}" )
for ELEMENT in "${ARRAY[@]}";do
    echo "|ELEMENT|$ELEMENT|"
done

$ ./script.sh file.txt
|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,n,e,e,d,2|
|ELEMENT|60,se,e|

Which works great except for the comma situation. I'm aware that there are multiple ways to skin this cat, but due to the larger script this is a part of, I'd really like to use parameter substitution to get to here:

|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,need2|
|ELEMENT|60,see|

Is this possible via parameter substitution?

  • 3
    Is there any reason you need to keep the text in an array, and why you can't let e.g. awk or sed do the processing of the data? – Kusalananda Mar 26 at 15:49
  • @Jeff -- Looping over the array will be a nightmare to implement in the larger script I'm working on. – Jon Red Mar 26 at 16:04
  • 3
    @JonRed I don't know what you are doing, so it's entirely possible that you may not have a choice in the matter, but generally, when you find yourself doing such complex string acrobatics in the shell, that's a very good indication that you should be using an actual programming language. The shell is not designed as a programming language, and while it can be used as one, it really isn't a good idea for more complex things. I strongly urge you to consider switching to perl or python or any other scripting language. – terdon Mar 26 at 16:12
  • @terdon It's funny, I just got done saying almost the exact same thing to my colleague before I read this post. I basically said this is the final version of this script and that any further requirements will necessitate re-writing in Perl. So yeah, I definitely agree – Jon Red Mar 26 at 17:37
9

I would remove what you need to remove using sed before loading into the array (also note the lower case variable names, in general it is best to avoid capitalized variables in shell scripts):

#!/bin/bash
readarray -t array< <(sed 's/"//g; s/  *//g; s/,/"/; s/,//g; s/"/,/' "$1")
for element in "${array[@]}";do
    echo "|ELEMENT|$element|"
done

This produces the following output on your example file:

$ foo.sh file 
|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,need2|
|ELEMENT|60,see|

If you really must use parameter substitution, try something like this:

#!/bin/bash
readarray -t array< "$1"
array=( "${array[@]// /}" )
array=( "${array[@]//\"/}" )
array=( "${array[@]/,/\"}" )
array=( "${array[@]//,/}" )
array=( "${array[@]/\"/,}" )

for element in "${array[@]}"; do
    echo "|ELEMENT|$element|"
done
  • 1
    @JonRed I added a version with parameter substitution but it's complex, cumbersome and ugly. Doing this sort of thing in the shell is very rarely a good idea. – terdon Mar 26 at 16:06
  • 1
    Note that if you've removed both spaces and double quotes, these characters becoma available to use instead of your RANDOMTEXTTHATWILLNEVERBEINTHEFILE. – Kusalananda Mar 26 at 16:08
  • 1
    @Kusalananda yeah, I just read your answer. Should have thought of that! Thanks :) – terdon Mar 26 at 16:10
  • Directly answers the question, illustrates why my preferred solution isn't ideal, and provides the most viable alternative. You win, best answer. – Jon Red Mar 26 at 16:20
10

As far as I can see, there's no need to read it into a bash array to create that output:

$ sed 's/[ "]//g; s/,/ /; s/,//g; s/ /,/; s/.*/|ELEMENT|&|/' <file
|ELEMENT|10,this|
|ELEMENT|20,is|
|ELEMENT|30,all|
|ELEMENT|40,I|
|ELEMENT|50,need2|
|ELEMENT|60,see|

The sed expression deletes spaces and double quotes, replaces the first comma with a space (there are no other spaces in the string at this point), deletes all other commas, restores the first comma, and the prepends and appends the extra data.

Alternatively, with GNU sed:

sed 's/[ "]//g; s/,//2g; s/.*/|ELEMENT|&|/' <file

(standard sed does not support the combination of 2 and g as flags to the s command).

  • 1
    with GNU sed, you can use 's/,//2g to remove commas, starting with the 2nd – glenn jackman Mar 26 at 16:02
  • 2
    And, the last 2 s/// commands can be s/.*/|ELEMENT|&|/ but that may be more effort for sed. – glenn jackman Mar 26 at 16:04
  • 1
    @glennjackman Possibly, but it looks rather neat. – Kusalananda Mar 26 at 16:05
  • Yeah, this is part of a larger script. The array is necessary, not just for the output. Hence my interest in parameter substitution. I could loop over the array with this but that will be a nightmare to implement. Terndon provided a loop-free solution using sed that I'll likely fall back on if parameter substitution is a no-go. – Jon Red Mar 26 at 16:06
  • If I wasn't tied to using an array, however, this would be best solution. – Jon Red Mar 26 at 16:24
9
ELEMENT='50,n,e,e,d,2'
IFS=, read -r first rest <<<"$ELEMENT"
printf "%s,%s\n" "$first" "${rest//,/}"
50,need2

Get out of the habit of using ALLCAPS variable names. You'll eventually collide with a crucial "system" variable like PATH and break your code.

  • Not parameter substitution. BUT, I was unaware that ALLCAPS variable names was a bad habit in Bash. You make a good point, one that a cursory googling definitely confirms. Thank you for improving my style! :) – Jon Red Mar 26 at 16:14
  • 1
    I've answer questions where the person wrote PATH=something; ls $PATH and then wondered about the ls: command not found error. – glenn jackman Mar 26 at 16:17
  • 1
    There are nearly a hundred built-in variables that are named in all caps (click through this man page link) to see... – Jeff Schaller Mar 26 at 16:19
8

[This is essentially a more fully developed version of glenn jackmann's answer]

Building an associative array from the stripped key and value, using the first comma as separator:

declare -A arr
while IFS=, read -r k v; do arr["${k//[ \"]}"]="${v//[ ,\"]}"; done < file.txt
for k in "${!arr[@]}"; do 
  printf '|ELEMENT|%s,%s|\n' "$k" "${arr[$k]}"
done
|ELEMENT|20,is|
|ELEMENT|10,this|
|ELEMENT|50,need2|
|ELEMENT|40,I|
|ELEMENT|60,see|
|ELEMENT|30,all|
6

You could loop over the array and use an intermediate variable:

for((i=0; i < "${#ARRAY[@]}"; i++))
do
  rest="${ARRAY[i]#*,}"
  ARRAY[i]="${ARRAY[i]%%,*}","${rest//,/}"
done

This assigns to rest the portion after the first comma; we then concatenate three pieces back into the original variable:

  • the portion before the first comma
  • a comma
  • the replacement in rest of every comma with nothing
  • This was my first thought and is simple enough for the example but this is part of larger script where the array is massive and there's already loops and it would be a whole thing. This would definitely work but would be very cumbersome to implement in the larger project I'm working on. – Jon Red Mar 26 at 16:26
  • 1
    Fair enough; I just tried to answer within the limitations (parameter expansion only). – Jeff Schaller Mar 26 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.