0

I'm going to give a short version of the kind of issue I'm facing. The real application I'm trying to write has many parts, so I've tried to isolate the part where I'm stuck with a small example. My situation, which involves file ID numbers, is similar to a hypothetical list of Student ID Numbers parsed from a directory or something.


What I Start With

In order to try and take advantage of bash brace expansion, something like a student directory has been parsed to give the following:

$ cat student_id_potential_expansions
0000{11,24}
0001{32,38,81}
0002{02,80,81,89,97}

Desired behavior

I want to use this input to create my own (Bourne Again) shell script which looks something like the following.

#!/bin/bash

get_transcripts_for_id.sh -id 000011
get_transcripts_for_id.sh -id 000024
get_transcripts_for_id.sh -id 000132
get_transcripts_for_id.sh -id 000133
get_transcripts_for_id.sh -id 000181
get_transcripts_for_id.sh -id 000202
...

and so on

I'm not too worried about the command before; I figured I'd just use an

awk '{print "get_transcripts_for_id.sh -id" $0}' student_id_list > my_shell_script.sh

or a

sed 's/^\(.*\)$/get_transcripts_for_id.sh -id \1/' student_id_list > my_shell_script.sh

Right now, I'm just trying to get the list of id numbers (the student_id_list from above) from the list of potential expansions.

I think I know the basic problem, I just don't know how to fix it. I can get the pieces of the list I want as follows:

$ echo 0000{11,24} | tr ' ' '\n'
000011
000024

$ echo 0001{32,38,81} | tr ' ' '\n'
000132
000138
000181

$ echo 0002{02,80,81,89,97} | tr ' ' '\n'
000202
000280
000281
000289
000297

However, my first thought of a command didn't expand things

# doesn't work, as you can see
$ while read -r line; do echo $line; done < student_id_potential_expansions
0000{11,24}
0001{32,38,81}
0002{02,80,81,89,97}

I DO NOT want to just copy/paste every line. I've been given hundreds to thousands of the potentially-expandable lines - and these were parsed by someone else from who-knows-where, and I don't have access to the person who did the original parsing from records. Also, each of my lines is more like

0000009{882,739,861,014,952,611,862,935,976,916,080,697,323,843,840,487,517,407,256,756,374,682,162,930,758,157,770,505,867,233,198,131,917,848,613,247,961,261,616,392,876,747,873,148,844,849,280,626,817,819,174,771,172,284,217,200,018,624,418,292,642,529,755,855,647,317,881,962,975,237,635,805,298,835,053}

My Understanding

From the bash manual, section 3.5.1

Brace expansion is a mechanism by which arbitrary strings may be generated.
...
A correctly-formed brace expansion must contain unquoted opening and closing braces, and at least one unquoted comma or a valid sequence expression. Any incorrectly formed brace expansion is left unchanged.

As I understand it, when I read each line, I'm basically giving a string to be fed to whatever is inside the while loop, i.e. with my example above, I first give $line as "0000{11,24}" (note the quotes). Carrying this on further, It's as if my first trip through the loop results in the following command

echo "0000{11,24}"

Once again, notice the quotes, even though they were not around $line in my while loop. To see if this seemed consistent, I ran the following commands and got the following results:

$ echo "0000{11,24}"
0000{11,24}

$ echo "0001{32,38,81}"
0001{32,38,81}

$ echo "0002{02,80,81,89,97}"
0002{02,80,81,89,97}

So, as far as I can tell, all of the braces ({ and }) and commas are quoted - something that the bash manual says makes for an invalid expansion statement. The main question is, then, how to I un-quotify or un-stringify each $line that I read in?


My Attempts/Research

At the risk of making this post too long, here are some attempts I've made.

Looking through this SO post, I'm pretty sure my issue is not a word separation issue.

This answer and this comment, specifically the mention about

In bash, {} expansion happens before $ expansion, so I don't think there's any other way to do this other than using eval or some other trick to cause two passes through the expression.

made me think of trying

$ while read -r line; do eval "$line"; done < student_id_potential_expansions
bash: 000011: command not found
bash: 000132: command not found
bash: 000202: command not found

and

$ while read -r line; do echo $(eval "$line"); done < student_id_potential_expansions
bash: 000011: command not found

bash: 000132: command not found

bash: 000202: command not found

where it seems only the first expansion gets carried out before the shell tries to run (evaluate) the command.

I also tried the commands that follow, looking at the comment here.

$ while read -r line; do expansions=$line; echo $expansions; done < trying_inner_echo.txt
$(echo 0000{11,24})
$(echo 0001{32,38,81})
$(echo 0002{02,80,81,89,97})

$ while read -r line; do echo $line; done < trying_inner_echo.txt
$(echo 0000{11,24})
$(echo 0001{32,38,81})
$(echo 0002{02,80,81,89,97})

$ while read -r line; do threads=${line}; exec $threads; done < trying_inner_echo.efl
bash: exec: $(echo: not found
bash: exec: $(echo: not found
bash: exec: $(echo: not found

$ while read -r line; do threads=${line}; exec "${threads}"; done < trying_inner_echo.txt
bash: exec: $(echo 0000{11,24}): not found
bash: exec: $(echo 0001{32,38,81}): not found
bash: exec: $(echo 0002{02,80,81,89,97}): not found

Some other attempts:

$ while read -r line; do echo "$line"; done < student_id_potential_expansions
$ while read -r line; do echo "$(echo $line)"; done < student_id_potential_expansions

System Info

$ uname -a
CYGWIN_NT-10.0 C-D-ENG-E-INT3 2.11.2(0.329/5/3) 2018-11-08 14:34 x86_64 Cygwin

$ bash --version
GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ systeminfo | sed -n 's/^OS\ *//p'
Name:                   Microsoft Windows 10 Enterprise
Version:                10.0.17134 N/A Build 17134
Manufacturer:           Microsoft Corporation
Configuration:          Member Workstation
Build Type:             Multiprocessor Free
1

The trick is to use eval

A very small example :

( echo '000{90,91}' ; echo '002{10,11}' ; echo '110{50,51}' ) | \
       while read L ; do eval "echo $L" | tr ' ' '\n'  ; done
  • Thank you. It seems that what I needed to do was toswitch the eval and echo in my attempt with echo $(eval "$line"). Thanks for getting me there. When I used while read -r line; do eval "echo $line"; done < student_id_potential_expansions | tr ' ' '\n', I got exactly the output I was looking for. I also really like the way you did the example, with the parentheses around the echo statements. That allows a much shorter way of showing what I showed with a big file. – bballdave025 Nov 8 '19 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.