6

I have to write a Bash script which checks if another Bash script contains a certain command line. Since Bash allows you to split a command line over multiple lines, my script must be able to merge the corresponding lines before the actual pattern matching can take place.

How do I parse all multi-line into single-line commands in a Bash script?

Example

I want to check if a certain script contains the ls command - and if it contains the ls command, I want to know which parameters are passed to the ls command. To answer this question, I could use sed. But therefore I have to merge all multi-line commands first.

Input:

# My comment \
ls \
-a \
-l

Output:

# My comment \
ls -a -l

Example for an invalid output:

# My comment ls -a -l
7

Just before shellshock, I answered a question on StackOverflow about eliminating comments in bash scripts. My answer used the simple trick of creating a function by enclosing the contents of the script file inside tmp_() { ... }, and then using declare -f tmp_ to pretty-print the function. In the pretty-printed output, there are no comments and lines continued with a backslash-newline have been resolved to single lines. (Except inside backticked command substitution.)

Some other reformatting is also done. For example, compound commands are split into several lines. And some forms of line continuation are not reformatted; for example, a line ending with a pipe symbol is not altered. But it should satisfy the use-case in this question. (See example output below.)

Of course, the function definition needs to be evaluated, which means that the script being pretty-printed might include an injection attack. In the code I suggested, the function definition is evaluated by way of the bash feature which allows functions to be exported and shared with a child process. At the time I wrote this little hack, I believed that mechanism to be safer than calling eval, but as it turns out I was wrong.

Since shellshock, there have been a number of improvements to the code bash uses to import function definitions, closing the door on at least some injection attacks, but there is clearly no assurance that the procedure is completely safe.

If you are going to run the script being analyzed, then using this procedure to pretty-print it probably does not increase your vulnerability; an attacker could simply insert the dangerous code directly in the script and there would be no need to jump through hoops to hide the attack in a way which might bypass the safety checks in the function import code.

All the same, you should think carefully about security issues, both with this little program and with whatever plans you might have to execute arbitrary scripts.

Here is the version of the pretty-printer which works with a post-shellshock-patched bash (and will not work with previous bash versions):

env "BASH_FUNC_tmp_%%=() {
$(<script_name)
}" bash -c 'declare -f tmp_' | tail -n+2

Substitute the name of the file containing the script for script_name, in the second line. You might want to adjust the tail command; it removes the wrapper function name, but does not remove the braces which surround the script body.

The original version, which will work on pre-shellshock versions of bash, can be found in the referenced SO answer.


Sample.

Tested against the input provided by Stéphane Chazelas:

{ 
    echo \\;
    echo a#b;
    echo 'foo\
bar';
    cat  <<EOF
thisis joined
this 'aswell'
$(ls -l)
EOF

    cat  <<'EOF'
this is\
not joined
EOF

    echo "$(ls -l)";
    echo `ls \\
-l`
}

This differs from Stéphane's suggested output:

  • Lines have been indented, and many have been terminated with semicolons. Whitespace has been added and/or deleted in many lines.
  • cat << E\OF has been changed to cat <<'EOF', which is semantically identical.
  • The nested continuation line in the backticked command substitution at the end has not been modified. (The continuation line in the $(...) command substituion is eliminated.)
  • Looks like you were very close to discovering shellshock then. – Stéphane Chazelas Sep 10 '15 at 8:18
2

This is not really an answer, just a note on the things to consider for a solution to work in the general case.

#! /bin/sh
echo \\
echo a#\
b
echo 'foo\
bar'
cat << EOF
this\
is joined
this 'as\
well'
$(ls \
-l)
EOF
cat << E\OF
this is\
not joined
EOF
echo "$(ls \
-l)"
echo `ls \\
-l`

My understanding of the intent of the question would be that it should be transformed to:

#! /bin/sh
echo \\
echo a#b
echo 'foo\
bar'
cat << EOF
thisis joined
this 'aswell'
$(ls -l)
EOF
cat << E\OF
this is\
not joined
EOF
echo "$(ls -l)"
echo `ls -l`
2

This works in more cases, now; see if it does what you're expecting:

sed ':loop /^[^#].*[^\\]\\$/N; s/\\\n//; t loop' input

It prints every line by default; if it finds a backslash (escaped because it's a special character) at the end of a line ($) -- and there is not a hash mark at the beginning of the line, then join it with the next line (N) modified by searching & replacing the backslash (escaped again) and newline character with nothing. If the search & replace did something, then go back to the "loop" tag and re-run the search.

Input:

# My comment \
ls \
-al

# leading comment
echo some \
long \
text
# trailing comment

ls -al

Output:

# My comment \
ls -al

# leading comment
echo some long text
# trailing comment

ls -al
  • This solution also merges comments. Example: It merges the lines "# My comment \" and "ls -al" into a single line "# My comment ls -al". Moreover, it doesn't work if you have a command which consists of more than two lines. – Lugaxx Sep 9 '15 at 15:24
  • 1
    I've updated the answer to take into account commented-and-backslashed lines as well as multi-lines. – Jeff Schaller Sep 9 '15 at 15:49
  • 1
    I added [^\\] for the double-backslash case; @StéphaneChazelas has pointed out some good cases. The rabbit-hole is too deep for me with here-docs and quoted strings. – Jeff Schaller Sep 9 '15 at 16:26
  • 1
    I've just upvoted your answer as it fullfills my requirements. I will accept this answer if nobody else provides a solution which also covers the cases which Stéphane Chazelas has pointed out. – Lugaxx Sep 9 '15 at 17:22
  • In case no one finds a general solution, you could work around the shortcomings of my approach by grepping for "<<" and " ' " and ' " ' (single and double quote marks) as places to manually inspect. – Jeff Schaller Sep 9 '15 at 17:43
0

The history converts typical multi line commands into a single line. Just type things in and then issue a "cursor up" to see the conversion.

  • 2
    Sorry, but this is a totally different topic. I know that my English isn't perfect, so if there is something unclear, just ask. – Lugaxx Sep 9 '15 at 13:18
  • I recommend you to try my proposal, it does apply to your question... – schily Sep 9 '15 at 16:58
  • While this would work in an interactive shell, I think the confusion stems from the question asking about making changes to a script file on disk -- not requiring the commands to be entered interactively (and thus into the shell history). – Jeff Schaller Sep 9 '15 at 17:26
0

To replace newlines by spaces, you can just

tr '\n' ' '

This can leave some backslashes in the code, so you should probably rather replace newlines preceded by backslash by spaces

 perl -pe 's/\\\n/ /'

This can still go wrong (HERE documents, comments, etc.), but to get it 100% correct, you'd have to write a full shell parser.

  • The problem with this solution is that it also merges lines which are commented. – Lugaxx Sep 9 '15 at 13:03
  • @Lugaxx: It mentions the problem, and also mentions the solution: writing a full shell parser. But do you really need to catch 100% of the matches? Can't you just overgenerate a bit and then process the resulting list manually? – choroba Sep 9 '15 at 13:05
  • Replacing newline by a space definitely does not work. In most cases, a semicolon works. – schily Sep 9 '15 at 13:08
  • @schily: For a single multiline command (check the example above), semicolon wouldn't work. – choroba Sep 9 '15 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.