Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there a way to specify multiple variables (not just integers) in for loops in bash? I may have 2 files containing arbitrary text that i would need to work with.

What i functionally need is something like this:

for i in $(cat file1) and j in $(cat file2); do command $i $j; done

Any ideas?

share|improve this question

3 Answers 3

up vote 8 down vote accepted

First, Don't read lines with for, as there are several unavoidable issues with reading lines via word-splitting.

Assuming files of equal length, or if you want to only loop until the shorter of two files are read, a simple solution is possible.

while read -r x && read -r y <&3; do
    ...
done <file1 3<file2

Putting together a more general solution is hard because of when read returns false and several other reasons. This example can read an arbitrary number of streams and return after either the shortest or longest input.

#!/usr/bin/env bash

# Open the given files and assign the resulting FDs to arrName.
# openFDs arrname file1 [file2 ...]
openFDs() {
    local x y i arr=$1

    [[ -v $arr ]] || return 1
    shift

    for x; do
        { exec {y}<"$x"; } 2>/dev/null || return 1
        printf -v "${arr}[i++]" %d "$y"
    done
}

# closeFDs FD1 [FD2 ...]
closeFDs() {
    local x
    for x; do
        exec {x}<&-
    done
}

# Read one line from each of the given FDs and assign the output to arrName.
# If the first argument is -l, returns false only when all FDs reach EOF.
# readN [ -l ] arrName FD1 [FD2 ...]
readN() {
    if [[ $1 == -l ]]; then
        local longest
        shift
    else
        local longest=
    fi

    local i x y status arr=$1
    [[ -v $arr ]] || return 1
    shift

    for x; do
        if IFS= read -ru "$x" "${arr}[i]" || { unset -v "${arr}[i]"; [[ ${longest+_} ]] && return 1; }; then
            status=0
        fi
        ((i++))
    done
    return ${status:-1}
}

# readLines file1 [file2 ...]
readLines() {
    local -a fds lines
    trap 'closeFDs "${fds[@]}"' RETURN
    openFDs fds "$@" || return 1

    while readN -l lines "${fds[@]}"; do
        printf '%-1s ' "${lines[@]}"
        echo
    done
}

{
    readLines /dev/fd/{3..6} || { echo 'error occured' >&2; exit 1; }
} <<<$'a\nb\nc\nd' 3<&0 <<<$'1\n2\n3\n4\n5' 4<&0 <<<$'x\ny\nz' 5<&0 <<<$'7\n8\n9\n10\n11\n12' 6<&0

# vim: set fenc=utf-8 ff=unix ts=4 sts=4 sw=4 ft=sh nowrap et:

So depending upon whether readN gets -l, the output is either

a 1 x 7
b 2 y 8
c 3 z 9
d 4 10
5 11
12

or

a 1 x 7
b 2 y 8
c 3 z 9

Having to read multiple streams in a loop without saving everything into multiple arrays isn't all that common. If you just want to read arrays you should have a look at mapfile.

share|improve this answer
    
|| doesn't work for me. It process file1 then file2, but it does keep the files synchronized with &&, which exits the while loop upon first eof. -- GNU bash 4.1.5 –  Peter.O Jun 20 '12 at 23:30
    
I didn't have a free moment to test - yes you would probably want both elements per iteration. The || "list" operator is short-circuiting like in most languages. Surely this has been answered a thousand times before... –  ormaaj Jun 21 '12 at 10:48
    
The point is not the frequency of how often something has been mentioned before. Rather, it is that the code you are currently showing in your answer does not work. Based on your " probably want both elements..", it sounds like you still haven't tested it. –  Peter.O Jun 21 '12 at 13:31
    
@Peter.O I'm aware. It's been fixed. –  ormaaj Jun 23 '12 at 14:52

Do-done; done - you need two fors and two dones:

for i in $(< file1); do for j in $(< file2); do echo $i $j;done ; done

File2 is, of course, processed for each word in file1 once.

Using < instead of cat should be an unremarkable performance gain in most cases. No subprocess involved. (Unsure about last sentence).

Uncompressed:

for i in $(< file1)
do
  for j in $(< file2)
  do
    echo $i $j
  done
done

If you don't like to read words, but lines, and preserve the whitespace, use while and read:

while read line; do while read second; do echo "${line}${second}"; done <file2 ; done <file1

If you want to append 2 files, and not nest them:

cat file1 file2 | while read line; do echo "${line}"; done

If you want to zip 2 files, use paste:

paste file1 file2
share|improve this answer
1  
You say there is no subprocess involed. Isn't the () a subshell? I don't want to be pendantic but I'm unsure what is happening now. For instance I know that when I have a script and I don't want it to polute the current shell with cd's for instance, I do it inside a (). Also running (sleep 4)& for example followed by a ps shows me the process is still running. Can you please elaborate. –  Silverrocker Jun 20 '12 at 21:55
    
Well - I'm sorry, I'm very much acting without fundament here. I thought I heared so, but maybe it is only the < compared to cat. And I'm unsure how to test it, and when it will become semantically important. If pipes are invoked, variables in subprocesses don't bubble up to the main process, but we don't have variables here. I strike through the critical sentence, until somebody can clarify it. –  user unknown Jun 20 '12 at 22:03

while has an interesitng syntax. You can put multiple commands before the do ... while loop, and the case in question may need to juggle this feature, depending on your particular requirements of: do you read to end of the longest file, or only to the end of the shortest.

For example, read || read simply does not work (as per the question's requirements), becase when a read of the first file is true, the second file's read is skipped until the first file has been read from start to finish... Then, because the status is still true, the while loop continues and reads the second file from start to finish.

read && read will read files concurrently (synchronously) if you only want to read as far as the shortest file. However, if you want to read both files to eof, then you need to work with while's syntax requirements, ie. by the command immediately before the do while loop producing a non-zero return code to break out of the while loop.

Here is an example of how to read both files to eof

while IFS= read -r line3 <&3 || ((eof3=1))
      IFS= read -r line4 <&4 || ((eof4=1))
      !((eof3 & eof4))
do
    echo "$line3, $line4"
done 3<file3 4<file4

(you may want to test eof3 and eof4 before the read, but the general idea is there, especially in the final true/false condition.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.