The given file must not be stored in variable and then traversed due to memory size restrictions:


var=$(cat FILE)
for i in $var
  echo $i

How do you traverse all strings in a file in the same way as the example above but extract each whitespace-separated directly from the file?


totalfilecount=$(cat FILE | wc -w)
while (( ${fileindex} <= ${totalfilecount} ))
  onefilename= ??? missing command using fileindex

Is there a command that can view a file as an array and allow you to extract words using their index positions?

The idea is to process every word in the file as though the file were an array.

Input file example:

one two
three four
five six

Here is the scenario that requires the above funtionality:

  • we have server_A and server_B
  • server_A needs to connect to server_B via sftp (sftp only) and 'get' some files
  • BOTH 'ls' or 'ls -l' commands in sftp can be using wild cards to filter specific files
  • each file needs to be processed individually (for various reasons) on the fly
  • the files cannot be copied as a group to server_B and then processed individually
  • a list of files must first be created on server_A and then each file in that list is copied from server_B and processed one file at a time

Where is the problem?

The problem is how the 'ls' command can create a dual column list of words if the list is long thus not allowing simple processing as with 'ls -l' which always creates a single column list.

This leads us to my initial question, if such a solution exists.

  • @h3rrmiller strings won't do what is being asked here. – Chris Down Nov 16 '12 at 15:46
  • @h3rrmiller How does using strings allow extraction of individual strings as stated in the question? – Chris Down Nov 16 '12 at 16:25
  • @h3rrmiller You may wish to read the comments. – Chris Down Nov 16 '12 at 16:30
  • @ChrisDown you deleted your comment about the new lines... – h3rrmiller Nov 16 '12 at 16:35

You can do this per word using awk, which should meet your memory requirements:

awk -v RS=\  '{
    # Do something with the word
}' file

You can specify which string you want by using NR.

$ awk -v RS=\  'NR==2{print}' <<< 'foo bar baz'
  • The idea is to process every string in the file as though the file were an array, without processing lines. – Radnaskela Samot Nov 16 '12 at 13:06
  • @AleksandarHadjikan Okay, edited. – Chris Down Nov 16 '12 at 13:28
  • This is very good. Is there a way to send an index to awk to specify which word to print out? – Radnaskela Samot Nov 16 '12 at 13:57
  • @AleksandarHadjikan Added it to my answer. – Chris Down Nov 16 '12 at 14:11
  • 1
    @AleksandarHadjikan It really sounds like you are going about this the complete wrong way. It would be much better not to do that. – Chris Down Nov 16 '12 at 16:22

When you say “strings” you mean “words”, right? Strings of characters separated by whitespace. And according to your examples, you want to access them sequentially.

You can do:

$ sed 's/[ \t]\+/\n/g' YOUR_FILE | while read -r word ; do PROCESS $word ; done

Example of use:

% echo word1 word2 > YOUR_FILE
% echo word3 word4 >> YOUR_FILE
% echo word5 word6 >> YOUR_FILE
% sed 's/[ \t]\+/\n/g' YOUR_FILE | while read -r word ; do echo _${word}_ ; done
  • Yes string=word. I tried your example however it did not work. I echo $word and it prints the entire file. – Radnaskela Samot Nov 16 '12 at 13:42
  • Works for me. See example. In what way do the contents of your file differ from lines of words separated by spaces? – angus Nov 16 '12 at 13:47
  • Will break if the file contains backslashes. – Chris Down Nov 16 '12 at 13:52
  • I create three seperate lines with two words in each line. – Radnaskela Samot Nov 16 '12 at 13:54
  • @ChrisDown Yes, apparently read is the culprit. Fixed. – angus Nov 16 '12 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.