3

I have a text file which contains several vectors, as follows. The components of these vectors are separated with space and they are spanned in a few lines. This file is generated after I run a command in the Ubuntu terminal.

0 -1 -0.494 0.12 -0.919 0.112 0.914 -0.681 -0.067 -0.918 -0.443 -0.216 -0.48 0.55 0.701 0.429 0.699 -0.726 -0.39 0.172 0.61 -0.599 0.728 -0.883 -0.32 0.044 -0.189 -0.732 -0.309 -0.286 -0.859 0.107 0.298 
0 0 0.869 0.641 -0.331 -0.631 -0.236 0.303 0.998 0.153 -0.89 -0.927 -0.671 -0.478 0.693 -0.007 -0.64 0.091 -0.249 -0.881 0.641 0.689 0.222 -0.398 0.548 -0.268 -0.877 -0.333 -0.55 0.858 0.504 0.215 -0.178 
0 0 0 0.758 -0.214 0.768 0.329 0.667 -0.013 0.367 0.103 -0.307 -0.565 0.685 0.171 -0.903 0.32 -0.682 -0.887 -0.44 -0.467 0.409 -0.649 0.249 0.772 -0.962 0.443 -0.594 0.776 -0.427 0.088 -0.971 0.938 

How can I convert this file into another file using shell commands with the following format, where each vector is in a separate line and the header of the file is the number of three-component vectors?

n
V1x V1y V1z
V2x V2y V2z
V3x V3y V3z
...
Vnx Vny Vnz

where n is the number of three-component vectors in the file. In the file that I have: V1x=0, V1y=-1, V1z=-0.494. V2x=0.12, V2y=-0.919, V2z=0.112 and so forth.

3

A Perl one-liner:

perl -p00E 'y/\n/ /;say s/(\S+\s+){3}\K/\n/g' file

Note that both this solution and Gnouc's assume the file is small enough to be stored in memory as a whole.

Explanation

  • -p means alias each record of the file to $_ and print the contents of $_ after each record has been processed.
  • -00 means set the record separator to null in order to read the whole file in as one record.
  • -E means treat the following string as Perl code. Using -E instead of the usual -e means I can use the say feature.
  • y/\n/ / makes the whole file one line (note that y/// is a synonym of tr/// in Perl in order to oblige sed users).
  • s/(\S+\s+){3}\K/\n/g means append a new line after every pattern of (non-whitespace followed by whitespace repeated 3 times == a vector).
  • Since s/// will return the number of substitutions it has successfully made, using it as an argument to say will print out the number of substitutions (=vectors).
  • After the count is printed, the contents of $_ are then printed because we used -p.

Update

If you want the maximum value:

perl -p00E 'y/\n/ /;s/(\S+\s+){1}\K/\n/g' file | sort -nr | head -1

The advantage of this solution

It only has one "magic number". In other words, if you suddenly started working with 2D vectors, all you need to do is change the {3} in the code to {2}.

The disadvantage of this solution

If you're unfamiliar with Perl, it reads like a black magic spell.

  • Thank you. It is a great solution. I started learning Perl to fully understand the answers for this post. – AFP May 18 '14 at 22:14
  • @Ahm I'm glad I could help. Please don't forget to mark the answer 'Accepted' if it helped you. As an additional piece of advice: if your work involves manipulating many textual files (like in this example), you would benefit a lot from learning a scripting language like Perl, Python, Ruby or Tcl. – Joseph R. May 18 '14 at 22:26
  • How can I also skip printing the number of vectors? – AFP May 19 '14 at 5:25
  • 1
    @A2009 just remove say. – Joseph R. May 19 '14 at 5:26
  • Thank you for your quick help. I modified your super flexible code in order to find the maximum of the values. So I changed it to perl -p00E 'y/\n/ /; s/(\S+\s+){1}\K/\n/g' file to wrap the each number in a separate line. Now I want to find the maximum value. Could you help me with this too? – AFP May 19 '14 at 5:37
2

something like

ruby -e 'ns = STDIN.read.split(/\s+/); puts(ns.size/3); 0.step(ns.size,3) do |i| puts(ns[i,3].join(" ")) end' < yourfile

should work, if you allow external programs to be called from the shell.

Edit: Maybe we should take this on the golf course :-)

  • 1
    "Maybe we should take this on the golf course" Challenge gladly accepted. Please see my updated answer :) – Joseph R. May 18 '14 at 19:24
2

So you want to do two things:

  • re-wrap the data to have exactly 3 coordinates per line;
  • prefix a line with the number of vectors.

It is simpler to handle this as two successive, independent problems. First, rewrap the data. You can use awk for this, telling it that any sequence of whitespace is an input record separator.

awk -v RS='[[:space:]]+' '{if (NR % 3) printf "%s ", $0; else print}' <input.txt >wrapped.txt

You can make this a little shorter by setting the output separator to a newline on lines numbers that are a multiple of 3, and a space otherwise.

awk -v RS='[[:space:]]+' '{ORS = NR % 3 ? " " : "\n"; print}' <input.txt >wrapped.txt

Since printing is the default action, this can be shortened to

awk -v RS='[[:space:]]+' 'ORS = NR % 3 ? " " : "\n"' <input.txt >wrapped.txt

The number of vectors is the number of lines in the intermediate files.

wc -l wrapped.txt >output.txt
cat wrapped.txt >>output.txt
1

A perl solution:

$ perl -anle 'push @e,@F; 
END {
    print @e/3;
    for ($i=0;$i<@e;$i+=3) {
        printf "%-6s %-6s %-6s\n",$e[$i],$e[$i+1],$e[$i+2];
    }
} ' file
33
0      -1     -0.494
0.12   -0.919 0.112 
0.914  -0.681 -0.067
-0.918 -0.443 -0.216
-0.48  0.55   0.701 
0.429  0.699  -0.726
-0.39  0.172  0.61  
-0.599 0.728  -0.883
-0.32  0.044  -0.189
-0.732 -0.309 -0.286
-0.859 0.107  0.298 
0      0      0.869 
0.641  -0.331 -0.631
-0.236 0.303  0.998 
0.153  -0.89  -0.927
-0.671 -0.478 0.693 
-0.007 -0.64  0.091 
-0.249 -0.881 0.641 
0.689  0.222  -0.398
0.548  -0.268 -0.877
-0.333 -0.55  0.858 
0.504  0.215  -0.178
0      0      0     
0.758  -0.214 0.768 
0.329  0.667  -0.013
0.367  0.103  -0.307
-0.565 0.685  0.171 
-0.903 0.32   -0.682
-0.887 -0.44  -0.467
0.409  -0.649 0.249 
0.772  -0.962 0.443 
-0.594 0.776  -0.427
0.088  -0.971 0.938
1

There's an interesting feature of the bash shell's printf builtin

  The format is reused as necessary to consume all  of  the  argu‐
  ments.

which appears to allow us to slurp up a file of whitespace-separated values and spit them out three-to-a-line using a simple printf

printf '%8.3f %8.3f %8.3f\n' $(<file)

(I've used 8.3 floating-point format just to prettify the output, but you could use %s to echo each field as a raw string).

To count the resulting vectors you could just use wc - if you don't mind the count coming after the data then you could just tee the output

printf '%8.3f %8.3f %8.3f\n' $(<file) | tee >(wc -l)

If you really insist on putting the count at the top, then one possibility might be to print to a variable, then count and print the variable (this will be subject to the same memory considerations as other in-place methods)

printf -v vecs '%8.3f %8.3f %8.3f\n' $(<file)
wc -l < <(printf "$vecs") ; printf "$vecs"

If you're a real shell purist then you could use mapfile (or its synonym readarray) to shove the re-formatted data into an array instead of a string variable - and then use the shell's ${#array[@]} count operator to avoid an external call to wc

mapfile vecs < <(printf '%8.3f %8.3f %8.3f\n' $(<file))
printf '%d\n' ${#vecs[@]} ; printf '%s' "${vecs[@]}"

The final printf makes use of the format reuse feature again to print each newline-terminated array element in turn.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.