I often find myself trying to do trivial text manipulations on quite large files. It seems like there ought to be a way to script this with one or other of the various Unix text mangling tools, but I can't figure out how exactly.

As a concrete example, suppose I have some source code that looks like this:

foo1 = undefined
foo2 = undefined
foo3 = foobar 7
foo4 = undefined

And I want to transform it to look like this:

foo1 = error "foo1"
foo2 = error "foo2"
foo3 = foobar 7
foo4 = error "foo4"

It looks like there ought to be some way in hell to do this transformation automatically. Obviously I can easily write a program in a real programming language. But surely there's some command-line tool that can do this. (?)

As a more complex example, how do I turn



Magic(ABC, DEF, GHI);
Magic(123, 456, 789);

More generally, which tool should I be looking at to do these sorts of transformations? Is that sed, or awk, or...?

  • What are the criteria for grouping Magic? Alpha vs, Numeric? Groupings of foo, bar, baz? Both? In groups of 3? Does the sequence matter? Are there likely to be non-Magic lines? ... really it dosn't matter that much if you are primarily interested in what app to use.. The answer to that is awk. sed is very good for simple text manipulation, but awk is the one when you need to do some serious juggling. – Peter.O Aug 6 '15 at 22:43
  • @Peter.O All good questions. It's one particular example of something I needed to do. The lines will always be foo followed by bar followed by baz, possibly with intervening lines that I don't care about. foo should always become the first argument to Magic(), and so on. – MathematicalOrchid Aug 7 '15 at 8:15

It depends on the particular situation. Your first example could resolved using either sed or awk. For example, using awk:

$ awk '
/undefined/ {printf "%s = error \"%s\"\n", $1, $1; next}
' input

Which produces:

foo1 = error "foo1"
foo2 = error "foo2"
foo3 = foobar 7
foo4 = error "foo4"

Or using sed:

sed '
  /undefined/ s/\([^ ]*\) =.*/\1 = error "\1"/
' input

Your second example would probably be easier to do with awk, or some higher level language like Perl or Python. Again, with awk:

awk '
    split($0, parts, "=")
    items[i++] = parts[2]
i%3 == 0 {
    printf "Magic(%s, %s, %s)\n", items[0], items[1], items[2]

' input

Which produces:

Magic(ABC, DEF, GHI)
Magic(123, 456, 789)

More generally...I'm not sure it's possible to provide a general answer. It really depends on the specific task you're trying to accomplish. Once you've pulled out awk you're already writing a program in a real programming language, so you shouldn't eschew a higher level tool (like Perl or Python or Ruby or whatever) just because it's more capable.

| improve this answer | |
  • Your first awk example is outputting the original undefined line after it prints the modified version. – Peter.O Aug 6 '15 at 23:39
  • It's not, actually ( with gawk 4.1.1). – larsks Aug 6 '15 at 23:50
  • Ah, interesting. It does with GNU Awk 3.1.6 – Peter.O Aug 7 '15 at 3:46
  • I have just tried it in GNU Awk 4.0.1 and it too outputs 7 lines (instead of 4) - here is the actual code I used: printf '%s\n' 'foo1 = undefined' 'foo2 = undefined' 'foo3 = foobar 7' 'foo4 = undefined' | awk '/undefined/ {printf "%s = error \"%s\"\n", $1, $1} {print}' ... – Peter.O Aug 7 '15 at 5:23
  • So for general "cut this text here, paste it there", awk is the puppy to use? You got any good references for learning about it? All the tutorials I could find only cover trivial 1-liners, and don't go into the syntax for larger blocks of code... – MathematicalOrchid Aug 7 '15 at 8:21

sed is great for simple text manipulation - typically single-line edits, though it can work with multi-lines (with a lot of effort). Overall sed is rather hamstrung, by the lack of variables and arithemtic calculations, but nonetheless it does offer the simplest solution in many instances.

awk is very effective for both simple and complex manipulation of text and numberic calculations, but it is not ideally suitable for things beyond that.

For the first example:

sed -E 's/^([^ ]+) = undefined$/\1 = error "\1"/' file1
awk '$3=="undefined"{ $3="error \"" $1 "\"" } {print $0}' file1

For the second example, based on =values being either all-UPPERCASE alpha, or all-NUMERIC. It also groups any non-Magic lines (to remove, just delete out[0] statements):

awk -F'=' 'BEGIN{ # split regular expressions, using `x7F` as delimiter (or any char not in the regex)  
                  n=split("^[A-Z]+$" "\x7F" "^[0-9]+$",rx,"\x7F") 
           { for( i=1;i<=n;i++ ){
                 if( $2 ~ rx[i] ){ 
                     out[i]=out[i] sprintf( (out[i] ?", " :"") "%s", $2)
             }   } 
             if( i>n ) out[0]=out[0] $0 RS # non-matching lines 
           END{ printf out[0]
                for( i=1;i<=n;i++ ){ print "Magic(" out[i] ")" }
           }' file2
| improve this answer | |

I'd use Perl, as it is a (reasonably "normal") programming language, built around powerful text manipulation (search, replace in vi(1) style with regular expressions). But many would consider me a heretic, and use Python instead. Both are distributed with any Linux distribution and have decent Windows implementations (and I'm sure for Mac too). Python has the advantage that it is use to build much of the GUI for administration tools in e.g. Fedora, so it should be alredy installed.

awk(1) and sed(1) are powerful tools, but somewhat single-minded...

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.