1

I have file like this:

Dir1/File1.cpp Dir2/File2.cpp \
Dir3/File1.h Dir4/File2.cpp \
Dir2/File1.cpp \
Dir2/File1.h \

I want to generate file like this:

Dir1/File1.cpp
Dir2/File2.cpp
Dir3/File1.h
Dir4/File2.cpp
Dir2/File1.cpp
Dir2/File1.h

How to do this by using Bash/Sed/Awk/Grep or something like this?

  • Dir1/File1.cpp Dir2/File2.cpp \[...]Dir2/File1.h what is? the content of the file? – zuazo Oct 3 '16 at 14:42
  • Yes, it is content. – marcin Oct 3 '16 at 14:47
2

If you have an Awk that supports regular expressions for the record separator RS, it can be done like this:

awk 'BEGIN { RS = " +| *\\\\?\\n" } 1'

The advantage of this is that we are not snarfing the entire file into memory and doing some regex replacement; your input could be gigabytes long.

We basically treat the file as having two record separators: either one or more spaces, or else zero or more spaces followed by a newline, which may be preceded by an optional backslash.

Having delimited the records this way, all we have to do is output them followed by the default output record separator (ORS), which, of course, is newline. That is achieved by a pattern-action rule consisting of 1.

Or a pipeline job with sed and tr, not using anything that isn't in POSIX:

tr '\n' ' ' | sed -e 's/\\//g' -e 's/ \+/ /g' | tr ' ' '\n'

Replace newlines with spaces. Then squash runs of multiple spaces to one space while removing backslashes. Then map spaces to newlines.

  • \+ is not in POSIX. The POSIX equivalent is \{1,\} (or use s/ */ /g (<SPC><SPC>*)) – Stéphane Chazelas Oct 3 '16 at 15:40
  • +1, the awk solution is much faster than grep one.. and the mash of tr/sed was faster than awk by a small margin for sample I used (each input line of OP's example repeated 10000 times) – Sundeep Oct 3 '16 at 15:41
  • Why not simply tr '\\ ' '[\n*]' | grep . – Stéphane Chazelas Oct 3 '16 at 15:46
  • tr '\\ ' '[\n*]' | grep . is more than twice as faster than awk solution (each input line of OP's example repeated 10000 times) – Sundeep Oct 3 '16 at 15:47
  • 1
    Probably better with tr -s '\\ ' '[\n*]' | grep . – Stéphane Chazelas Oct 3 '16 at 16:09
2

With GNU grep

$ cat file 
Dir1/File1.cpp Dir2/File2.cpp \
Dir3/File1.h Dir4/File2.cpp \
Dir2/File1.cpp \
Dir2/File1.h \

$ grep -o '[^\ ]*' file 
Dir1/File1.cpp
Dir2/File2.cpp
Dir3/File1.h
Dir4/File2.cpp
Dir2/File1.cpp
Dir2/File1.h
  • -o extract only matching pattern
  • [^\ ]* zero or more of non-space and non \ characters, since * is greedy, it will try to match as many such characters as possible

to save the result to another file, use

$ grep -o '[^\ ]*' file > out_file


As pointed out by @Stéphane Chazelas, it is better to use the following for being more portable:

grep -oE '[^\ ]+' file

where -E invokes extended regex and [^\ ]+ matches one or more of non-space and non \ characters


Performance analysis:

$ perl -ne 'print "$_"x100000' file > file_big
$ shuf file_big -o file_big 

$ du -sh file_big 
9.0M    file_big

All answers and suggestions from comments used for comparison:

$ time grep -o '[^\ ]*' file_big > o1

real    0m2.090s
user    0m2.076s
sys 0m0.016s

$ time grep -oE '[^\ ]+' file_big > o2

real    0m1.523s
user    0m1.504s
sys 0m0.012s

$ time awk 'BEGIN { RS = " +| *\\\\?\\n" } 1' file_big > o3

real    0m0.331s
user    0m0.320s
sys 0m0.008s

$ time tr -s '\\ ' '[\n*]' < file_big | grep . > o4

real    0m0.095s
user    0m0.124s
sys 0m0.008s

$ time tr '\\ ' '[\n*]' < file_big | grep . > o5

real    0m0.105s
user    0m0.104s
sys 0m0.016s

Sanity check

$ diff -s o1 o2
Files o1 and o2 are identical
$ diff -s o1 o3
Files o1 and o3 are identical
$ diff -s o1 o4
Files o1 and o4 are identical
$ diff -s o1 o5
Files o1 and o5 are identical
  • 1
    -o is a GNU extension. It's now also supported by a few other implementations, but in some like ast-open's grep, if the pattern matches the empty string, it will run into an infinite loop. You'd want grep -Eo '[^\ ]+' here instead. – Stéphane Chazelas Oct 3 '16 at 15:44
  • @StéphaneChazelas, doesn't * being greedy ensure it doesn't match empty string? – Sundeep Oct 3 '16 at 15:52
  • 1
    on an input like Dir2/File1.h \​, [^\ ]* finds 3 matches: <Dir2/File1.h> <>\<> (try with sed 's/[\ ]*/<&>/g'. GNU grep -o doesn't print the empty matches. ast-open does and gets stuck because (I suppose) it doesn't advance to the next character to carry on finding other matches. – Stéphane Chazelas Oct 3 '16 at 16:06
  • is this specific to -o and * for ast-open or does it suffer for even regex like grep 'c*' ? either way, I will add that my answer is tested with GNU grep.. – Sundeep Oct 3 '16 at 16:13
  • 1
    AFAIK, ast-open's grep -o has a problem with any pattern that matches the empty string (including c*, ^, ''). In any case, -o being a non-standard option, there's no telling what other implementations might do with it, so yes, it's a good idea to mention it would work with GNU grep and clones (like FreeBSD's which used to be GNU grep and was then rewritten). IMO though, the grep -Eo '[^\ ]+' version looks more correct to me. (and is portable to ast-open's grep). – Stéphane Chazelas Oct 3 '16 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.