I have a data set like below:

\"XXX \ START sapiodj \\" aj \d 2387 END hddo\" START bbcc  \\" END ss

My Requirement: I want to remove all occurrences of backslash\ and double quotes " between START and END.

Desired Output:

\"XXX \ START sapiodj  aj d 2387 END hddo\" START bbcc   END ss


  1. Multiple START/END in same line
  2. I want to remove \ and " only between START and END and nowhere else
  3. And my file has multiple line (lines similar to what is shown above)
  4. I have to use sed only

I tried something like below (was trying to get rid of " alone first) and it didn't give me the desired result:

sed '/START/,/END/ s/"//g'
  • I removed the asterisks at the beginning and end of your examples as I assumed they were intended for mark-up purposes. If I was wrong, please roll back to your earlier version.
    – Joseph R.
    Dec 18 '14 at 9:13
  • Thx Joseph ! I am new to this forum, All that you are changing is exactly what I would have done too. Thx again Dec 18 '14 at 9:17
  • 1
    You got it. Welcome to U&L :)
    – Joseph R.
    Dec 18 '14 at 9:19
  • 1
    awk can be used. Some unix commands are not available in hdfs, I was not sure if awk was one of them. I just confirmed that both sed and awk can be used ! Dec 18 '14 at 9:36
  • 1
    @mikeserv One's man "easy" is another man's hieroglyphs, I guess :D
    – Joseph R.
    Dec 19 '14 at 6:12

Assuming you don't have ` character in the file. If you do just change in the line bellow all ` to any other character that for sure will not be present in the input.

sed -e 's/END/`/g;:X' -e 's/\(START[^`]*\)["\]/\1/g;tX' -e 's/`/END/g'
  • You are relying on a specific character not being present. That is clever and pragmatic but dangerous - assumptions have a tendency of breaking things when you are not around.
    – Floris
    Dec 18 '14 at 12:39
  • @mikeserv Thanks for suggestion, you are right, I made the answer shorter with ["\] and by removing other unnecessary stuff. However using \n as a delimiter sounds too complicated for me, I've upvoted your answer though.
    – jimmij
    Dec 19 '14 at 10:56
  • it may be more simply explained here where the delimiters are already unique and single characters. I'm pretty sure the {} example in that answer is a fluke though - haven't revisited it lately. Anyway, I don't use it so much as a delimiter here as I do a spare character for offloading the actual delimiter. And mostly because [^\n]* is not portable.
    – mikeserv
    Dec 19 '14 at 11:01

It's not too hard with sed really. You can always delimit a section with a \newline or you can trade out a delimiter for a \newline temporarily. And you can do it without a loop:

sed 's/$/START/;s/END/&
/g;  y/D\n/\nD/
' <<\IN                                           
\"XXX \ START sapiodj \\" aj \d 2387 END hddo\" START bbcc  \\" END ss

Sometimes you've just got to think about a problem a little differently. Instead of removing all of the \\" between START and END if we instead switch the problem around to how we might save \\" only if they occur between the head of the line and START, START and END strings, and the last END and the tail of the line it gets a little easier (if, admittedly, not intuitively so). This is because of the way sed handles *zero-or-more matches in a global s///ubstitution context.

While the head-to-first-START bit will wash-out as a natural result of the rest of this, the last-END-to-tail bit does not - and so we need to append another START to the end of the line. After getting our extra START we then append a \newline character to every occurrence of END. And then with the y/// transliterate command we simultaneously trade all D chars for \newlines and vice versa. The y/// transliteration command, by the way, is not only very handy here, but is also more efficient than a s///ubstitution would be.

At this point a look at our pattern space would print:

\\"XXX \\ START sapiodj \\\\" aj \\d 2387 EN\nD hddo\\" START bbcc  \\\\" EN\nD ssSTART$

As you can see, now all \\" characters that need saving lie squarely between either the head of the line or a D and START strings and there are no Ds between them. So the global s///ubstitution that removes the unwanted chars - to include our extra Ds - also replaces the ones that need saving with themselves. Last we need only to swap \n and Ds again and remove the last START.

In this way you can reliably delimit fields with sed no matter the input and you need not rely on any not-occurring character but the one that is guaranteed never to occur on a line - and that is the \newline character of course.

When it has finished that prints:

\"XXX \ START sapiodj  aj d 2387 END hddo\" START bbcc   END ss
  • 1
    Clever to use the newline "delimiter"!
    – Floris
    Dec 19 '14 at 11:53

With sed:

sed 's/:/::/g;s/</:l/g;s/>/:g/g; # escape :, <, >
     s/START/&</g; s/END/>&/g;   # replace START/END with <>
     s/[<>]//g;s/:g/>/g;s/:l/</g;s/::/:/g; # restore <>:'

With perl:

perl -pe's|START.*?END|$&=~y/\\"//rd|ge'
  • you might drop the test loop if you s/\(<[^>]*>\)*[\"]*/\1/g. The backref'd sequence won't squeeze in a global context - but you would probably have to handle the line's head/tail in that case.
    – mikeserv
    Dec 19 '14 at 5:31

You have indicated in a comment that awk is also allowed. So I'm basing my answer on this.

Assuming your STARTs and ENDs are balanced, if you split the line on either word, you find that you want to remove backslashes and double quotes from all even-numbered fields. To this end:

awk -F 'START|END' '{
                      for(i=2;i<=NF;i+=2){ # For each even-numbered field
                        gsub(/["\\]/,"",$i) # Remove " and \ from it
                        $i="START"$i"END" # Put START and END back around it
                    }' your_file

This assumes your implementation of awk has the gsub function which I can't vouch for.

As a side note, your sed doesn't work because it's basically saying "apply the substitution to the range of lines that starts with a line matching START and ends with a line matching END".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.