Shell Script to remove double quotes within the column value

Question

I have a input text file with 10 columns, while processing this file, in one of the middle column, I am getting this type of data.I required column value to be as below:

Input Column value: "This is my new program:"Hello World""

Required Column value: "This is my new program: Hello World".

Please help me in any Unix shell scripts or any commands. Really appreciate your time and thanks in advance.

Answer 1

A very simple option is to use sed as @Dani proposes if you want to remove all double-quotes.

$ echo "This is my program \"Hello World\"" | sed 's/"//g'

This is my program Hello World

Nevertheless, if you want to remove only internal quotes, I would suggest removing all quotes and adding one at the beginning and one at the end as follows.

Let's say we have a file sample.txt with these contents:

$ cat sample.txt

"This is the "First" Line"
"This is the "Second" Line"
"This is the "Third" Line"

Then, if you want to remove only internal quotes, I would suggest the following:

$ cat sample.txt | sed 's/"//g' | sed 's/^/"/' |sed 's/$/"/'

"This is the First Line"
"This is the Second Line"
"This is the Third Line"

Explanation:

sed 's/"//g' removes all double quotes on each line

sed 's/^/"/' adds a double-quote at the beginning of each line

sed 's/$/"/' adds a double-quote at the end of each line

sed 's/|/"|"/g' adds a quote before and after each pipe.

Hope this helps.

EDIT: As per the pipe separator comment, we have to slightly change the command

Let sample.txt be:

$ cat sample.txt

"This is the "First" column"|"This is the "Second" column"|"This is the "Third" column"

Then, adding a replacer command for the pipe give us the final solution.

$ cat sample.txt | sed 's/"//g' | sed 's/^/"/' |sed 's/$/"/' | sed 's/|/"|"/g'

"This is the First column"|"This is the Second column"|"This is the Third column"

The script option

Using this sample.txt file

$ cat sample.txt
"This is the "first" column"|12345|"This is the "second" column"|67890|"This is the "third" column"

And this script

#!/bin/ksh

counter=1
column="initialized"
result=""
while [[ "$column" != "" ]]
do
    eval "column=$(cat sample.txt | cut -d"|" -f$counter)"
    eval "text=$(cat sample.txt | cut -d"|" -f$counter | grep '"')"
    if [[ "$column" = "$text" && -n "$column" ]]
    then
        if [[ "$result" = "" ]]
        then
            result="_2quotehere_${column}_2quotehere_"
        else
            result="${result}|_2quotehere_${column}_2quotehere_"
        fi
    else
        if [[ -n "$column" ]]
        then
            if [[ "$result" = "" ]]
            then
                result="${column}"
            else
                result="${result}|${column}"
            fi
        fi
    fi
    echo $result | sed 's/_2quotehere_/"/g' > output.txt
    (( counter+=1 ))
done
cat output.txt
exit 0

You will get this:

$ ./process.sh
"This is the first column"|12345|"This is the second column"|67890|"This is the third column"

$ cat output.txt
"This is the first column"|12345|"This is the second column"|67890|"This is the third column"

I hope this is the processing you need.

Let me know!

FINAL EDIT

This script processes the input line you provided, several times included. Only restriction is that all 20 columns MUST BE on the same line.

#!/bin/ksh

rm output.txt > /dev/null 2>&1
column="initialized"
result=""
lineCounter=1
while read line
do
    print "LINE $lineCounter: $line"
    counter=1
    while [[ ${counter} -le 20 ]]
    do
        eval 'column=$(print ${line} | cut -d"|" -f$counter)'
        eval 'text=$(print ${line} | cut -d"|" -f$counter | grep \")'
        print "LINE ${lineCounter} COLUMN ${counter}: $column"
        if [[ "$column" = "$text" && -n ${column} ]]
        then
            if [[ "$result" = "" ]]
            then
                result="_2quotehere_$(echo ${column} | sed 's/\"//g')_2quotehere_"
            else
                result="${result}|_2quotehere_$( echo ${column} | sed 's/\"//g')_2quotehere_"
            fi
        else
            if [[ "$result" = "" ]]
            then
                result=${column}
            else
                result="${result}|${column}"
            fi
        fi
        (( counter+=1 ))
    done
    (( lineCounter+=1 ))
    echo -e $result | sed 's/_2quotehere_/"/g' >> output.txt
    result=""
done < input.txt
print "OUTPUT CONTENTS:"
cat output.txt

exit 0

From here, you must be able to get it working for your particular case.

Answer 2

The simplest criteria to edit a field is "if it has letters".
Fields with only numbers (and some symbols .,- etc.) should be left alone.
This simple awk script does the job:

#!/bin/bash

awk -v FS='|' -v OFS='|' '{
for ( i=1; i<=NF; i++) {
    if ( $i ~ /[a-zA-Z]/ ) {
        gsub(/["]/,"",$i); $i="\"" $i "\""    # Remove dquotes, add them back.
    }
} }1' input.txt >output.txt