7

I have a huge file (more than 2 GB) where data is like below.

12,324,32342,E:fsdsf,23432,34534,45345,324

13,3224,342,E:werwefsdsf,23432,34534,45345,324

121,3244,33442,E:,23432,34534,45345,324

Here I need to replace all the occurrences of 4th column wherever it is encountering E: with a blank space "", but wherever it finds a string after E:, that should remain as it is.

The expected output would be:

12,324,32342,E:fsdsf,23432,34534,45345,324

13,3224,342,E:werwefsdsf,23432,34534,45345,324

121,3244,33442,,23432,34534,45345,324
  • I tried the below - awk -F , '{ if ($4 = "E:") $4="";}1' OFS=, data.final but this is replacing all the occurrences of $4 with a blank space. – anurag Dec 21 '15 at 6:49
8

Your syntax is almost right. To test for string equality in awk, use a double equals sign: ==. To assign a value, use a single equals sign.

So use if ($4 == "E:") and you should get the result you want.

The full command would look like this. Note there is only one character difference from the command you used; that was your only error:

awk -F , '{ if ($4 == "E:") $4="";}1' OFS=, data.final

To show some different syntax and different approaches, the following versions are exactly equivalent:

awk -F, -v OFS=, '$4 == "E:" { $4 = "" }; 1' data.final

awk 'BEGIN { FS=OFS="," }; $4 == "E:" { $4 = "" }; {print}' data.final

awk -F, -v OFS=, '{sub( /^E:$/, "", $4); print}' data.final

Notes regarding the above:

  1. If your whole code block is just an if/then, you can just use the condition as a filter for the code block. Hence $4 == "E:" {$4 = ""} is exactly equivalent to {if ($4 == "E:") {$4 = ""}}
  2. It is good practice to enclose "then" statements in a code block even when there is only one of them, i.e. if ($4 == "E:") {$4 = ""} rather than if ($4 == "E:") $4 = "";
  3. -F sets the value of FS and -v can be used to set the value of any variable before awk considers the first line of the first file. (You likely knew that.) You can also use a BEGIN block to do the same thing; worth knowing for when you want to make the awk script stand alone.
  4. The reason why 1 prints lines in awk is that it is a condition (a filter) which always evaluates to true, and the default action in awk when no code block is attached to the filter is to print $0. So 1 by itself is equivalent to 1 {print} or 1 {print $0} or just {print}.
  5. In my last variation, I used a sub function to substitute the regex /^E:$/ (start of string, E:, end of string) with "" in $4.

Since the sub function returns the number of substitutions that were made (either 1 or 0; use gsub to make more than one replacement), you can code golf this problem by adding one to the sub function result to ensure you have a pattern that is always true so that the resulting line is printed whether a substitution is made or not. Here is the code golf version, not recommended for beginner use if you are putting this in a script you will then maintain:

awk -F, -v OFS=, 'sub(/^E:$/,"",$4)+1' data.final

:)

2

With sed:

sed -r 's/^([^,]+,[^,]+,[^ ]+,)E:(,)/\1\2/' file.txt

The 4th comma separated field will be made blank if it contains only E:.

Example:

% cat file.txt
12,324,32342,E:fsdsf,23432,34534,45345,324
13,3224,342,E:werwefsdsf,23432,34534,45345,324
121,3244,33442,E:,23432,34534,45345,324

% sed -r 's/^([^,]+,[^,]+,[^ ]+,)E:(,)/\1\2/' file.txt 
12,324,32342,E:fsdsf,23432,34534,45345,324
13,3224,342,E:werwefsdsf,23432,34534,45345,324
121,3244,33442,,23432,34534,45345,324
  • stream editor +1 – Pandya Dec 21 '15 at 9:04
0

Assuming your file name is file, you can try the following:

while read -r line; 
do 
var="$(echo "$line" | cut -d ',' -f 4)";

  if [[ "$var" = "E:" ]]; then echo "$line" | sed s/"$var"/''/g ; 
  else echo "$line";
  fi; 

done < file

or:

while read -r line; do var="$(echo "$line" | cut -d ',' -f 4)"; if [[ "$var" = "E:" ]]; then echo "$line" | sed s/"$var"/''/g ; else echo "$line";fi; done < file

Explanation:

  1. while read -r line; reads file line by line
  2. var="$(echo "$line" | cut -d ',' -f 4)"; sores the string at 4th place separated by , into variable var
  3. if [[ && "$var" = "E:" ]]; then echo "$line" | sed s/"$var"/' '/g ; if $var has string exactly E: then sed s/"$var"/''/g ; replaces it with blank ""
  4. else echo "$line"; other wise it prints line as it is

Example output (as expected from question):

  • file:

    $ cat file
    12,324,32342,E:fsdsf,23432,34534,45345,324
    
    13,3224,342,E:werwefsdsf,23432,34534,45345,324
    
    121,3244,33442,E:,23432,34534,45345,324
    
  • Running command:

    $ while read -r line; do var="$(echo "$line" | cut -d ',' -f 4)"; if [[ "$var" = "E:" ]]; then echo "$line" |sed s/"$var"/' '/g ; else echo "$line";fi; done < file
    12,324,32342,E:fsdsf,23432,34534,45345,324
    
    13,3224,342,E:werwefsdsf,23432,34534,45345,324
    
    121,3244,33442,,23432,34534,45345,324
    

You can also redirect your output to file by using >> file2 or |tee file2 at the last of command:

while read -r line; do var="$(echo "$line" | cut -d ',' -f 4)"; if [[ "$var" = "E:" ]]; then echo "$line" |sed s/"$var"/' '/g ; else echo "$line";fi; done < file | tee file2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.