9

I have data like this:

chr1    134901  139379  -   "ENSG00000237683.5";
chr1    860260  879955  +   "ENSG00000187634.6";
chr1    861264  866445  -   "ENSG00000268179.1";
chr1    879584  894689  -   "ENSG00000188976.6";
chr1    895967  901095  +   "ENSG00000187961.9";

I generated by parsing a GTF file

I want to remove the "'s and ;'s from column 5 using awk or sed if it possible. The result would look like this:

chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9
2
  • 1
    you can also use multiple seach and replace statements in sed. sed 's/"//g; s/;//g' filename
    – jbrahy
    Jan 14, 2016 at 19:55
  • @DigitalTrauma ya, but Dani_l already gave that solution.
    – jbrahy
    Jan 14, 2016 at 23:59

7 Answers 7

11

Using gsub:

awk '{gsub(/\"|\;/,"")}1' file
chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9

If you want to operate only on the fifth field and preserve any quotes or semicolons in other fields:

awk '{gsub(/\"|\;/,"",$5)}1' file 
7
  • 1
    This would remove from all columns, not just 5th, no?
    – Dani_l
    Jan 14, 2016 at 19:55
  • This is what I thought initally, but after using the code it seemed to keep all columns.
    – System
    Jan 14, 2016 at 19:57
  • @Dani_l Yes, it can be refined to operate only on the fifth field, but that was not a requirement...
    – jasonwryan
    Jan 14, 2016 at 19:57
  • Sorry I must have not made it clear, I DO want to keep all columns. This is why it is marked as the answer.
    – System
    Jan 14, 2016 at 19:58
  • 2
    Why not just use a character class? /[;"]/ is a lot more readable and simpler in my opinion than /\"|\;/.
    – Wildcard
    Jan 17, 2016 at 7:46
6

If your data is formatted exactly as shown (i.e. no other " or ; in other columns that need to be preserved), then you can simply use tr to remove these characters:

tr -d '";' < input.txt > output.txt
5

Using sed to remove all instances of '";': sed -i 's/[";]//g' file

To only remove from 5th column sed is probably not the best option.

3

I know the original post asked for sed or awk but if you want to remove the " and ; from only the fifth column I'd use regex and php. There's probably a way to do this in AWK but I like to use the easiest tools.

<?php

foreach(file($argv[1]) as $line){

    $matches = array();
    preg_match('/^(\w+)\s+(\d+)\s+(\d+)\s+(\-|\+)\s+"(\w+.\d)"\;/',$line,$matches);
    $matched_line = array_shift($matches); // remove the first element
    vprintf("%s\t%s\t%s\t%s\t%s\n",$matches);
}

this would output this

$ php /tmp/preg_replace.php /tmp/data
chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9
5
  • 1
    I'm not sure how this satisfies the "easiest tools" criteria; just the amont of typing alone...
    – jasonwryan
    Jan 14, 2016 at 20:17
  • I prefer php to awk and sed and this is the only answer that actually does what the original post requested by removing " and ; from only the fifth column. Give me that point back.
    – jbrahy
    Jan 14, 2016 at 20:18
  • I wasn't the downvoter, and no, my edited answer also only operates on the fifth field (and has other advantages besides brevity)...
    – jasonwryan
    Jan 14, 2016 at 20:24
  • ah, ok. I didn't see the edited version. $5 is definitely less typing. For me PHP code is easier so I provided a solution I thought would help someone.
    – jbrahy
    Jan 14, 2016 at 20:25
  • 1
    Fair enough, it is always good to see solutions using different approaches...
    – jasonwryan
    Jan 14, 2016 at 20:46
3

A sed solution that makes sure we're only fiddling around with the fifth column:

sed -E 's/^(([^ ]+ +){4})"([^"]+)";$/\1\3/' infile
chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9

This works also without ERE (-E, or -r for some older sed), but requires a lot more backslashes. The +-quantifier is ERE-only according to the POSIX spec1 and can be replaced by {1,} (or \{1,\} for BRE).

In case the columns aren't space-separated, the spaces can be replaced by the [:blank:] POSIX character class to also match tabs.

The regex in detail:

^               # Anchored at start of line
(               # Capture group 1 for first 4 columns
    (           # Capture group 2 for repeat count
        [^ ]+   # 1 or more non-spaces
         +      # 1 or more spaces
    ){4}        # 4 times "word plus spaces" (columns)
)               # End capture group 1
"               # Column 5 starts with double quote (not captured)
(               # Capture group 3 for column 5
    [^"]+       # One or more non-quote characters
)               # End capture group 3
";              # Quote and semicolon at end of column 5
$               # Anchored at end of line

1 GNU sed, as an extension, allows \+ to be used in BRE as well.

2

If every line has fixed length (as in the example) than

cut -c1-28,30-46 INFILE

will work.

1

In bash you can use string manipulation to achieve what you want. Here is the code

[root@localhost]# cat ./test.sh
#!/usr/bin/env bash

while IFS= read -r line; do
        echo ${line//[\";]/}
done < sample.txt

and this is the output

[root@localhost]# ./test.sh
chr1 134901 139379 - ENSG00000237683.5
chr1 860260 879955 + ENSG00000187634.6
chr1 861264 866445 - ENSG00000268179.1
chr1 879584 894689 - ENSG00000188976.6
chr1 895967 901095 + ENSG00000187961.9

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .