6

I have data like this:

chr1    134901  139379  -   "ENSG00000237683.5";
chr1    860260  879955  +   "ENSG00000187634.6";
chr1    861264  866445  -   "ENSG00000268179.1";
chr1    879584  894689  -   "ENSG00000188976.6";
chr1    895967  901095  +   "ENSG00000187961.9";

I generated by parsing a GTF file

I want to remove the "'s and ;'s from column 5 using awk or sed if it possible. The result would look like this:

chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9
  • 1
    you can also use multiple seach and replace statements in sed. sed 's/"//g; s/;//g' filename – jbrahy Jan 14 '16 at 19:55
  • @DigitalTrauma ya, but Dani_l already gave that solution. – jbrahy Jan 14 '16 at 23:59
6

Using gsub:

awk '{gsub(/\"|\;/,"")}1' file
chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9

If you want to operate only on the fifth field and preserve any quotes or semicolons in other fields:

awk '{gsub(/\"|\;/,"",$5)}1' file 
  • 1
    This would remove from all columns, not just 5th, no? – Dani_l Jan 14 '16 at 19:55
  • This is what I thought initally, but after using the code it seemed to keep all columns. – System Jan 14 '16 at 19:57
  • @Dani_l Yes, it can be refined to operate only on the fifth field, but that was not a requirement... – jasonwryan Jan 14 '16 at 19:57
  • Sorry I must have not made it clear, I DO want to keep all columns. This is why it is marked as the answer. – System Jan 14 '16 at 19:58
  • @System updated to ensure it only operates on the fifth field. – jasonwryan Jan 14 '16 at 20:15
5

Using sed to remove all instances of '";': sed -i 's/[";]//g' file

To only remove from 5th column sed is probably not the best option.

5

If your data is formatted exactly as shown (i.e. no other " or ; in other columns that need to be preserved), then you can simply use tr to remove these characters:

tr -d '";' < input.txt > output.txt
3

I know the original post asked for sed or awk but if you want to remove the " and ; from only the fifth column I'd use regex and php. There's probably a way to do this in AWK but I like to use the easiest tools.

<?php

foreach(file($argv[1]) as $line){

    $matches = array();
    preg_match('/^(\w+)\s+(\d+)\s+(\d+)\s+(\-|\+)\s+"(\w+.\d)"\;/',$line,$matches);
    $matched_line = array_shift($matches); // remove the first element
    vprintf("%s\t%s\t%s\t%s\t%s\n",$matches);
}

this would output this

$ php /tmp/preg_replace.php /tmp/data
chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9
  • 1
    I'm not sure how this satisfies the "easiest tools" criteria; just the amont of typing alone... – jasonwryan Jan 14 '16 at 20:17
  • I prefer php to awk and sed and this is the only answer that actually does what the original post requested by removing " and ; from only the fifth column. Give me that point back. – jbrahy Jan 14 '16 at 20:18
  • I wasn't the downvoter, and no, my edited answer also only operates on the fifth field (and has other advantages besides brevity)... – jasonwryan Jan 14 '16 at 20:24
  • ah, ok. I didn't see the edited version. $5 is definitely less typing. For me PHP code is easier so I provided a solution I thought would help someone. – jbrahy Jan 14 '16 at 20:25
  • Fair enough, it is always good to see solutions using different approaches... – jasonwryan Jan 14 '16 at 20:46
3

A sed solution that makes sure we're only fiddling around with the fifth column:

sed -E 's/^(([^ ]+ +){4})"([^"]+)";$/\1\3/' infile
chr1    134901  139379  -   ENSG00000237683.5
chr1    860260  879955  +   ENSG00000187634.6
chr1    861264  866445  -   ENSG00000268179.1
chr1    879584  894689  -   ENSG00000188976.6
chr1    895967  901095  +   ENSG00000187961.9

This works also without ERE (-E, or -r for some older sed), but requires a lot more backslashes. The +-quantifier is ERE-only according to the POSIX spec1 and can be replaced by {1,} (or \{1,\} for BRE).

In case the columns aren't space-separated, the spaces can be replaced by the [:blank:] POSIX character class to also match tabs.

The regex in detail:

^               # Anchored at start of line
(               # Capture group 1 for first 4 columns
    (           # Capture group 2 for repeat count
        [^ ]+   # 1 or more non-spaces
         +      # 1 or more spaces
    ){4}        # 4 times "word plus spaces" (columns)
)               # End capture group 1
"               # Column 5 starts with double quote (not captured)
(               # Capture group 3 for column 5
    [^"]+       # One or more non-quote characters
)               # End capture group 3
";              # Quote and semicolon at end of column 5
$               # Anchored at end of line

1 GNU sed, as an extension, allows \+ to be used in BRE as well.

2

If every line has fixed length (as in the example) than

cut -c1-28,30-46 INFILE

will work.

0

In bash you can use string manipulation to achieve what you want. Here is the code

[root@localhost]# cat ./test.sh
#!/usr/bin/env bash

while IFS= read -r line; do
        echo ${line//[\";]/}
done < sample.txt

and this is the output

[root@localhost]# ./test.sh
chr1 134901 139379 - ENSG00000237683.5
chr1 860260 879955 + ENSG00000187634.6
chr1 861264 866445 - ENSG00000268179.1
chr1 879584 894689 - ENSG00000188976.6
chr1 895967 901095 + ENSG00000187961.9

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.