6

I have this type of file

#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
chr1    69511   rs2691305       A       G       .       PASS    AC=70482;AN=83312;AF=0.846001;popmax=eas;faf95_popmax=0.975176;AC_non_v2_XX=28734;AN_non_v2_XX=33774;AF_non_v2_XX=0.850773;nhomalt_non_v2_XX=13253;AC_non_cancer_fin_XX=1080;AN_non_cancer_fin_XX=1090;AF_non_cancer_fin_XX=0.990826;nhomalt_non_cancer_fin_XX=537;AC_non_neuro_nfe=32992;AN_non_neuro_nfe=34106;AF_non_neuro_nfe=0.967337;nhomalt_non_neuro_nfe=16243;AC_non_neuro_afr_XY=5275;AN_non_neuro_afr_XY=8862;AF_non_neuro_afr_XY=0.595238;nhomalt_non_neuro_afr_XY=1908;AC_non_neuro_nfe_XY=13529;AN_non_neuro_nfe_XY=13954;AF_non_neuro_nfe_XY=0.969543;nhomalt_non_neuro_nfe_XY=6668;AC_controls_and_biobanks_eas_XY=1209;AN_controls_and_biobanks_eas_XY=1210;AF_controls_and_biobanks_eas_XY=0.999174;nhomalt_controls_and_biobanks_eas_XY=604;AC_non_neuro_sas_XX=623;AN_non_neuro_sas_XX=646;AF_non_neuro_sas_XX=0.964396;nhomalt_non_neuro_sas_XX=304;AC_non_v2=53211;AN_non_v2=62346;AF_non_v2=0.853479;nhomalt_non_v2=24615;AC_non_topmed_nfe_XX=4123;AN_non_topmed_nfe_XX=4274;AF_non_topmed_nfe_XX=0.96467;nhomalt_non_topmed_nfe_XX=2022;AC_non_v2_mid=132;AN_non_v2_mid=146;AF_non_v2_mid=0.90411;nhomalt_non_v2_mid=63;AC_non_topmed_sas=2561;AN_non_topmed_sas=2620;AF_non_topmed_sas=0.977481;nhomalt_non_topmed_sas=1262;AC_non_cancer_eas_XX=1882;AN_non_cancer_eas_XX=1882;AC_nfe=34637;AN_nfe=35806;AF_nfe=0.967352;nhomalt_nfe=17050;AC_popmax=4423;AN_popmax=4424;AF_popmax=0.999774;nhomalt_popmax=2211;faf95_sas=0.945756;faf99_sas=0.93297;faf95_eas=0.975176;faf99_eas=0.965135;faf95_amr=0.876957;faf99_amr=0.869526;faf95_afr=0.586993;faf99_afr=0.583779;faf95=0.840765;faf99=0.838605;faf95_nfe=0.958818;faf99_nfe=0.955301;age_hist_het_bin_freq=123|146|149|217|287|242|200|187|124|72;age_hist_het_n_smaller=460;age_hist_het_n_larger=24;age_hist_hom_bin_freq=569|655|765|1405|2028|1746|1691|1518|1005|654;age_hist_hom_n_smaller=1206;age_hist_hom_n_larger=215;FS=0;MQ=41.6849;MQRankSum=-3.98;QD=26.0414;ReadPosRankSum=0.51;VarDP=1761397;QUALapprox=45869299;AS_FS=0;AS_MQ=41.6847;AS_MQRankSum=-3.972;AS_pab_max=1;AS_QD=26.0413;AS_ReadPosRankSum=0.51;AS_SOR=0.865485;InbreedingCoeff=0.521183;AS_SB_TABLE=99037,96244|856526,709549;AS_VQSLOD=0.4266;AS_culprit=AS_MQRankSum;NEGATIVE_TRAIN_SITE;allele_type=snv;n_alt_alleles=2;variant_type=multi-snv;segdup;gq_hist_alt_bin_freq=0|0|0|0|39|89|5016|5173|2494|4247|3343|1353|2059|1764|858|1286|1197|599|785|7691;gq_hist_all_bin_freq=0|0|0|0|2833|660|5232|5237|2506|4249|3343|1354|2060|1764|859|1287|1197|599|785|7691;dp_hist_alt_bin_freq=0|0|12706|9449|5594|4053|2707|1532|828|486|262|166|89|45|36|12|6|7|2|3;dp_hist_alt_n_larger=10;dp_hist_all_bin_freq=0|0|13838|11344|6042|4209|2735|1535|828|487|262|166|89|45|36|12|6|7|2|3;dp_hist_all_n_smaller=0;dp_hist_all_n_larger=10;ab_hist_alt_bin_freq=0|0|0|0|193|367|513|688|876|676|772|504|439|263|124|56|25|8|0|0;cadd_raw_score=0.112916;cadd_phred=2.209;revel_score=0.053;splice_ai_max_ds=0.02;splice_ai_consequence=donor_gain;primate_ai_score=0.632586;vep=G|missense_variant|MODERATE|OR4F5|ENSG00000186092|Transcript|ENST00000335137|protein_coding|1/1||ENST00000335137.4:c.421A>G|ENSP00000334393.3:p.Thr141Ala|457|421|141|T/A|Aca/Gca|1||1|SNV||HGNC|HGNC:14825|YES||P1|CCDS30547.1|ENSP00000334393|||||tolerated(0.820)|benign(0.000)|Gene3D:1&Pfam:PF13853&PROSITE_profiles:PS50262&Superfamily:SSF81321&Transmembrane_helices:TMhelix&CDD:cd15226|||||||||,G|missense_variant|MODERATE|OR4F5|ENSG00000186092|Transcript|ENST00000641515|protein_coding|3/3||ENST00000641515.2:c.484A>G|ENSP00000493376.2:p.Thr162Ala|544|484|162|T/A|Aca/Gca|1||1|SNV||HGNC|HGNC:14825|||||ENSP00000493376|||||tolerated(0.850)|benign(0.000)|Transmembrane_helices:TMhelix&CDD:cd15226&PANTHER:PTHR26451&PANTHER:PTHR26451&Pfam:PF13853&PROSITE_profiles:PS50262&Gene3D:1&Superfamily:SSF81321|||||||||,G|missense_variant|MODERATE|OR4F5|79501|Transcript|NM_001005484.1|protein_coding|1/1||NM_001005484.1:c.421A>G|NP_001005484.1:p.Thr141Ala|421|421|141|T/A|Aca/Gca|1||1|SNV||EntrezGene|HGNC:14825|YES||||NP_001005484.1|||||tolerated(0.820)|benign(0.000)||||||||||

and all the strings containing _XX between semicolons should be removed, so that from

AF_controls_and_biobanks_eas_XY=0.999174;nhomalt_controls_and_biobanks_eas_XY=604;AC_non_neuro_sas_XX=623;AN_non_neuro_sas_XX=646;

only

AF_controls_and_biobanks_eas_XY=0.999174;nhomalt_controls_and_biobanks_eas_XY=604;

stays.

I tried to use sed, but unsucesfully:

s/[^ ]*_XX[^ ]*//ig

sed -e 's/[^ ]*_XX[^ ]*//ig'

sed '/^_XX/s//'
   
sed 's/*XX//' 

sed 's/^[0-9]*_XX|//'

sed 's/;.*_XX//'

Any help will be appreciated to save my life.

Thank you!

4
  • Welcome to the site. Can you specify how your columns are separated (spaces, tabs, combinations of those)? Is it always the 8th column in the file that contains this ;-separated list of items? Can there be spaces in any of these ;-separated "sub-fields"?
    – AdminBee
    Jul 2, 2021 at 16:00
  • Yes, what @AdminBee said. That looks like a VCF file in which case, fields are separated by tabs, and there can be spaces (but not tabs) in the ;-separated subfields.
    – terdon
    Jul 2, 2021 at 16:01
  • This is gnomad vcf file; it is always 8th column, INFO field; columns are separated with tabs; yes, there can be space in ; separated fields. Fields to be removed always contain "_XX= " in the middle and are located in ; ; Jul 2, 2021 at 16:06
  • So the XX can never be the first sub-field of the INFO field? Also, do you also want to remove the corresponding entries from the header (not shown in your question)? Leaving them isn't a problem for the VCF standard, so up to you.
    – terdon
    Jul 2, 2021 at 16:07

5 Answers 5

7

Here is the solution:

sed 's/;[^;]*_XX[^;]*//g'

You need to look for _XX within two ;s and so, you should let every other character pass.

4
  • 2
    That won't work if the sub-field with the _XX is the first sub-field of the INFO field. Try sed -E 's/(\t|;)[^;]+_XX=[^;]+;?/$1/g' instead.
    – terdon
    Jul 2, 2021 at 16:06
  • 1
    @terdon that won't work for overlapping matches
    – rowboat
    Jul 2, 2021 at 17:06
  • @rowboat like what? The pattern is anchored by either a tab or a ; on the left and (I now see) isn't anchored on the right which is a problem, but easily fixed with sed -E 's/(\t|;)[^;]+_XX=[^;]+(;|\t)/$1/g'. However, I don't see how you could get overlapping patterns with these data.
    – terdon
    Jul 2, 2021 at 17:31
  • 1
    @rowboat the suggestion from my second comment does indeed not work (it's what I get for posting without testing). But the first does for all except the 1st and last sub-field and shouldn't be affected by overlapping matches in any way. Anyway, I posted my own answer, so if you still find issues with it, please leave a comment there so we don't spam poor unxnut :)
    – terdon
    Jul 2, 2021 at 19:35
4

Unxnut already gave you a working solution, but in order for this to work safely in VCF files, you would also want to ensure that you skip the header lines (those that start with #) and only apply the change to the INFO field, the 8th tab-separated field of the line. Finally, you may as well make it so that it also works if the _XX sub-field is the first sub-field of the INFO field. Putting all this together, you can do:

sed -E '/^[^#]/{s/(\t|;)[^;]+_XX=[^;\t]+/\1/g}' file.vcf
1
  • 1
    @daniele yes, thank you, I was thinking in Perl.
    – terdon
    Jul 3, 2021 at 14:32
4

To manipulate text in the 8th field of tab separated data, and only on lines that don't start with #:

awk 'BEGIN { FS = OFS = "\t" }
!/^#/ {
    gsub(/;[^;=]*_XX=[^;]*/, "", $8)
    sub(/^[^;=]*_XX=[^;]*(;|$)/, "", $8)
} 1
' file
3
awk -F'\t' -v OFS='\t' 'NR>1{
  split($NF, a, ";")
  $NF=""
  for(i=1; i in a; i++)
    if(a[i]!~/_XX=/)
      $NF = $NF ";" a[i]
  sub(/^;/, "", $NF)
}1' data

Use the tab as field separator and identifie the last field.

Split this by each ; and store it in an array.

Scan each element in the array and if this hold an _XX then skip this otherwyse use it to rebuild the field.

1
  • 1
    Doing for(I in a) can break the order of the subfields while reconstructing $NF. Hence the change to a C-style for loop.
    – guest_7
    Jul 3, 2021 at 16:59
3

Using Miller, splitting the INFO field into subfields, regex-filtering the subkeys, then joining it back together:

mlr --tsv put -S '
  m = splitkvx($INFO,"=",";"); 
  for(k,v in m){if(string(k) =~ "_XX$"){m = mapexcept(m,k)}}; 
  $INFO = joinkv(m,"=",";")
' file.vcf
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
chr1    69511   rs2691305       A       G       .       PASS    AC=70482;AN=83312;AF=0.846001;popmax=eas;faf95_popmax=0.975176;AC_non_neuro_nfe=32992;AN_non_neuro_nfe=34106;AF_non_neuro_nfe=0.967337;nhomalt_non_neuro_nfe=16243;AC_non_neuro_afr_XY=5275;AN_non_neuro_afr_XY=8862;AF_non_neuro_afr_XY=0.595238;nhomalt_non_neuro_afr_XY=1908;AC_non_neuro_nfe_XY=13529;AN_non_neuro_nfe_XY=13954;AF_non_neuro_nfe_XY=0.969543;nhomalt_non_neuro_nfe_XY=6668;AC_controls_and_biobanks_eas_XY=1209;AN_controls_and_biobanks_eas_XY=1210;AF_controls_and_biobanks_eas_XY=0.999174;nhomalt_controls_and_biobanks_eas_XY=604;AC_non_v2=53211;AN_non_v2=62346;AF_non_v2=0.853479;nhomalt_non_v2=24615;AC_non_v2_mid=132;AN_non_v2_mid=146;AF_non_v2_mid=0.90411;nhomalt_non_v2_mid=63;AC_non_topmed_sas=2561;47=AN_non_t

or (perhaps more efficient) assemble the keys whose elements are to be excluded into a map, then use mapdiff to remove them together:

$ mlr --tsv put -S '
  m = splitkvx($INFO,"=",";");
  e = {};
  for(k,v in m){if(string(k) =~ "_XX$"){e[k] = 1}};
  $INFO = joinkv(mapdiff(m,e),"=",";")
' file.vcf
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
chr1    69511   rs2691305       A       G       .       PASS    AC=70482;AN=83312;AF=0.846001;popmax=eas;faf95_popmax=0.975176;AC_non_neuro_nfe=32992;AN_non_neuro_nfe=34106;AF_non_neuro_nfe=0.967337;nhomalt_non_neuro_nfe=16243;AC_non_neuro_afr_XY=5275;AN_non_neuro_afr_XY=8862;AF_non_neuro_afr_XY=0.595238;nhomalt_non_neuro_afr_XY=1908;AC_non_neuro_nfe_XY=13529;AN_non_neuro_nfe_XY=13954;AF_non_neuro_nfe_XY=0.969543;nhomalt_non_neuro_nfe_XY=6668;AC_controls_and_biobanks_eas_XY=1209;AN_controls_and_biobanks_eas_XY=1210;AF_controls_and_biobanks_eas_XY=0.999174;nhomalt_controls_and_biobanks_eas_XY=604;AC_non_v2=53211;AN_non_v2=62346;AF_non_v2=0.853479;nhomalt_non_v2=24615;AC_non_v2_mid=132;AN_non_v2_mid=146;AF_non_v2_mid=0.90411;nhomalt_non_v2_mid=63;AC_non_topmed_sas=2561;47=AN_non_t

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.