adds ALL option to extract all VEP fields

docs/vep_annotation_reporter.rst

@@ -19,6 +19,11 @@ input VCF can be identified by inspecting the ``Description`` field of the
 ``CSQ`` ``INFO`` header in the VCF. Everything after ``Format:`` is a field
 available in the VCF (delimited by ``|``).
 
+To report every VEP field, pass the keyword ``ALL`` instead of a list of
+fields. The tool will parse the ``CSQ`` header and add one column per field,
+in the order they appear in the header. ``ALL`` cannot be combined with other
+field names.
+
 By default, VEP annotates each variant with it's consequences in every transcript it intersects. In this case, the values for all transcript annotation will be returned as comma-separated values. 
 
 If VEP was run with one of the ``--flag_pick`` options, then it has labelled one of the consequences as the "best", and set the ``PICK`` field. If this field is
@@ -55,7 +60,15 @@ Download the example data used below:
 
 Outputs one row: ``22 18644673 C T missense_variant USP18 ENST00000215794 A/V``.
 
-**2. Multi-transcript variant, default (no PICK) behavior**
+**2. Report all VEP fields**
+
+.. code-block:: none
+
+   vep-annotation-reporter sample.vcf ALL -o sample.vep_report_all.tsv
+
+Outputs one column for every field listed in the VCF's ``CSQ`` header (``Allele``, ``Consequence``, ``IMPACT``, ``SYMBOL``, ...).
+
+**3. Multi-transcript variant, default (no PICK) behavior**
 
 .. code-block:: none
 
@@ -64,7 +77,7 @@ Outputs one row: ``22 18644673 C T missense_variant USP18 ENST00000215794 A/V``.
 
 Since VEP wasn't run with ``--flag_pick`` here, both transcripts' values are comma-separated in each field: ``intron_variant&nc_transcript_variant,non_coding_exon_variant&nc_transcript_variant``.
 
-**3.** ``-p`` **preferred transcripts**
+**4.** ``-p`` **preferred transcripts**
 
 .. code-block:: none
 
@@ -73,7 +86,7 @@ Since VEP wasn't run with ``--flag_pick`` here, both transcripts' values are com
 
 Restricts the row to only ``ENST00000423297`` (``GRAMD4P2``) instead of both transcripts.
 
-**4.** ``-t`` **merge into an existing TSV**
+**5.** ``-t`` **merge into an existing TSV**
 
 .. code-block:: none
 

tests/test_data/multiple_samples.gx.gx.vcf

@@ -0,0 +1,82 @@
+##fileformat=VCFv4.0
+##source=VarscanSomatic
+##reference=ftp://ftp.ncbi.nih.gov/genbank/genomes/Eukaryotes/vertebrates_mammals/Homo_sapiens/GRCh37/special_requests/GRCh37-lite.fa.gz
+##phasing=none
+##center=genome.wustl.edu
+##FILTER=<ID=PASS,Description="Passed all filters">
+##FILTER=<ID=VarscanHighConfidenceIndel,Description="Filter description">
+##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
+##FORMAT=<ID=DP4,Number=4,Type=Integer,Description="Number of high-quality ref-forward, ref-reverse, alt-forward and alt-reverse bases">
+##FORMAT=<ID=BQ,Number=.,Type=Integer,Description="Average base quality for reads supporting alleles">
+##FORMAT=<ID=SS,Number=1,Type=Integer,Description="Variant status relative to non-adjacent Normal,0=wildtype,1=germline,2=somatic,3=LOH,4=post-transcriptional modification,5=unknown">
+##FORMAT=<ID=GQ,Number=.,Type=Integer,Description="Conditional Phred-scaled genotype quality">
+##FORMAT=<ID=MQ,Number=1,Type=Integer,Description="Phred style probability score that the variant is novel with respect to the genome's ancestor">
+##FORMAT=<ID=FA,Number=1,Type=Float,Description="Fraction of reads supporting ALT">
+##FORMAT=<ID=VAQ,Number=1,Type=Integer,Description="Variant allele quality">
+##FORMAT=<ID=FT,Number=1,Type=String,Description="Sample genotype filter">
+##FORMAT=<ID=AMQ,Number=.,Type=Integer,Description="Average mapping quality for each allele present in the genotype">
+##FORMAT=<ID=IGT,Number=1,Type=String,Description="Genotype when called independently (only filled if called in joint prior mode)">
+##FORMAT=<ID=BCOUNT,Number=4,Type=Integer,Description="Occurrence count for each base at this site (A,C,G,T)">
+##FORMAT=<ID=JGQ,Number=1,Type=Integer,Description="Joint genotype quality (only filled if called in join prior mode)">
+##FORMAT=<ID=SSC,Number=1,Type=Integer,Description="Somatic score between 0 and 255">
+##source=Strelka
+##FORMAT=<ID=FDP,Number=1,Type=Integer,Description="Number of basecalls filtered from original read depth for tier1">
+##FORMAT=<ID=SDP,Number=1,Type=Integer,Description="Number of reads with deletions spanning this site at tier1">
+##FORMAT=<ID=SUBDP,Number=1,Type=Integer,Description="Number of reads below tier1 mapping quality threshold aligned across this site">
+##FORMAT=<ID=AU,Number=2,Type=Integer,Description="Number of 'A' alleles used in tiers 1,2">
+##FORMAT=<ID=CU,Number=2,Type=Integer,Description="Number of 'C' alleles used in tiers 1,2">
+##FORMAT=<ID=GU,Number=2,Type=Integer,Description="Number of 'G' alleles used in tiers 1,2">
+##FORMAT=<ID=TU,Number=2,Type=Integer,Description="Number of 'T' alleles used in tiers 1,2">
+##FORMAT=<ID=DP2,Number=1,Type=Integer,Description="Read depth for tier2">
+##FORMAT=<ID=TAR,Number=2,Type=Integer,Description="Reads strongly supporting alternate allele for tiers 1,2">
+##FORMAT=<ID=TIR,Number=2,Type=Integer,Description="Reads strongly supporting indel allele for tiers 1,2">
+##FORMAT=<ID=TOR,Number=2,Type=Integer,Description="Other reads (weak support or insufficient indel breakpoint overlap) for tiers 1,2">
+##FORMAT=<ID=DP50,Number=1,Type=Float,Description="Average tier1 read depth within 50 bases">
+##FORMAT=<ID=FDP50,Number=1,Type=Float,Description="Average tier1 number of basecalls filtered from original read depth within 50 bases">
+##FORMAT=<ID=SUBDP50,Number=1,Type=Float,Description="Average number of reads below tier1 mapping quality threshold aligned across sites within 50 bases">
+##INFO=<ID=QSI,Number=1,Type=Integer,Description="Quality score for any somatic variant, ie. for the ALT haplotype to be present at a significantly different frequency in the tumor and normal">
+##INFO=<ID=TQSI,Number=1,Type=Integer,Description="Data tier used to compute QSI">
+##INFO=<ID=NT,Number=1,Type=String,Description="Genotype of the normal in all data tiers, as used to classify somatic variants. One of {ref,het,hom,conflict}.">
+##INFO=<ID=QSI_NT,Number=1,Type=Integer,Description="Quality score reflecting the joint probability of a somatic variant and NT">
+##INFO=<ID=TQSI_NT,Number=1,Type=Integer,Description="Data tier used to compute QSI_NT">
+##INFO=<ID=SGT,Number=1,Type=String,Description="Most likely somatic genotype excluding normal noise states">
+##INFO=<ID=RU,Number=1,Type=String,Description="Smallest repeating sequence unit in inserted or deleted sequence">
+##INFO=<ID=RC,Number=1,Type=Integer,Description="Number of times RU repeats in the reference allele">
+##INFO=<ID=IC,Number=1,Type=Integer,Description="Number of times RU repeats in the indel allele">
+##INFO=<ID=IHP,Number=1,Type=Integer,Description="Largest reference interupted homopolymer length intersecting with the indel">
+##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
+##INFO=<ID=OVERLAP,Number=0,Type=Flag,Description="Somatic indel possibly overlaps a second indel.">
+##FILTER=<ID=DP,Description="Greater than 3.0x chromosomal mean depth in Normal sample">
+##FILTER=<ID=Repeat,Description="Sequence repeat of more than 8x in the reference sequence">
+##FILTER=<ID=iHpol,Description="Indel overlaps an interupted homopolymer longer than 14x in the reference sequence">
+##FILTER=<ID=BCNoise,Description="Average fraction of filtered basecalls within 50 bases of the indel exceeds 0.3">
+##FILTER=<ID=QSI_ref,Description="Normal sample is not homozygous ref or sindel Q-score < 30, ie calls with NT!=ref or QSI_NT < 30">
+##source=Pindel
+##FILTER=<ID=PindelSomaticCalls,Description="Filter description">
+##FILTER=<ID=PindelVafFilter,Description="Filter description">
+##FILTER=<ID=PindelReadSupport,Description="Filter description">
+##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
+##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
+##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
+##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
+##INFO=<ID=NTLEN,Number=.,Type=Integer,Description="Number of bases inserted in place of deleted code">
+##source=GatkSomaticIndel
+##fileDate=20160321
+##INFO=<ID=QSS,Number=1,Type=Integer,Description="Quality score for any somatic snv, ie. for the ALT allele to be present at a significantly different frequency in the tumor and normal">
+##INFO=<ID=TQSS,Number=1,Type=Integer,Description="Data tier used to compute QSS">
+##INFO=<ID=QSS_NT,Number=1,Type=Integer,Description="Quality score reflecting the joint probability of a somatic variant and NT">
+##INFO=<ID=TQSS_NT,Number=1,Type=Integer,Description="Data tier used to compute QSS_NT">
+##FILTER=<ID=VarFilterSnv,Description="Filter description">
+##FILTER=<ID=FalsePositiveVcf,Description="Filter description">
+##FILTER=<ID=FalsePositive,Description="Filter description">
+##FILTER=<ID=SomaticScoreMappingQuality,Description="Filter description">
+##FILTER=<ID=IntersectionFailure,Description="Variant callers do not agree on this position">
+##FILTER=<ID=SpanDel,Description="Fraction of reads crossing site with spanning deletions in either sample exceeeds 0.75">
+##FILTER=<ID=QSS_ref,Description="Normal sample is not homozygous ref or ssnv Q-score < 15, ie calls with NT!=ref or QSS_NT < 15">
+##FORMAT=<ID=TLOD,Number=.,Type=Float,Description="Log of (likelihood tumor event is real / likelihood event is sequencing error)">
+##FILTER=<ID=REJECT,Description="Rejected as a confident somatic mutation by MuTect">
+##FILTER=<ID=VarscanHighConfidence,Description="Filter description">
+##FILTER=<ID=PASS,Description="Passed all filters">
+##INFO=<ID=CSQ,Number=.,Type=String,Description="Consequence type as predicted by VEP. Format: Allele|Gene|Feature|Feature_type|Consequence|cDNA_position|CDS_position|Protein_position|Amino_acids|Codons|Existing_variation|MOTIF_NAME|MOTIF_POS|HIGH_INF_POS|MOTIF_SCORE_CHANGE|DISTANCE|STRAND|CANONICAL|SYMBOL|SYMBOL_SOURCE|SIFT|PolyPhen|HGVSc|HGVSp|DownstreamProtein|ProteinLengthChange|WildtypeProtein">
+##FORMAT=<ID=GX,Number=.,Type=String,Description="Gene Expression">
+#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	H_NJ-HCC1395-HCC1395	H_NJ-HCC1395-HCC1396

tests/test_data/vep_annotation_reporter/output.all_fields.input_tsv.tsv

@@ -0,0 +1,3 @@
+CHROM	POS	REF	ALT	AF	H_NJ-HCC1395-HCC1395.GT	H_NJ-HCC1395-HCC1395.AD	H_NJ-HCC1395-HCC1395.DP	Allele	Consequence	IMPACT	SYMBOL	Gene	Feature_type	Feature	BIOTYPE	EXON	INTRON	HGVSc	HGVSp	cDNA_position	CDS_position	Protein_position	Amino_acids	Codons	Existing_variation	DISTANCE	STRAND	FLAGS	PICK	SYMBOL_SOURCE	HGNC_ID	TSL	SOURCE	HGVS_OFFSET	CLIN_SIG	SOMATIC	PHENO	DownstreamProtein	ProteinLengthChange	WildtypeProtein	gnomADe	gnomADe_AF	gnomADe_AF_AFR	gnomADe_AF_AMR	gnomADe_AF_ASJ	gnomADe_AF_EAS	gnomADe_AF_FIN	gnomADe_AF_NFE	gnomADe_AF_OTH	gnomADe_AF_SAS
+chr17	7675088	C	T	1.00	T/T	0,95	95	T	missense_variant	MODERATE	TP53	ENSG00000141510	Transcript	ENST00000269305.8	protein_coding	5/11		ENST00000269305.8:c.524G>A	ENSP00000269305.4:p.Arg175His	714	524	175	R/H	cGc/cAc	CM062017&CM951224&TP53_g.12512G>A&rs28934578&COSM10648&COSM10718&COSM1640851&COSM1679508&COSM1679509&COSM1679510&COSM1679511&COSM1679512&COSM3355994&COSM4271918&COSM45416&COSM99022&COSM99023&COSM99024&COSM99914		-1		1	HGNC	HGNC:11998	1			not_provided&likely_pathogenic&pathogenic	0&0&0&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1	1&1&0&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1			MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGPDEAPRMPEAAPPVAPAPAAPTPAAPAPAPSWPLSSSVPSQKTYQGSYGFRLGFLHSGTAKSVTCTYSPALNKMFCQLAKTCPVQLWVDSTPPPGTRVRAMAIYKQSQHMTEVVRRCPHHERCSDSDGLAPPQHLIRVEGNLRVEYLDDRNTFRHSVVVPYEPPEVGSDCTTIHYNYMCNSSCMGGMNRRPILTIITLEDSSGNLLGRNSFEVRVCACPGRDRRTEEENLRKKGEPHHELPPGSTKRALPNNTSSSPQPKKKPLDGEYFTLQIRGRERFEMFRELNEALELKDAQAGKEPGGSRAHSSHLKSKKGQSTSRHKKLMFKTEGPDSD	rs28934578	4.06392e-06	0.00000e+00	0.00000e+00	0.00000e+00	0.00000e+00	0.00000e+00	8.96459e-06	0.00000e+00	0.00000e+00
+chr17	43057078	G	A	1.00	A/A	0,38	38	A	stop_gained	HIGH	BRCA1	ENSG00000012048	Transcript	ENST00000471181.6	protein_coding	20/24		ENST00000471181.6:c.5314C>T	ENSP00000418960.2:p.Arg1772Ter	5546	5314	1772	R/*	Cga/Tga	rs80357123&CM970175&COSM4066736&COSM51256		-1		1	HGNC	HGNC:1100	1			pathogenic	0&0&1&1	1&1&1&1			MDLSALRVEEVQNVINAMQKILECPICLELIKEPVSTKCDHIFCKFCMLKLLNQKKGPSQCPLCKNDITKRSLQESTRFSQLVEELLKIICAFQLDTGLEYANSYNFAKKENNSPEHLKDEVSIIQSMGYRNRAKRLLQSEPENPSLQETSLSVQLSNLGTVRTLRTKQRIQPQKTSVYIELGSDSSEDTVNKATYCSVGDQELLQITPQGTRDEISLDSAKKAACEFSETDVTNTEHHQPSNNDLNTTEKRAAERHPEKYQGSSVSNLHVEPCGTNTHASSLQHENSSLLLTKDRMNVEKAEFCNKSKQPGLARSQHNRWAGSKETCNDRRTPSTEKKVDLNADPLCERKEWNKQKLPCSENPRDTEDVPWITLNSSIQKVNEWFSRSDELLGSDDSHDGESESNAKVADVLDVLNEVDEYSGSSEKIDLLASDPHEALICKSERVHSKSVESNIEDKIFGKTYRKKASLPNLSHVTENLIIGAFVTEPQIIQERPLTNKLKRKRRPTSGLHPEDFIKKADLAVQKTPEMINQGTNQTEQNGQVMNITNSGHENKTKGDSIQNEKNPNPIESLEKESAFKTKAEPISSSISNMELELNIHNSKAPKKNRLRRKSSTRHIHALELVVSRNLSPPNCTELQIDSCSSSEEIKKKKYNQMPVRHSRNLQLMEGKEPATGAKKSNKPNEQTSKRHDSDTFPELKLTNAPGSFTKCSNTSELKEFVNPSLPREEKEEKLETVKVSNNAEDPKDLMLSGERVLQTERSVESSSISLVPGTDYGTQESISLLEVSTLGKAKTEPNKCVSQCAAFENPKGLIHGCSKDNRNDTEGFKYPLGHEVNHSRETSIEMEESELDAQYLQNTFKVSKRQSFAPFSNPGNAEEECATFSAHSGSLKKQSPKVTFECEQKEENQGKNESNIKPVQTVNITAGFPVVGQKDKPVDNAKCSIKGGSRFCLSSQFRGNETGLITPNKHGLLQNPYRIPPLFPIKSFVKTKCKKNLLEENFEEHSMSPEREMGNENIPSTVSTISRNNIRENVFKEASSSNINEVGSSTNEVGSSINEIGSSDENIQAELGRNRGPKLNAMLRLGVLQPEVYKQSLPGSNCKHPEIKKQEYEEVVQTVNTDFSPYLISDNLEQPMGSSHASQVCSETPDDLLDDGEIKEDTSFAENDIKESSAVFSKSVQKGELSRSPSPFTHTHLAQGYRRGAKKLESSEENLSSEDEELPCFQHLLFGKVNNIPSQSTRHSTVATECLSKNTEENLLSLKNSLNDCSNQVILAKASQEHHLSEETKCSASLFSSQCSELEDLTANTNTQDPFLIGSSKQMRHQSESQGVGLSDKELVSDDEERGTGLEENNQEEQSMDSNLGEAASGCESETSVSEDCSGLSSQSDILTTQQRDTMQHNLIKLQQEMAELEAVLEQHGSQPSNSYPSIISDSSALEDLRNPEQSTSEKDSHIHGQRNNSMFSKRPREHISVLTSQKSSEYPISQNPEGLSADKFEVSADSSTSKNKEPGVERSSPSKCPSLDDRWYMHSCSGSLQNRNYPSQEELIKVVDVEEQQLEESGPHDLTETSYLPRQDLEGTPYLESGISLFSDDPESDPSEDRAPESARVGNIPSSTSALKVPQLKVAESAQSPAAAHTTDTAGYNAMEESVSREKPELTASTERVNKRMSMVVSGLTPEEFMLVYKFARKHHITLTNLITEETTHVVMKTDAEFVCERTLKYFLGIAGGKWVVSYFWVTQSIKERKMLNEHDFEVRGDVVNGRNHQGPKRARESQDRKIFRGLEICCYGPFTNMPTDQLEWMVQLCGASVVKELSSFTLGTGVHPIVVVQPDAWTEDNGFHAIGQMCEAPVVTREWVLDSVALYQCQELDTYLIPQIPHSHY	rs80357123	1.21818e-05	0.00000e+00	0.00000e+00	0.00000e+00	5.79777e-05	4.48430e-05	8.95111e-06	0.00000e+00	0.00000e+00