| Home | C. elegans | AcePerl | Gramene | Reactome | GMOD | Course | DAS | WWW |
This is a working document describing the rationale and protocol for a distributed sequence annotation system.
The pace of human genomic sequencing has outstripped the ability of sequencing centers to annotate and understand the sequence prior to submitting it to the archival databases. Multiple third-party groups have stepped into the breach and are currently annotating the human sequence with a combination of computational and experimental methods. Their analytic tools, data models, and visualization methods are diverse, and it is self-evident that this diversity enhances, rather than diminishes, the value of their work.
The main risk of third-party annotation is that it may fracture knowledge about the genome. Instead of having a convenient one-stop source for genomic annotation, such as Entrez, researchers may have to check multiple Web sites for information about a particular region of interest, download the data in several different formats, and perform a manual integration in order to get the whole picture. Clearly, this is undesirable.
There are several possible approaches to this problem. One is for each of the annotation centers to submit their annotations to a centralized database, such as GenBank. However, this option raises a number of political and technical problems, not the least of which is the long-held tradition of GenBank and its sister databases of allowing only the sequence submitter to modify or comment on a GenBank entry. Another option would be a system which uses Web links to point from the GenBank entry to one or more annotation Web sites. Such a system is available now in the form of the NCBI LinkOut service. However, while this makes it easier for researchers to find third-party annotation sites, it does not solve the problem of data integration.
The solution that we advocate allows sequence annotation to be decentralized among multiple third-party annotators and integrated on an as-needed basis by client-side software. A single server is designated the "reference server." It serves essential structural information about the genome: the physical map which relates one entry to another (where an "entry" is an arbitrary segment of the sequence, such as a sequenced BAC or a contig), the DNA sequence for each entry, and the standard authorship information. Multiple sites then act as third-party "annotation servers." Using a web browser-like application, researchers can interrogate one or more annotation servers to retrieve features in a region of interest. The servers return the results using a standard data format, allowing the sequence browser to integrate the annotations and display them in graphical or tabular form. No attempt is made to automatically resolve contradictions between different third-party annotations. Indeed, it is the ability to facilitate comparison among different centers' annotations that distinguish this proposal. We currently have a working prototype of this system based on ACeDB servers and CGI scripts, and are now generalizing this architecture to support other client and server combinations.
The key development that is necessary for a successful distributed annotation system is the adoption of a standard format to describe sequence features. While almost any one of the existing standards could be adapted for this purpose, certain characteristics are very desirable:
In the ideal world, the genome would be finished to the base pair, and we would be able to unambiguously refer to an annotation based on its position from the top of the chromosome. This will not happen for a very long time. For the conceivable future, the genome will consist of multiple segments of high confidence, related to one another by mapping information of lower confidence. In order to deal with annotations in this dynamic and changeable environment, the format must be able to deal with relative coordinates in which annotations are related to arbitrary hierarchical landmarks. For example, a "clone end" annotation may be related to the start of a contig, an "mRNA" annotation may be related to the clone end, and an "exon" annotation may be related to the start of the mRNA.
Experience has shown that it is difficult to convince groups to adopt complex and sophisticated data formats. For this reason, a "lowest common denominator" format is desirable, even if it sacrifices some of the expressiveness of the more sophisticated formats. A human-readable format, such as tab-delimited tables, XML, or even ".ace" format is also desirable.
Any format must be extensible to allow for new types of annotations. Specifically, we feel that it is desirable to create a category of annotation that has to do with the availability of experimental data concerning the region of interest. For example, the format should allow a researcher to note the presence of RNAi results overlapping the region of interest. The format should also provide a mechanism for pointing the researcher to a location where he or she can get more information about a selected annotation. In the ACeDB-based system, each annotation contains a pointer into an ACeDB entry somewhere on the Internet. This entry is in turn linked to related biological and experimental information.
To further enhance the extensibility of the format, it is desirable to group specific annotations into functional categories rather than maintaining an unsorted "laundry list" of feature types. For example, splice sites, polyA signals, introns and exons are all annotations having to do with a generic "mRNA" category, while clone ends, primer pairs, and hybridization probes are "structural" features. Grouping annotations into conceptual categories makes the data more manageable, and facilitates formulating biologically relevant queries on the annotation servers.
The remainder of this document describes a simple client/server system that satisifes many of these requirements. Information on the ACEDB-prototype and a text-only demo can be found here.
This section provides a high-level view of the system architecture.
It is possible for each entry point to have a substructure, basically a series of subsequences and their starting and ending points. This structure is recursive. Annotations take the form of an assertion about a region of the reference sequence. Each annotation is unambiguously located by providing its position as the start and stop positions relative to a "reference sequence." The reference sequence can be one of the entry points, or any of the subsequences within the entry point.
To give a concrete example, the C. elegans reference map consists of six chromosome-length entry points. Each chromosome is formed from several contigs called "superlinks", and each superlink contains one or more smaller contigs called "links". Links in turn are composed of one or more fully-sequenced clones. One could refer to an annotation by specifying its start or stop positions in clone, link, superlink, or chromosome coordinates. The distributed annotation system automatically converts any coordinate system into any other. Because coordinates within clones are more stable to revisions than coordinates within links or chromosomes, it is recommended that annotation coordinates be stored relative to the smallest sequencing unit.
The hierarchy is extensible. If the C. elegans gene predictions were stable, it would make sense to store certain annotations, such as the positions of exons, relative to the transcriptional unit.
The DAS consists of a reference sequence server, and one or more annotation servers.
The reference sequence server is specialized to provide the reference sequence map and the underlying DNA. The server can provide a list of sequence entry points, and given an entry point can return its structure. The reference server can provide arbitrarily long stretches of DNA given a reference subsequence, start position and stop position, and is capable of translating from one coordinate system to another.
Annotation servers are specialized for returning lists of annotations across a certain region of the genome. Each annotation is anchored to the genome map by way of a start and stop position relative to one of the reference subsequences. Annotations have an ID that is unique to the server and a structured description that describes its nature and attributes. Annotations may also be associated with Web URLs that provide additional human readable information about the annotation.
Annotations have types, methods and categories. The annotation type is selected from a list of types that have biological significance, and correspond roughly to EMBL/GenBank feature table tags. Examples of annotation types include "exon", "intron", "CDS" and "splice3." The annotation method is intended to describe how the annotated feature was discovered, and may include a reference to a software program. The annotation category is a broad functional category that can be used to filter, group and sort annotations. "Homology", "variation" and "transcribed" are all valid categories. The existence of these categories allows researchers to add new annotation types if the existing list is inadequate without entirely losing all semantic value. The Annotation Categories section contains a list of the annotation types in use in the C. elegans project.
It is intended that larger annotation servers provide pointers to human-readable data that describes its types, methods and categories in more detail. Another optional feature of annotation servers is the ability to provide hints to clients on how the annotations should be rendered visually. This is done by returning a XML "stylesheet".
Although the servers are conceptually divided between reference servers and annotation servers, there is in fact no key difference between them. A single server can provide both reference sequence information and annotation information. The main functional difference is that the reference sequence server is required to serve the DNA itself, while annotation servers have no such requirement.
The DAS is Web-based. Clients query the reference and annotation servers by sending a formatted URL request to the server. This request must follow the conventions of the HTTP/1.0 protocol (see RFC2616. Servers process the request and return a response in the form of a formatted XML document (see W3C Extensible Markup Language).
All DAS requests take the form of a URL. Each URL has a site-specific prefix, followed by a standardized path and query string. The standardized path begins with the string /das. This is followed by URL components containing the data source name and a command. For example:
In this case, the site-specific prefix is http://stein.cshl.org/. The request begins with the standardized path /das, and the data source, in this case /elegans. This is followed by the command /features, which requests a list of features, and a query string providing named arguments to the /features command.http://stein.cshl.org/das/elegans/features?ref=CHROMOSOME_I;start=1000;stop=2000 ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^ ^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ site-specific prefix data src command arguments
The data source component allows a single server to provide information on several genomes.
More information on the format of the request and the various available commands is given below.
Here is an example HTTP header: (provided by Web server)
HTTP/1.1 200 OK Date: Sun, 12 Mar 2000 16:13:51 GMT Server: Apache/1.3.6 (Unix) mod_perl/1.19 Last-Modified: Fri, 18 Feb 2000 20:57:52 GMT Connection: close Content-Type: text/plain X-DAS-Version: DAS/0.95 X-DAS-Status: 200 data follows...
The defined status codes are listed in Table 1.
| 200 | OK, data follows |
|---|---|
| 400 | Bad command (command not recognized) |
| 401 | Bad data source (data source unknown) |
| 402 | Bad command arguments (arguments invalid) |
| 403 | Bad reference object (reference sequence unknown) |
| 404 | Bad stylesheet (requested stylesheet unknown) |
| 405 | Coordinate error (sequence coordinate is out of bounds/invalid) |
| 500 | Server error, not otherwise specified |
| 501 | Unimplemented feature |
This section lists the queries recognized by sequence and annotation servers. Each of these queries begins with some site-specific prefix, denoted here as PREFIX. The other meta-variable used in these examples is DSN, which is a symbolic data source. Data sources are standardized across DAS servers in such a way that a data source name has a one-to-one correspondence with a reference sequence.
Scope: Sequence and annotation servers.
Command: dsn
Format:
This query returns the list of data sources that are available from this server. Return value: See here.PREFIX/das/dsn
Scope:Sequence and annotation servers.
Command: entry_points
Format:
This query returns the list of sequence entry points available and their sizes in base pairs.PREFIX/das/DSN/entry_points[?ref=REF]
Arguments:
Scope: Sequence servers.
Command: dna
Format:
This query returns the DNA corresponding to the indicated segment.PREFIX/das/DSN/dna?ref=REF[;start=X;stop=Y]
Arguments:
Scope: Sequence and annotation servers.
Command: resolve
Format:
PREFIX/das/DSN/resolve?segment=segmentID;ref=REF
[;start=X;stop=Y]
Arguments:
Scope: Annotation Servers
Command: types
Format:
PREFIX/das/DSN/types?ref=REF
[;start=X;stop=Y]
[;type=TYPEPATTERN]
Arguments:
Scope: Sequence and annotation Servers
Command: features
Format:
PREFIX/das/DSN/features?ref=REF
[;start=X&stop=Y]
[;type=TYPEPATTERN]
[;category=CATEGORYPATTERN]
[;categorize=yes|no]
Arguments:
Scope: Annotation Servers
Command: link
Format:
This query can be issued in order to retrieve further human-readable information about an annotation. It is best to pass this URL directly to a browser, as the type of the returned data is not specified (it will typically be an HTML file, but any MIME format is allowed).PREFIX/das/DSN/link?field=TAG;id=ID
Arguments:
Scope: Annotation Servers
Command: stylesheet
Format:
This query can be issued to an annotation server in order to retrieve the server's recommendations on formatting annotations retrieved from it. These recommendations are not normative. A viewer is free to use any display format it chooses.PREFIX/das/DSN/stylesheet
Arguments: None. Return Value: See Annotation Stylesheets.
This section describes the format of the various documents that are returned in response to DAS queries.
Scope: Sequence and annotation servers.
In Response to Command: dsn
Format:
<?xml version="1.0" standalone="no"?>
<!DOCTYPE DASDSN SYSTEM "dasdsn.dtd">
<DASDSN>
<DSN>
<SOURCE id="id1">source name 1</SOURCE>
<MAPMASTER>URL<MAPMASTER>
<DESCRIPTION>descriptive text 1</DESCRIPTION>
</DSN>
<DSN>
<SOURCE id="id2">source name 2</SOURCE>
<MAPMASTER>URL<MAPMASTER>
<DESCRIPTION href="url">descriptive text 2</DESCRIPTION>
</DSN>
...
</DASDSN>
Scope: Sequence and annotation servers.
In Response to Command: entry_points
Format:
<?xml version="1.0" standalone="no"?>
<!DOCTYPE DASEP SYSTEM "dasep.dtd">
<DASEP>
<ENTRY_POINTS href="url" version="X.XX" ref="refid">
<SEGMENT id="id1" start="start1" stop="stop1">descriptive text</SEGMENT>
<SEGMENT id="id2" start="start2" stop="stop2">descriptive text</SEGMENT>
<SEGMENT id="id3" start="start3" stop="stop3">descriptive text</SEGMENT>
...
</ENTRY_POINTS>
</DASEP>
If the entry points are not "top level", that is, if they were generated by a request for the substructure of a sequence, then the ref attribute will be present and will indicate the ID of the reference sequence.
The href (required) attribute echoes the URL query that was used to fetch the current document.
The body of the <SEGMENT> sections contains human-readable text (optional) for the purposes of display and selection.
Scope: Sequence servers
In Response to Command: dna
Format:
<?xml version="1.0" standalone="no"?> <!DOCTYPE DASDNA SYSTEM "dasdna.dtd"> <DASDNA> <SEQUENCE id="id" start="start" stop="stop" version="X.XX"> <DNA length="NNNN"> atttcttggcgtaaataagagtctcaatgagactctcagaagaaaattgataaatattat taatgatataataataatcttgttgatccgttctatctccagacgattttcctagtctcc agtcgattttgcgctgaaaatgggatatttaatggaattgtttttgtttttattaataaa taggaataaatttacgaaaatcacaaaattttcaataaaaaacaccaaaaaaaagagaaa aaatgagaaaaatcgacgaaaatcggtataaaatcaaataaaaatagaaggaaaatattc agctcgtaaacccacacgtgcggcacggtttcgtgggcggggcgtctctgccgggaaaat tttgcgtttaaaaactcacatataggcatccaatggattttcggattttaaaaattaata taaaatcagggaaatttttttaaattttttcacatcgatattcggtatcaggggcaaaat tagagtcagaaacatatatttccccacaaactctactccccctttaaacaaagcaaagag cgatactcattgcctgtagcctctatattatgccttatgggaatgcatttgattgtttcc gcatattgtttacaaccatttatacaacatgtgacgtagacgcactgggcggttgtaaaa cctgacagaaagaattggtcccgtcatctactttctgattttttggaaaatatgtacaat gtcgtccagtattctattccttctcggcgatttggccaagttattcaaacacgtataaat aaaaatcaataaagctaggaaaatattttcagccatcacaaagtttcgtcagccttgtta tgtcaaccactttttatacaaattatataaccagaaatactattaaataagtatttgtat gaaacaatgaacactattataacattttcagaaaatgtagtatttaagcgaaggtagtgc acatcaaggccgtcaaacggaaaaatttttgcaagaatca </DNA> </SEQUENCE> </DASDNA>
Scope: Sequence and Annotation servers
In Response to Command: resolve
Format:
<?xml version="1.0" standalone="no"?>
<!DOCTYPE DASRES SYSTEM "dasres.dtd">
<DASRES>
<RESOLVE version="X.XX">
<SEGMENT id="id" start="start" stop="stop">
<RELCOORD ref="id" start="start" stop="stop"/>
</SEGMENT>
</RESOLVE>
</DASRES>
Scope: Annotation servers
In Response to Command: types
Format:
There are two documents formats, the first is a shortened form of the full features format (see below) and is used to summarize the number of annotations of each type available.
<?xml version="1.0" standalone="no"?>
<!DOCTYPE DASTYPES SYSTEM "dastypes.dtd">
<DASTYPES>
<GFF version="1.2" source="$url">
<SEGMENT id="id" start="start" stop="stop" version="X.XX">
<TYPE id="id1" category="category">Type Count 1</TYPE>
<TYPE id="id2" category="category">Type Count 2</TYPE>
...
</SEGMENT>
</GFF>
</DASTYPES>
Scope: Sequence and annotation servers
In Response to Command: features
Format:
The "full" format use to retrieve detailed information on annotations across a segment. This information is summarized through the types query (see above).
<?xml version="1.0" standalone="no"?>
<!DOCTYPE DASGFF SYSTEM "dasgff.dtd">
<DASGFF>
<GFF version="1.2" source="$url">
<SEGMENT id="id" start="start" stop="stop" version="X.XX">
<FEATURE id="id" label="label">
<TYPE id="id" category="category" reference="yes|no">type label</TYPE>
<METHOD id="id">method label</TYPE>
<START>start</START>
<END>end</END>
<SCORE>[X.XX|-]</SCORE>
<ORIENTATION>[0|-|+]</ORIENTATION>
<PHASE>[0|1|2|-]</PHASE>
<GROUP id="hash">
<NOTE>note text</NOTE>
<LINK href="url">link text</LINK>
<TARGET ref="id" start="x" stop="y">target name</TARGET>
</GROUP>
</FEATURE>
...
</SEGMENT>
</GFF>
</DASGFF>
Scope: Annotation servers
In Response to Command: stylesheet
This document is intended to provide hints to the annotation display client. It maps feature categories and individual types to a series of glyphs known to the display client. The complete list of glyphs and their attributes is in preparation.
Format:
<?xml version="1.0" standalone="no"?>
<!DOCTYPE DASSTYLE SYSTEM "dasstyle.dtd">
<DASSTYLE>
<STYLESHEET version="X.XX">
<CATEGORY id="default">
<TYPE id="default">
<GLYPH>
<ID>
<ATTR>value</ATTR>
<ATTR>value</ATTR>
...
</ID>
</GLYPH>
</TYPE>
</CATEGORY>
<CATEGORY id="category1">
<TYPE id="default">
<GLYPH>
<ID>
<ATTR>value</ATTR>
...
</ID>
</GLYPH>
</TYPE>
<TYPE id="type1">
<GLYPH>
<ID>
<ATTR>value</ATTR>
...
</ID>
</GLYPH>
</TYPE>
<TYPE id="type2">
<GLYPH>
<ID>
<ATTR>value</ATTR>
...
</ID>
</GLYPH>
</TYPE>
...
</CATEGORY>
<CATEGORY id="category2">
<TYPE id="default">
<GLYPH>
<ID>
<ATTR>value</ATTR>
...
</ID>
</GLYPH>
</TYPE>
...
</CATEGORY>
...
</STYLESHEET>
</DASSTYLE>
For example:
...
<CATEGORY id="Similarity">
<TYPE id="default">
<GLYPH>
<LINE>
<COLOR>gray</COLOR>
</LINE>
</GLYPH>
</TYPE>
<TYPE id="NN">
<GLYPH>
<BOX>
<COLOR>red</COLOR>
<WIDTH>4</WIDTH>
<OUTLINECOLOR>black</OUTLINECOLOR>
</BOX>
</GLYPH>
</TYPE>
<TYPE id="NP">
<GLYPH>
<TOOMANY>
<COLOR>blue</COLOR>
<WIDTH>4</WIDTH>
<OUTLINECOLOR>black</OUTLINECOLOR>
</TOOMANY>
</GLYPH>
</TYPE>
<TYPE id="PN">
<GLYPH>
<BOX>
<COLOR>green</COLOR>
<WIDTH>3</WIDTH>
<OUTLINECOLOR>blue</OUTLINECOLOR>
</BOX>
</GLYPH>
</TYPE>
<TYPE id="PP">
<GLYPH>
<SPAN>
<COLOR>gray</COLOR>
<WIDTH>4</WIDTH>
</SPAN>
</GLYPH>
</TYPE>
</CATEGORY>
...
This is a list of generic feature categories and specific feature types within them. This list was derived from the features currently exported by ACeDB/GFF and is not (yet) comprehensive. Suggestions for modifications, additions and deletions are welcomed.
The Translated category is used for features that relate to regions of the sequence that are translated into proteins. Features that relate to transcription are separate (see below).
Features:
It is recommended, but not required, that the <GROUP> section contain <LINK> and/or <NOTE> tags that provide further information on the transcription feature.
The Transcribed category is used for features that relate to regions of the sequence that are transcribed into RNA.
Features:
It is recommended, but not required, that the <GROUP> section contain <LINK> and/or <NOTE> tags that provide further information on the transcription feature.
The Variation category is used for features that relate to regions of the sequence that are polymorphic.
Features:
It is recommended, but not required, that the <GROUP> section contain <LINK> and/or <NOTE> tags that provide further information on the variation.
The Structural category is used for features that relate to mapping, sequencing and assembly, as well as for various landmarks that carry no intrinsic biological information.
Features:
It is recommended, but not required, that the <GROUP> section contain <LINK> and/or <NOTE> tags that provide further information on the structural feature.
The Homology category is used for areas that are homologous to other sequences. Homology features should have a <METHOD> tag that indicates the algorithm used for the sequence comparison, and a <TARGET> tag in the <GROUP> field that indicates the target of the match.
Features:
The Repeat category is used for areas that contain repetitive DNA. This category is used both for low-complexity regions, such as microsatellites, and for more biologically interesting features, such as transposon insertion sites.
Features:
It is recommended, but not required, that the <GROUP> section contain <LINK> and/or <NOTE> tags that provide further information on the repetitive element.
The Experimental category is a catchall used to flag areas where there is interesting experimental data of one sort or another. It is intended for use with high-throughput functional genomics work, such as knockouts or insertional mutagenesis screens.
Features:
It is recommended, but not required, that the <GROUP> section contain <LINK> and/or <NOTE> tags that provide further information on the nature of the experimental data.
This section describes a set of generic "glyphs" that can be used by sequence display programs to display the position of features on a sequence map. The annotation server may use these glyphs to send display suggestions to the viewer via the stylesheet document.
The current set of glyph ID values are:
Each glyph has a set of generic attributes associated with it. Attribute values come in the following flavors:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
Attributes:
The distributed annotation system must have a mechanism for detecting and resolving version skew across reference and annotation servers. Although one such mechanism is currently incorporated into the ACeDB-based prototype, it is largely untested and hence not yet a part of the DAS standard.
| Home | C. elegans | AcePerl | Jade | Course | DAS | BoulderIO | WWW | Linux |