Database resources of the National Center for Biotechnology Information

PubMed

NCBI has continued to update the new PubMed with features and improvements following its launch in May 2020 and the retirement of the legacy site in October 2020. We have added many of the most requested updates, such as a custom date range filter, additional display formats and sort options, MyNCBI highlighting preferences, ‘Related Information’ links on Abstract pages, and more convenient positioning of the ‘Save’, ‘Email’ and ‘Send to’ buttons based on usability testing where we observe volunteers interacting with various design options. In addition, users can now email up to 1000 citations at a time. We also updated the Single Citation Matcher and Batch Citation Matcher to the new PubMed cloud environment and deployed an updated relevance-based algorithm supporting Best Match retrieval that includes feedback from clicked articles (7). This year we reviewed and refined the protocols for updating the model that places articles with the greatest number of clicks for a given query type near the top of the result list. The result increases the likelihood that a searcher will easily find articles of interest.

The rapid surge in biomedical research on the COVID-19 pandemic in 2021 has made it increasingly challenging to navigate and analyze the biomedical literature related to COVID-19. In response, we updated the PubMed Clinical Queries interface following a usability study (similar to that employed with PubMed) to support user-friendly, efficient searching on clinical and disease-related topics. Clinical Queries currently includes filters for Clinical Studies and COVID-19 articles (https://pubmed.ncbi.nlm.nih.gov/clinical/?term=nursing+home&filter_category=covid-19&clinical_study_category=therapy&covid_category=prevention&clinical_study_scope=broad) based on the eight topic categories in LitCovid, a specialized literature hub for tracking SARS-CoV-2 (8,9).

PubMed Central (PMC)

In January 2021, PMC reached a significant milestone in surpassing 7 million journal articles and author manuscripts publicly available in the archive. This milestone is the result of contributions from publishers, journals, authors, and data providers in addition to partnerships with numerous research funding organizations. Two additional projects that launched in 2021 also continued to add content to the archive for public discovery and use. First, the Public Health Emergency COVID-19 Initiative (https://www.ncbi.nlm.nih.gov/pmc/about/covid-19/), a collaboration with more than 50 publishers and societies launched in March 2020, added 185 000 articles in human and machine-readable formats to PMC through September 2021 and continues to enable human and machine-readable access on a global scale to the most current research on COVID-19 and SARS-CoV-2. Second, the NIH Preprint Pilot, which started in June 2020 as a pilot project to include preprints resulting from NIH-funded research in PMC (https://www.ncbi.nlm.nih.gov/pmc/about/nihpreprints/), remained in Phase 1, limiting the scope of the effort to NIH-supported preprints reporting on COVID-19 research. Through September 2021, 2700 preprint records had been identified as in scope for the pilot and made available in PMC and PubMed; just under half of these preprints (48%) had been linked to a peer-reviewed journal article version.

To further support machine-readable access to journal articles, manuscripts, and preprints made available under license terms that allow for text mining and other types of reuses (e.g. Creative Commons licenses), NCBI updated the structure of the PMC FTP directories and added a cloud distribution option. Beginning in July 2021, articles from the PMC Open Access Subset and Author Manuscript Dataset were discoverable in the Registry of Open Data on Amazon Web Services (AWS) in full-text XML and plain text formats. Additionally, in June 2021 NCBI announced the release of PMC Labs (https://www.ncbi.nlm.nih.gov/labs/pmc/), the first phase of a project aimed at creating a more modern and accessible user experience in PMC. Ongoing direct feedback, usability testing, and user research on PMC Labs will guide future improvements to the PMC website.

Bookshelf

The NCBI Bookshelf provides free online access to over 9600 books and documents in life science and healthcare from over 150 content providers. This past year, in collaboration with participating U.S. health agencies, international health organizations, and other sponsors and publishers of evidence-based clinical or public health information, Bookshelf made available a collection of COVID-19 guidelines and reviews (https://www.ncbi.nlm.nih.gov/books/about/covid/) that meet Bookshelf's content selection criteria (https://www.ncbi.nlm.nih.gov/books/about/scientificeval/). The targeted search query (https://www.ncbi.nlm.nih.gov/books?term=%22covid+19%22%5BResource+ Type%5D&cmd=DetailsSearch) is regularly updated with new and updated guidelines and reviews.

RefSeq

The RefSeq prokaryotic collection has grown from 198 640 genomes in August 2020 to 223 542 as of 6 August 2021 (10). Because this collection is large and unevenly distributed across the taxonomic tree, we also provide a representative set, containing exactly one genome per species, available as both nucleotide and protein BLAST databases. The number of species in that set has grown from 11 735 in August 2020 to 14 606, reflecting the increase in the diversity of the RefSeq collection.

We recently updated the Prokaryotic Genome Annotation Pipeline (PGAP) that is used to annotate all RefSeq genomes (except for 15 reference genomes). The update addresses limitations on comparative analyses caused by a lack of common gene identifiers across genomes. This new module in PGAP is similar to one used in the Eukaryotic Genome Annotation Pipeline (11); it assigns gene symbols (e.g. recA) to annotated genes using an orthology calculation between the PGAP-annotated and the reference genome for five common species. After reannotation of all RefSeq genomes for these five species in scope, the percentage increase in the number of gene symbols ranged from 25% to 110%. Now, 73% of PGAP-annotated Escherichia coli genes and 79% of Bacillus subtilis genes have symbols (35% for Mycobacterium tuberculosis, 40% for Acinetobacter pittii and 46% for Campylobacter jejuni). PGAP is available as a stand-alone package, and as a service available to GenBank submitters (https://www.ncbi.nlm.nih.gov/genome/annotation_prok/).

The RefSeq eukaryotic collection has now grown to 1290 genomes from 1272 species as of 6 August 2021, including 71 species updated with new genomes in the last 12 months. Genomes from 780 species are now annotated with NCBI’s Eukaryotic Genome Annotation Pipeline, including all vertebrates and most other multicellular eukaryotes, with most models completely based on RNA-seq and/or protein alignment evidence. We have added reporting of BUSCO statistics (12) computed on the longest protein from each gene. BUSCO provides an assessment of the completeness of the annotated gene set based on single-copy orthologs. These statistics are available in the annotation reports for recent annotation releases and for over 800 RefSeq genomes in data returns from E-utility calls to NCBI Assembly. Over half of the RefSeq eukaryote annotations produced by NCBI score over 98% complete by BUSCO using the most specific BUSCO lineage available for each species. One-to-one orthologs for fish species are now computed using the zebrafish genome as a reference, and orthologs for insects compared to the Drosophila melanogaster genome are available on the NCBI Gene FTP site. The collection of curated, well-supported, protein-coding transcripts (NM_ accessions) for human has grown to over 62 000 transcripts, of which 99.5% exactly match the GRCh38 genome and 87% have been revised based on CAGE and polyA-seq data. In addition to quarterly updates to the RefSeq GRCh38.p13 genome annotation, we provide yearly updates for the prior GRCh37.p13 genome to support clinical users that have not yet updated to the current reference assembly. To help properly annotate SNPs on RefSeq transcripts that have sequence differences relative to the genome, we have added transcript-to-genome alignments in BAM format with each annotation release on FTP. Finally, the data for a collection of human genes involved in coronavirus infection and disease have been revised and are available in NCBI gene with a query for ‘coronavirus related’[filter].

Sequence read archive

NCBI maintains the NIH Sequence Read Archive (SRA), an archival database designed to support storage, retrieval, and analysis of next-generation nucleotide sequence data. A single copy of the archive now includes 11.5 Petabytes of publicly available data and another 4.9PB of controlled-access dbGaP data. This is one of the largest publicly available, biological data repositories, and while the archive holds tremendous promise for biomedical research, the sheer size of this dataset makes it difficult to store, retrieve, and analyze. NCBI is currently implementing more efficient approaches to data storage and improving data accessibility. As part of the NIH Science and Technology Research Infrastructure for Discovery, Experimentation and Sustainability (STRIDES) Initiative (https://datascience.nih.gov/strides/), NCBI is now maintaining the entire SRA on two commercial cloud platforms, AWS and Google Cloud Platform (GCP) (https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud/). Moving SRA to the cloud provides several benefits: long-term sustainability and scalability, increased download speeds, unlimited concurrent downloads, and access to the originally submitted data files. To support SRA sequence data selection and analysis, SRA run metadata, BioProject data, and BioSample data are available through GCP BigQuery and AWS Athena to support selection and analysis of SRA sequence data (https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud-based-examples/). The metadata available in BigQuery and Athena includes results from the k-mer-based Sequence Taxonomic Analysis Tool that describe the organismal content present in samples (13).

Both originally submitted and normalized SRA files are available through a combination of ‘hot’ and ‘cold’ storage tiers on both AWS and GCP cloud platforms. The SRA toolkit can be used to retrieve data from hot cloud storage to local and cloud locations, and for cloud-based users, will pull data from the appropriate commercial cloud platform to user generated buckets (https://github.com/ncbi/sra-tools/wiki/02.-Installing-SRA-Toolkit). A web-based interface can also be used to retrieve hot and/or cold stored data and move it to user-created cloud buckets (https://www.ncbi.nlm.nih.gov/Traces/cloud-delivery/). More information about selecting and retrieving SRA data from cloud platforms can be found at https://www.ncbi.nlm.nih.gov/sra/docs/data-delivery/.

NCBI remains committed to continuing to expand SRA and improving access to the archive based on FAIR data principles. As part of this effort, normalized SRA files are also available through the AWS Open Data Program (ODP) (https://registry.opendata.aws/ncbi-sra/) where they can be egressed or downloaded for free by both cloud and non-cloud users. This program supports enhanced access to a variety of cloud-based and non-cloud data users, and several tools can be used to download and/or egress based on specific use cases, including the SRA toolkit and tools maintained by AWS.

Virus

In response to the COVID-19 pandemic, NCBI has provided enhanced access to >700 000 SARS-CoV-2 SRA samples through the AWS ODP and the Google Public Dataset Program. These samples are identified using STAT organismal content analysis (13) and user-provided sample descriptions. We provide several data files for each sample on the cloud platforms, including originally formatted sequence files submitted to SRA, normalized format SRA sequence files, and SRA aligned read files where submitted reads are aligned to contigs assembled from runs using the SARS-CoV-2 RefSeq guided assembly. We annotate these contigs with VIGOR3 (14), then translate and include them in BLASTn and BLASTp databases available in the AWS ODP. Nucleotide variations relative to the RefSeq reference (NC_045512.2) are also calculated for each run and made available in variant call format (VCF) files (14) and SPDI (Sequence, Position, Deletion, Insertion) format (15). Finally, SRA, BioSample and BioProject metadata and analysis data calculated by NCBI are available through BigQuery and Athena tables (https://www.ncbi.nlm.nih.gov/sra/docs/aligned-metadata-tables/).

dbSNP

The Database of Single Nucleotide Polymorphisms (dbSNP) is a repository of human genomic variations and frequency data that includes both common and rare single-nucleotide variations and other small-scale variations. In 2021, dbSNP reached a milestone of over 1 billion RS numbers with build release 155 (June 2021). This dbSNP release included the NCBI Allele Frequency Aggregator (ALFA) data (https://www.ncbi.nlm.nih.gov/snp/docs/gsr/alfa/) with over 900 million variants and allele frequencies for 12 populations from 192 000 dbGaP subjects. We improved the Entrez SNP search to support queries using HGVS, protein position, SPDI, LitVar, and ALFA attributes. We also redesigned the RefSNP page to include more intuitive horizontal tabs for variant information including ClinVar clinical significance, allele frequency, and publications (e.g. https://www.ncbi.nlm.nih.gov/snp/rs732609?horizontal_tab=true). In addition, we added a function to retrieve RefSNP flank sequences with a user-specified length.

dbVar

In the past year dbVar added 26 new human studies for structural variation (SV) that include 1 221 062 SV regions and 1 241 943 SV calls, bringing the total in the June 2021 release to 207 studies, 6 964 796 SV regions, and 37 081 907 calls. Included in dbVar are structural variants from ClinVar (nstd102), gnomAD from 15 000 genomes (nstd106), Genome in a Bottle (nstd175), 1000 Genomes 30X (nstd206), NCBI Curated Common Structural Variants with frequency (nstd186), and other population studies (nstd200 and nstd207). dbVar now had improved displays of selected structural variation datasets, including somatic and presumed normal SV, available on TrackHub, the NCBI Genome Data Viewer, the UCSC genome browser, and other genomic browsers. Additionally, we improved mouse-over tips, track colors, and links to ClinVar.

ClinicalTrials.gov

Launched in 2000, ClinicalTrials.gov serves as an essential, transparent, and trusted part of the research ecosystem by providing patients, healthcare professionals, and researchers access to information on ongoing and completed clinical research (16). Study information is provided, voluntarily or to meet legal and policy requirements (17), by the sponsor or investigator responsible for the research. As of August 2021, the ClinicalTrials.gov registry listed over 384 000 clinical studies from all 50 U.S. states and 220 countries, and the ClinicalTrials.gov results database reached a milestone of listing summary results information for over 50 000 clinical studies, about half of which have not been published in the biomedical literature.

ClinicalTrials.gov is designed to support several use cases. These include helping potential participants discover recruiting studies, allowing researchers to identify gaps and redundancies across the clinical research enterprise, aiding consumers of evidence-based medicine by reducing publication bias and preventing selective publication of outcomes, and enabling journal editors to evaluate results reported in manuscripts against prespecified research plans. To advance these several uses further, NLM launched a multi-year effort in October 2019 to modernize ClinicalTrials.gov by delivering an improved user experience on a cloud-based platform that will accommodate growth and enhance efficiency (https://clinicaltrials.gov/ct2/about-site/modernization/). Initial work has focused on technical infrastructure enhancements and obtaining broad and focused user input to develop prototypes using an agile development process with iterative design based on user testing.

In response to the COVID-19 public health emergency, ClinicalTrials.gov has expedited the processing of COVID-19 related study submissions to facilitate discovery and access. As of August 2021, over 6200 studies related to COVID-19 were listed (https://clinicaltrials.gov/ct2/results?cond=COVID-19) and another nearly 4700 COVID-19 studies from the WHO registry search portal (https://clinicaltrials.gov/ct2/who_table) were available from ClinicalTrials.gov. In November 2020, the NIH Director issued a statement calling on researchers to swiftly share COVID-19 clinical research information by registering and submitting results to ClinicalTrials.gov as quickly as possible, ahead of legal and policy deadlines (https://www.nih.gov/about-nih/who-we-are/nih-director/statements/nih-calls-clinical-researchers-swiftly-share-covid-19-results).

MMDB

NCBI has upgraded both the MMDB (Molecular Modeling Database) and GenBank sequence record tools so that they can process the complete set of Protein Data Bank (PDB) molecular structure data. With advances in structure determination and the ability to process much bigger molecular assemblies, the PDB has been releasing larger structures only in the more flexible mmCIF-format (https://mmcif.wwpdb.org/). The latter allows for longer ‘chain identifiers’, as opposed to single alphanumeric characters, which are necessary to accommodate very large structural assemblies with many different polypeptide chains. Whereas previously all protein and nucleotide sequences derived from PDB structures could be retrieved using accessions like ‘1ABC_X’, instances such as ‘1ABC_XX’ or ‘1ABC_WXYZ’, or ‘1ABC_abc’ can now be found. As of this writing there are about 1200 such structures containing long chain identifiers, and corresponding entries have been added to the sequence databases; the MMDB structure database and sequence databases reflect the complete set of all the structures in the Protein Data Bank and we continue to update them weekly.

iCn3D

With the release of iCn3D 3.0, users can perform three-dimensional structure analyses on lists of structures using Node.js scripts, based on the npm icn3d package (https://www.npmjs.com/package/icn3d). Some example scripts are provided at https://github.com/ncbi/icn3d/tree/master/icn3dnode and demonstrate how to analyze protein-ligand interactions, protein-protein interactions, and change of interactions due to residue mutations (18). Additional examples demonstrate how to compute electrostatic potentials using DelPhi (19), solvent accessible surface areas, and more. Users can easily generate their own Node.js scripts based on the functions in iCn3D: https://www.ncbi.nlm.nih.gov/Structure/icn3d/icn3d.html#classstructure. iCn3D also allows users to align multiple chains from different structures to a reference chain and visualize the corresponding super-impositions, to calculate the intramolecular symmetry dynamically (20), and visualize structures as 2D cartoons at the level of chains, protein domains, and protein secondary structure.