Content

Evolution of the Big Data Program

The Big Data Program started out as the Big Data Project in April 2015, when NOAA signed three year Non-competitive Cooperative Research and Development Agreements (CRADAs) with Amazon Web Services, Google Cloud Platform, IBM, Microsoft Azure, and the Open Common Consortium. The CRADA was later extended for an extra year ending May 4, 2019. The CRADA phase has been a success and as a result:

  • Over 120 NOAA datasets have moved to the Collaborators’ systems, including weather radar data, historical weather data, satellite imagery (GOES-16, GOES-17), lightning observations [Geostationary Lightning Mapper (GLM) and Vaisala aggregate], fisheries data, and a variety of numerical model outputs including the National Water Model, Global Forecast System (GFS), Climate Forecast System (CFSv2) and the High Resolution Rapid Refresh (HRRR).

  • Significant increases in data usage were observed, although detailed usage statistics are not a requirement of the CRADA and thus are incomplete.

    • GOES-16 satellite data were accessed from Collaborator platforms at a rate of 10x the incoming data rate.

    • A 130% increase in the utilization of weather radar data over previous years, with a 50% reduction in loads on the NOAA access system, and 80% of archive data orders are now fulfilled on Collaborator platforms.

    • Over 30 times more historical weather data were served through Collaborator platforms (800,000 queries for a total of 1.2 PBs of data) between January-April 2017 than through the NOAA archive.

  • An increasing number of datasets were pulled from existing NOAA internet-based data services by the Collaborators, in consultation with NOAA experts. 

  • The Collaborators have stated that access to NOAA’s expertise has been the most valuable commodity in the Project.

  • Integration of NOAA data into Collaborators’ existing cloud-based access and analytical tools has driven the largest increases in data usage.

Timeline

  • April 2015 - Start of three year CRADA

  • October 2015 - NEXRAD is the first dataset moved under the CRADA

  • May 2018 - One year CRADA extension begins

  • May 2019 - CRADA ends

  • September 2019 - Operational phase begins

  • Oct 2019 - First year of BDP Initial Operating Capability (IOC)