Weather: Big Data&Big Forecast Progress

Weather Channel Article

Behind the Scenes: How We Make Billions of Forecasts 96 Times a Day

David Kenny
Chairman & CEO at The Weather Company

In the past several months, we have improved the accuracy of precipitation forecasts by another 15% and the accuracy of temperature forecasts by 2 degrees Fahrenheit. This is entirely because we are now forecasting for every location on Earth (at least 3 billion named locations) instead of mapping people to the approximately 2MM locations where forecasts have traditionally been done.

….mobile access is always current. So this means that our current day forecasts are now updated 96 times per day, and days 2-15 are updated at least 24 times per day.

….Our new forecasting platform is the largest application in the Amazon Web Services cloud. We are currently scaled to deliver 115,000 unique forecasts per second, or more than 10 Billion forecasts per day.

….We are voracious consumers of data to make this happen. We build on the incredibly skillful global models of government agencies, such as NOAA and ECMWF, as well as models built by some of the world’s great universities and research organizations. We add the data we collect from airplanes (who provide that data as part of our aviation weather service contracts with most airlines and private operators.) We also work with tens of thousands of individuals who buy and build personal weather stations, connect those stations to our Weather Underground network and send us their own weather data every 2.5 seconds. And we are exploring new potential observations from home sensors, windshield wipers, smartphone air pressure gages, and literally billions of additional sources that can sustain this faster rate of innovation and improved forecasting skill.

….Two years ago, we began creating this new forecasting system. The bulk of the work was building the statistical optimization to calibrate and blend the world’s best global weather models. And then we supported these algorithms with more and faster computing power, as well as more observations, to enable constant machine learning. At the same time, machine learning and algorithmic forecasts sometimes miss enough color for people to truly know what the forecast means, and how to act. So we invented an exception process when our human meteorologists and social scientists need to interpret and verbalize certain forecasts, especially when life and property-threatening storm systems are predicted.

Weather Channel Dashboard