NOAA leverages Globus in proof of concept

Research Scientists at NOAA’s Global Systems Lab are looking five to ten years out on how to apply emerging technologies and HPC to numerical weather prediction in order to prepare NOAA’s modeling infrastructure to use the latest technology to run “the really, really big stuff”, and overcome current capacity challenges. As the resolution of global models goes to cloud-resolving scales of less than four kilometers, and the number of ensemble members that comprise the collections of simulations all running the model over the same period of time to make more accurate weather forecasts exceeds one hundred, the computation will require distributed resources. Several hundred thousand or more cores will be required, and will exceed machine capacity. Some ensemble members will run in one place, while others will need to run elsewhere. Additionally, some model components will likely require specialized resources such as GPUs or storage archives, so they will need to run where those resources are available.

The scientists are currently building and testing out a distributed workflow which leverages Parsl, and extends parallelism in Python across multiple cores and nodes; and uses the Globus Compute service to orchestrate the management of multiple parallel jobs. They plan to wrap all the tasks for building a real numerical weather prediction model in Python, and are testing Globus Compute to run their tasks across diverse remote resources. With Globus Compute, and with the installation of Globus endpoints at multiple locations, users will be able to authenticate through Globus, run different parts of the workflow in different administrative domains or even different organizations, leverage optimally-configured systems to run faster, and have access to increased capacity to solve grand challenge problems.

“With the diversification of resources there is no one place with all the diverse resources that a workflow needs. We need to be able to schedule things onto the resources where they run best. With Globus Auth we get federated identity management. Then with Globus Compute we can register functions on remote systems and run them,” says Chris Harrop, Sr. Associate Scientist at the Cooperative Institute for Research in Environmental Sciences, working within the NOAA Global Systems Laboratory. Chris goes on to explain, “With Parsl and Globus Compute we will be able to run distributed scientific workflows. We will be able to use containers for consistency to run parallel jobs across diverse system architectures, whether they are in the cloud or on-premise.”

Globus Compute allows scientists to easily leverage this large, distributed cyberinfrastructure by implementing a Function-as-a-Service model that enables arbitrarily complex code to run on multiple compute resources, as needed, using a unified interface.