Contact
One of our experts will contact you as soon as possible to discuss your needs and organize a tailor-made demo of the platform.
Ever since the first numerical forecast in the 1950s, numerical weather prediction (NWP) has been improving steadily. Through the development of more powerful computers, NWP models can now predict the weather for the next 5 days with reliable accuracy. Let’s see how it has evolved from physics to numerical computation, and how you can perform your own weather simulations on Qarnot.
Most processes - whether in agriculture, transportation or energy - are affected by the weather. Therefore, knowing its state in the near future or even in the longer term, can come in handy. So how come to accurately predict the atmospheric phenomena that constantly occurs around us?
The Navier-Stokes equations have been around since the mid-19th century to express the physical principles of conservation of mass, momentum and energy in the context of viscous fluids. They can be used to model ocean currents, airflow around a wing and, in our case, weather.
The only drawback is that even though these partial differential equations are valid at any point in space and time, they are not solvable as such.
The production of realistic forecast as we know did not begin until the 20th century, when scientists showed that these equations could be approximated at a finite number of points in space. With the development of the first computers in the 1950s, numerical forecast was made possible, under considerable approximations.
Nowadays, with advances in computing power and the development of supercomputers, more accurate models are constantly being created.
Even with less need for approximations, some still remain necessary to reduce the computation of weather forecasts.
To give you an idea, a 16-day forecast made by the Global Forecast System models requires about 5 hours of execution on the National Oceanic and Atmospheric Administration’s supercomputer.
The hydrostatic approximation, for instance, is still used in most modern models (except those specialized in storms, like hurricane models). This approximation assumes that vertical motion is much smaller than the horizontal one, which makes the calculation of the pressure force to be performed as if it were in equilibrium with the gravitational force.
Under this assumption, the equation system used in NWP models is composed of the primitive equations, which consist of a thermal energy equation and a simplified version of the Navier-Stokes equations, coupled with the ideal gas law. This system can then be solved in grid points models with the finite difference method, in all three spatial dimensions.
Solutions of the equations mentioned above can be computed for a finite number of points, leaving the step of determination of the points at which the solutions will be calculated. This is usually done by discretizing the region of interest into small cells, the totality of which forms what is called a mesh.
Different meshes are adapted to different scenarios, and choosing the type and size of the mesh is essential as it will affect precision, errors and computational power requirements.
While most grid points models use a latitude-longitude grid as a mesh, with a resolution determined by the size of a cell, some use more sophisticated ones, to include variable resolution for example. That is the case with Voronoï meshes: they partition space into regions close to each point of interest (POI), creating an irregular or regular mesh depending on the POI chosen. This principle is convenient for large domains that cover several terrestrial topographies, such as mountains, that involve smaller physics phenomena. It can also be useful for localized regional forecasts, such as a country that wants to run a weather simulation over the entire globe with a finer precision over its ground.
Although we chose to focus on NWP models here, the entire weather prediction pipeline that happens before we can see tomorrow’s forecast on your television screen is way more sophisticated.
Firstly, observations of the state of the atmosphere are retrieved by satellites, weather stations and other measuring devices, before being processed to fit the forecast models.
A model is then run on powerful computers with this data and produces outputs that are post-processed to correct for model bias and create variables of interest, such as daily precipitation and wind speed.
Computational resources might also be necessary for other steps such as data assimilation, but since NWP models use the finite difference method, they are definitely the most power-hungry component of all.
In order to illustrate what we have learned so far, We are now going to verify the accuracy achieved by a forecast made with WRF, a popular open source model first developed by the National Center for Atmospheric Research (NCAR) and run it on Qarnot.
Before launching the case, please ensure that the following prerequisites have been met.
For our test case we are going to use the Weather Research and Forecasting (WRF) model , a mesoscale model able to perform high-resolution simulations. Its execution consists of two components: the WRF Preprocessing System (WPS) and the WRF model.
The WPS package processes terrestrial and meteorological data in order to create horizontal initial and boundary conditions. It supports the following three programs:
The WRF model can then be run through:
Create a WRF folder, and inside it an input folder. In the input folder, you will include all the files needed for the computation:
DATA folder: folder with the initial grib files needed by the ungrib program. You can download GFS outputs from 2020, or you can download your own data and modify the following two configuration files accordingly.namelist.input: configuration file, which defines the parameters setup for the WRF package execution. You can find the configuration file corresponding to the above shared data, modify it or create your own.namelist.wps: file which defines the parameters setup for WPS. A sample configuration file is provided if needed.running_script.sh : bash script with all commands that you would write were you to locally execute all of the above programs one after the other, from setting up the WPS environment to the execution of the file wrf.exe. Save the following script in your input folder - it could also be adapted to your simulation if you see it fit.## running_script.sh
<details><summary>Click to see the code</summary>
<p>
```
#!/bin/bash
# move files uploaded in the bucket to their respective folders for execution
echo "Moving input files..."
mv /job/DATA/ /opt/Build_WRF/
mv /job/namelist.wps /opt/WPS
mv /job/namelist.input /opt/WRF/test/em_real
### WPS ###
cd /opt/WPS/
# WPS output folder
mkdir /job/WPS
# GEOGRID
echo "Running geogrid.exe..."
./geogrid.exe 2>&1 | tee /job/WPS/geogrid.out
# UNGRIB with files in DATA
echo "Running ungrib.exe..."
./link_grib.csh /opt/Build_WRF/DATA/GFS_
ln -sf ungrib/Variable_Tables/Vtable.GFS Vtable
./ungrib.exe 2>&1 | tee /job/WPS/ungrib.out
# METGRID
echo "Running metgrid.exe..."
./metgrid.exe 2>&1 | tee /job/WPS/metgrid.out
# get metgrid levels
MET_FILE=$(ls met_em* | head -n 1)
TMP_LEVELS=$(ncdump -h $MET_FILE | grep -i "num_metgrid_levels =" | cut -d' ' -f3)
TMP_SOIL_LEVELS=$(ncdump -h $MET_FILE | grep -i num_metgrid_soil_levels | cut -d' ' -f3)
# copy files automatically generated in the executables folder to the job folder
cp --update /opt/WPS/FILE* /job/WPS/
cp --update /opt/WPS/geo_* /job/WPS/
cp --update /opt/WPS/met_em* /job/WPS/
cp --update /opt/WPS/*.log /job/WPS/
### WRF ###
cd /opt/WRF/test/em_real/
# WRF output folders
mkdir -p /job/WRF/real
# set metgrid levels
LINE_LEVELS=$(grep "num_metgrid_levels" namelist.input)
sed -i "s|$LINE_LEVELS| num_metgrid_levels = $TMP_LEVELS,|g" namelist.input
LINE_SOIL_LEVELS=$(grep "num_metgrid_soil_levels" namelist.input)
sed -i "s|$LINE_SOIL_LEVELS| num_metgrid_soil_levels = $TMP_SOIL_LEVELS,|g" namelist.input
# REAL
echo "Running real.exe..."
ln -sf /job/WPS/met_em* .
mpirun --allow-run-as-root ./real.exe 2>&1 | tee /job/WRF/real.out
cp --update /opt/WRF/test/em_real/rsl* /job/WRF/real/
# WRF
echo "Running wrf.exe..."
mpirun --allow-run-as-root ./wrf.exe 2>&1 | tee /job/WRF/wrf.out
echo "Done running!"
# copy files automatically generated in the executables folder to the job folder
cp --update /opt/WRF/test/em_real/wrfbdy* /job/WRF/
cp --update /opt/WRF/test/em_real/wrfinput* /job/WRF/
cp --update /opt/WRF/test/em_real/rsl* /job/WRF/
cp --update /opt/WRF/test/em_real/wrfout* /job/WRF/
# if restart interval not null
cp --update /opt/WRF/test/em_real/wrfrst* /job/WRF/
```
</p>
</details>Now that you have all the necessary elements for the simulation, let’s use the Qarnot Python SDK to launch the calculation. You can save the following python script under run.py in the WRF folder.
Be sure you have copied your authentication token in the script (instead of a<<<put your secret token here>>>) to be able to launch the task on Qarnot.
## wrf_sdk_qarnot.py
<details><summary>Click to see the code</summary>
<p>
```
#!/usr/bin/env python
import sys
import qarnot
import os
# Edit 'samples.conf' to provide your own credentials
# Create a connection, from which all other objects will be derived
conn = qarnot.Connection(client_token="<<<put your secret token here>>>")
# Create a task
task = conn.create_task('wrf_demo', 'docker-batch', 1)
# Store if an error happened during the process
error_happened = False
try:
# Create a resource bucket and add input files
input_bucket = conn.create_bucket('wrf-files')
input_bucket.sync_directory('input')
# Attach the bucket to the task
task.resources.append(input_bucket)
# Create a result bucket and attach it to the task
output_bucket = conn.create_bucket('wrf-files-output')
task.results = output_bucket
# Set the command to run when launching the container, by overriding a
# constant.
# Task constants are the main way of controlling a task's behaviour
task.constants['DOCKER_REPO'] = "qarnotlab/wrf"
task.constants['DOCKER_TAG'] = "v1"
task.constants['DOCKER_CMD'] = './running_script.sh'
# Update results every 5 seconds
task.snapshot(5)
# Submit the task to the Api, that will launch it on the cluster
task.submit()
# Wait for the task to be finished, and monitor the progress of its
# deployment
last_state = ''
done = False
while not done:
# Update task state changes
if task.state != last_state:
last_state = task.state
print("** {}".format(last_state))
# Wait for the task to complete, with a timeout of 2 seconds.
# This will return True as soon as the task is complete, or False
# after the timeout.
done = task.wait(2)
# Display fresh stdout / stderr
sys.stdout.write(task.fresh_stdout())
sys.stderr.write(task.fresh_stderr())
# Display errors on failure
if task.state == 'Failure':
print("** Errors: %s" % task.errors[0])
error_happened = True
else:
# Download results from output_bucket into given folder
task.download_results('output')
finally:
# Delete the task
# task.delete(purge_resources=True, purge_results=True)
# Exit code in case of error
if error_happened:
sys.exit(1)
```
</p>
</details>The final tree that you obtain after downloading or creating all the above files should resemble this:
WRFinputDATAGFS_......GFS_...namelist.inputnamelist.wpsrunning_script.shwrf_sdk_qarnot.pyThe only thing left to do is to follow these steps to set up a Python virtual environment in the WRF folder, and then run the Python script by typing python run.py in a terminal.
You can then view the tasks details on your own console or on the Qarnot console by clicking on your task. Once the task is completed, the results (i.e. the created files, including final output and restart files, and logs) will be downloaded to your computer.
If you have run the above simulation with the data and scripts provided, you should obtain one output file wrfout_d01_2020-09-10_00:00:00 containing 36-hour predictions for numerous variables. You can ;use any visualization software - here we used , adapted for a simple first glance at the data - to display the simulation results
By comparing this prediction with observed data (from the Meteociel archives) at the time of the simulation, we obtain two temperature animations showing the same variations, thus validating the forecast.
Like most atmospheric models, WRF is a modular system and can therefore be coupled with other models such as ocean or land surface ones. Coupled models allow finer simulations for specific applications, among which the modeling of climate, wildfires, tropical cyclones, air quality, solar energy or crops.
WRF actually integrates since version 4 an extension, WRF-Chem, which couples the WRF model with chemistry to simulate the emission, transport, mixing, and chemical transformations of trace gases and aerosols simultaneously with the meteorology at a regional-scale. We have tested on Qarnot a dust emission simulation in the Mediterranean region in 2014, along with other tutorials that enable users to get started with this module. We obtain the following forecasts for dust emissions from all radii over the desired period.
Although many other models than the one presented here exist, with different equations, approximations, parameters or even applications, for instance for climate forecasts, they all have common limitations.
Their simulation resolution will always be constrained by the computational power available, even if progress is always made in this domain. Moreover, the wide range of influences of complex processes and the increasing number of observations and data available render the weather forecasting task ever more challenging.
One of the latest trends in this domain is the possible application of machine learning algorithms to numerical weather prediction and climate forecasts. While it is already integrated into some prediction and post-processing methods and much research is being conducted, it also faces its own challenges, such as instability, reproducibility and interpretability.
It remains to be seen where the research will lead.
We hope you enjoyed this tutorial! Should you have any questions, please contact qlab@qarnot.com and we will help you with pleasure!
