Templates demonstration#

The previous version of numpy used cookiecutter to contruct model execution environment from vanilla cookiecutter setup of a template that was rendered from a cookiecutter.json. This has the advantage of leveraging a lot of cookiecutter features out of the box, but caused some confusion with regards to how configuration was managed. The cookiecutter.json served the purpose as a config file for model configuration, as well as holding the state of the prodecually run wrapper.

For this new iteration, we have retained the internal use of cookiecutter in an effort to retain some of the features is provides, but have abstracted it away from the user. All coniguration is now passed an model run instantiation time, all through standard arguments. These arguments then intiate pydantic objects that are used to to things like retrieving and interpolating data inputs if required and then used in a cookiecutter template to render the template. This has been deliberately set up to be general, and flexible, allowing users to either invest in writing high level objects that are quite smart, or to just use the templating engine at a lower level to provide a light python framework around existing configurations.

This notebook will highlight some of the aspects of how this has been set up.

[4]:

# Helper functions to dump the contents of input and template
from pathlib import Path
def dump_input(model):
    input_file = Path(model.output_dir) / model.run_id / "INPUT"
    print(input_file.read_text())

def dump_template(model):
    template_file = list(Path(model.config.template).glob("*"))[0] / "INPUT"
    print(template_file.read_text())

[5]:

# We will start here with the basic example from the demo.ipynb notebook that you have hopefully already run.
# Here we will just initialise the model using the config file and run it
from rompy.swan import SwanConfig
from rompy import ModelRun
import yaml
args = yaml.load(open('demo.yml', 'r'), Loader=yaml.FullLoader)
run = ModelRun(**args)

# and then calling as before
run()

INFO:rompy.model:
INFO:rompy.model:-----------------------------------------------------
INFO:rompy.model:Model settings:
INFO:rompy.model:
period:

        Start: 2023-01-01 00:00:00
        End: 2023-01-04 00:00:00
        Duration: 3 days, 0:00:00
        Interval: 1:00:00
        Include End: True


output_dir:
simulations

config:
grid:
        SwanGrid: REG, 390x150
spectral_resolution:
        fmin=0.0464 fmax=1.0 nfreqs=31 ndirs=36
forcing:
        bottom: DatasetXarray(uri=simulations/test_swantemplate/datasets/bathy.nc
        wind: DatasetXarray(uri=simulations/test_swantemplate/datasets/wind_inputs.nc
        boundary: DatasetXarray(uri=../tests/data/aus-20230101.nc

physics:
        friction='MAD' friction_coeff=0.1
outputs:
        Grid:
        variables: DEPTH UBOT HSIGN HSWELL DIR TPS TM01 WIND
        Spec
                locations:

template:
        /source/rompy/rompy/templates/swan

INFO:rompy.model:-----------------------------------------------------
INFO:rompy.model:Generating model input files in simulations
INFO:rompy.swan.config:  Processing bottom forcing
INFO:rompy.swan.data:   Writing bottom to simulations/test_swantemplate/bottom.grd
INFO:rompy.swan.config:  Processing wind forcing
INFO:rompy.swan.data:   Writing wind to simulations/test_swantemplate/wind.grd
INFO:rompy.swan.config:  Processing boundary forcing
INFO:rompy.model:
INFO:rompy.model:Successfully generated project in simulations
INFO:rompy.model:-----------------------------------------------------

[5]:

'/source/rompy/notebooks/simulations/test_swantemplate'

[7]:

# lets have a look at the contents of the template
run.config.template

[7]:

'/source/rompy/rompy/templates/swan'

[8]:

# There are a few things there, but lets begin with the model control file
dump_template(run)

$
$ SWAN - Simple example template used by rompy
$ Template: {{_template}}
$ Generated: {{runtime._generated_at}} on {{runtime._generated_on}} by {{runtime._generated_by}}
$ projection: wgs84
$

MODE NONSTATIONARY TWODIMENSIONAL
COORDINATES SPHERICAL
SET NAUTICAL

{{config.grid}}
{{config.forcing['forcing']}}
{{config.physics}}
{{config.forcing['boundary']}}
{{config.outputs}}
COMPUTE NONST {{runtime.period.start.strftime(runtime._datefmt)}} {{runtime.frequency}} {{runtime.period.end.strftime(runtime._datefmt)}}

STOP

[9]:

# Of note here is that there are two namespaces, runtime, and config. To explore the differneces, lets look at the generate method of the run
run.generate??

Signature: run.generate() -> str
Source:
    def generate(self) -> str:
        """Generate the model input files

        returns
        -------
        staging_dir : str
        """
        logger.info("")
        logger.info("-----------------------------------------------------")
        logger.info("Model settings:")
        logger.info(self)
        logger.info("-----------------------------------------------------")
        logger.info(f"Generating model input files in {self.output_dir}")

        cc_full = {}
        cc_full["runtime"] = self.dict()
        cc_full["runtime"].update(self._generation_medatadata)
        cc_full["runtime"].update({"_datefmt": self._datefmt})
        # TODO calculate from period
        cc_full["runtime"]["frequency"] = "0.25 HR"

        if callable(self.config):
            cc_full["config"] = self.config(self)
        else:
            cc_full["config"] = self.config

        staging_dir = render(
            cc_full, self.config.template, self.output_dir, self.config.checkout
        )

        logger.info("")
        logger.info(f"Successfully generated project in {self.output_dir}")
        logger.info("-----------------------------------------------------")
        return staging_dir
File:      /source/rompy/rompy/model.py
Type:      method

[10]:

# The cc_full dictionary is the full context used to render the template. The runtime values are populated from the ModelRun object, while the
# config values are populated from either the config input. This separation between a model configuration and how it is run means that this config
# can be implemneted at varying degrees of complexity.

# Lets look at teh current swan implementation as an example. This object is callable, so lets look at the __call__ function to see what it is doing
run.config.__call__??

Signature: run.config.__call__(runtime) -> str
Docstring: Call self as a function.
Source:
    def __call__(self, runtime) -> str:
        ret = {}
        if not self.outputs.grid.period:
            self.outputs.grid.period = runtime.period
        if not self.outputs.spec.period:
            self.outputs.spec.period = runtime.period
        ret["grid"] = f"{self.domain}"
        ret["forcing"] = self.forcing.get(
            self.grid, runtime.period, runtime.staging_dir
        )
        ret["physics"] = f"{self.physics.cmd}"
        ret["outputs"] = self.outputs.cmd
        ret["output_locs"] = self.outputs.spec.locations
        return ret
File:      /source/rompy/rompy/swan/config.py
Type:      method

[11]:

# So it is calling the various pydantic components that make up the config which do any work required and return the
# relavent values to fill in the template. We have looked at these in a bit more detail in the demo.ipynb notebook, but we can see
# populated dictionary by calling the config object directly
run.config(runtime=run)

INFO:rompy.swan.config:  Processing bottom forcing
INFO:rompy.swan.data:   Writing bottom to simulations/test_swantemplate/bottom.grd
INFO:rompy.swan.config:  Processing wind forcing
INFO:rompy.swan.data:   Writing wind to simulations/test_swantemplate/wind.grd
INFO:rompy.swan.config:  Processing boundary forcing

[11]:

{'grid': 'CGRID REG 115.68 -32.76 77.0 0.39 0.15 389 149 CIRCLE 36 0.0464 1.0 31\n\n',
 'forcing': {'forcing': "INPGRID BOTTOM REG 115.43844490282329 -32.85000000000018 0.0 42 60 0.010000000000005116 0.00999999999999801 EXC -99.0\nREADINP BOTTOM 1.0 'bottom.grd' 3 FREE\n\nINPGRID WIND REG 115.68 -32.76 77.0 389 149 0.001 0.001 NONSTATION 20230101.000000 24.0 HR\nREADINP WIND 1 'wind.grd' 3 0 1 0 FREE\n",
  'boundary': "BOUNDNEST1 NEST 'bnd.bnd' CLOSED"},
 'physics': 'GEN3 WESTH 0.000075 0.00175\nBREAKING\nFRICTION MAD 0.1\n\nTRIADS\n\nPROP BSBT\nNUM ACCUR 0.02 0.02 0.02 95 NONSTAT 20\n',
 'outputs': "OUTPUT OPTIONS BLOCK 8\nBLOCK 'COMPGRID' HEADER 'outputs/swan_out.nc' LAYOUT 1 DEPTH UBOT HSIGN HSWELL DIR TPS TM01 WIND OUT 20230101.000000 1.0 HR\n\nPOINTs 'pts' FILE 'out.loc'\nSPECout 'pts' SPEC2D ABS 'outputs/spec_out.nc' OUTPUT 20230101.000000 1.0 HR\nTABle 'pts' HEADer 'outputs/tab_out.nc' TIME XP YP HS TPS TM01 DIR DSPR WIND OUTPUT 20230101.000000 1.0 HR\n",
 'output_locs': OutputLocs}

[12]:

# Now this is an example of a medium complexity config object. It is made up of a number of sub componponents grouped into components such as
# grid, data, physics etc. Howeever the config and template can be both simpler, and more complex.

# As an example. lets consider a simpler case. The repo contains a similar configuration, where much of hte configuration is hard coded.

!cat ../rompy/templates/swanbasic/\{\{runtime.run_id\}\}/INPUT

$
$ SWAN - Simple example template used by rompy
$ Template: {{_template}}
$ Generated: {{runtime._generated_at}} on {{runtime._generated_on}} by {{runtime._generated_by}}
$ projection: wgs84
$

MODE NONSTATIONARY TWODIMENSIONAL
COORDINATES SPHERICAL
SET NAUTICAL

CGRID REG 115.68 -32.76 77.0 0.39 0.15 389 149 CIRCLE 36 0.0464 1.0 31


INPGRID BOTTOM REG 115.68 -32.76 0.0 187 150 0.0009999999999999872 0.0009999999999999905 EXC -99.0
READINP BOTTOM 1.0 'bottom.grd' 3 FREE

INPGRID WIND REG 115.68 -32.76 77.0 389 149 0.001 0.001 NONSTATION {{runtime.period.start.strftime(runtime._datefmt)}} {{runtime.frequency}}
READINP WIND 1 'wind.grd' 3 0 1 0 FREE

GEN3 WESTH 0.000075 0.00175
BREAKING
FRICTION MAD {{config.friction_coefficient}}

TRIADS

PROP BSBT
NUM ACCUR 0.02 0.02 0.02 95 NONSTAT 20

BOUNDNEST1 NEST 'boundary.bnd' CLOSED
OUTPUT OPTIONS BLOCK 8
BLOCK 'COMPGRID' HEADER 'outputs/swan_out.nc' LAYOUT 1 DEPTH UBOT HSIGN HSWELL DIR TPS TM01 WIND OUT 20200221.040000 1.0 HR

POINTs 'pts' FILE 'out.loc'
SPECout 'pts' SPEC2D ABS 'outputs/spec_out.nc' OUTPUT 20200221.040000 1.0 HR
TABle 'pts' HEADer 'outputs/tab_out.nc' TIME XP YP HS TPS TM01 DIR DSPR WIND OUTPUT 20200221.040000 1.0 HR

COMPUTE NONST {{runtime.period.start.strftime(runtime._datefmt)}} {{runtime.frequency}} {{runtime.period.end.strftime(runtime._datefmt)}}

STOP

[17]:

# So in this case, nothing only a single value is rendered from the config object, only runtime object is used. This demonstrates how an existing configuration can be
# utilised in Rompy with basic template changes so it could be used to run a hindcast with multiple montthly submissions for example. e.g. to run this
# simple case, we just initialise the model with the config pointing to this template

basic = ModelRun(run_id='basic', config=dict(template="../rompy/templates/swanbasic/",friction_coefficient=0.2, model_type='base'))

# an run it
basic()

INFO:rompy.model:
INFO:rompy.model:-----------------------------------------------------
INFO:rompy.model:Model settings:
INFO:rompy.model:
period:

        Start: 2020-02-21 04:00:00
        End: 2020-02-24 04:00:00
        Duration: 3 days, 0:00:00
        Interval: 0:15:00
        Include End: True


output_dir:
./simulations

config:
model_type='base' template='../rompy/templates/swanbasic/' checkout='main' friction_coefficient=0.2
INFO:rompy.model:-----------------------------------------------------
INFO:rompy.model:Generating model input files in ./simulations
INFO:rompy.model:
INFO:rompy.model:Successfully generated project in ./simulations
INFO:rompy.model:-----------------------------------------------------

[17]:

'/source/rompy/notebooks/simulations/basic'

[18]:

# If we inspect the control file, we should see the run times have been popoluated by the runtime, and the friction coefficient has been set to 0.2.
# No python code was required to do this, just a template and a dictionary or required arguments.
dump_input(basic)

$
$ SWAN - Simple example template used by rompy
$ Template: ../rompy/templates/swanbasic/
$ Generated: 2023-06-19 06:15:51.237042 on tom-xps by tdurrant
$ projection: wgs84
$

MODE NONSTATIONARY TWODIMENSIONAL
COORDINATES SPHERICAL
SET NAUTICAL

CGRID REG 115.68 -32.76 77.0 0.39 0.15 389 149 CIRCLE 36 0.0464 1.0 31


INPGRID BOTTOM REG 115.68 -32.76 0.0 187 150 0.0009999999999999872 0.0009999999999999905 EXC -99.0
READINP BOTTOM 1.0 'bottom.grd' 3 FREE

INPGRID WIND REG 115.68 -32.76 77.0 389 149 0.001 0.001 NONSTATION 20200221.040000 0.25 HR
READINP WIND 1 'wind.grd' 3 0 1 0 FREE

GEN3 WESTH 0.000075 0.00175
BREAKING
FRICTION MAD 0.2

TRIADS

PROP BSBT
NUM ACCUR 0.02 0.02 0.02 95 NONSTAT 20

BOUNDNEST1 NEST 'boundary.bnd' CLOSED
OUTPUT OPTIONS BLOCK 8
BLOCK 'COMPGRID' HEADER 'outputs/swan_out.nc' LAYOUT 1 DEPTH UBOT HSIGN HSWELL DIR TPS TM01 WIND OUT 20200221.040000 1.0 HR

POINTs 'pts' FILE 'out.loc'
SPECout 'pts' SPEC2D ABS 'outputs/spec_out.nc' OUTPUT 20200221.040000 1.0 HR
TABle 'pts' HEADer 'outputs/tab_out.nc' TIME XP YP HS TPS TM01 DIR DSPR WIND OUTPUT 20200221.040000 1.0 HR

COMPUTE NONST 20200221.040000 0.25 HR 20200224.040000

STOP

[19]:

# Config objects can also go the other way. At the moment, only a subset of the SWAN configuration is implemented in Rompy. However, it is possible to
# create pydantic objects similar to those on teh SwanConfig object, but mirroring the entire functionaly of the SWAN model. This is a work in progress,
# and will be demonstrated in a this notebook components/swan-config-components.ipynb.

# Hopefully this shows that the model config object can be as simple or as complex as requuired. This also provided a soft on ramp to developiong
# support for other models as you are not required to write do weeks of development before you can even start.

[ ]:

Table of Contents

Templates demonstration#