Running SCHISM¶

What this shows: how to run a SCHISM workspace generated by rompy-schism: with the Docker image or a local installation, with MPI and scribes, and how to tell whether the run succeeded.

Prerequisites: Tutorial 1: Your first SCHISM model.

You will learn:

how rompy's run backends (DockerConfig, LocalConfig) execute SCHISM
how MPI processes and scribes add up
why SCHISM's exit code cannot be trusted, and how to check a run instead
what SCHISM writes about the run itself

Data used: hgrid.gr3 and tides/. The runs need Docker (or SCHISM installed locally) and are skipped otherwise.

Setup¶

A short tidal model, generated once per run below.

In [1]:

Copied!





import shutil
import subprocess
from pathlib import Path

from rompy.backends import DockerConfig, LocalConfig
from rompy.core.data import DataBlob
from rompy.core.time import TimeRange
from rompy.logging import config as logging_config
from rompy.model import ModelRun

from rompy_schism.boundary_core import TidalDataset
from rompy_schism.config import SCHISMConfig
from rompy_schism.data import (
    BoundarySetupWithSource,
    SCHISMData,
    SCHISMDataBoundaryConditions,
)
from rompy_schism.grid import SCHISMGrid
from rompy_schism.namelists import NML, Param

logging_config.update(level="ERROR")

DATA_DIR = Path("../data")
OUT_DIR = Path("_output") / "running_schism"
shutil.rmtree(OUT_DIR, ignore_errors=True)
SCHISM_IMAGE = "ghcr.io/rom-py/schism:5.13.0"

config = SCHISMConfig(
    grid=SCHISMGrid(hgrid=DataBlob(source=DATA_DIR / "hgrid.gr3"), drag=0.0025),
    data=SCHISMData(
        boundary_conditions=SCHISMDataBoundaryConditions(
            tidal_data=TidalDataset(
                tidal_database=DATA_DIR / "tides",
                tidal_model="TPXO9-perth",
                constituents=["M2", "S2", "K1", "O1"],
            ),
            default_boundary=BoundarySetupWithSource(elev_type=3, vel_type=3),
        )
    ),
    nml=NML(
        param=Param(
            core={"ibc": 1, "ibtp": 0, "dt": 120.0, "nspool": 30, "ihfskip": 360},
            schout={"iof_hydro__1": 1, "iof_hydro__16": 1, "iof_hydro__26": 0},
        )
    ),
)
period = TimeRange(start="2023-01-01T00:00", end="2023-01-01T12:00", interval="1h")


def new_run(run_id: str) -> tuple[ModelRun, Path]:
    """Generate a workspace for this model under its own run id."""
    modelrun = ModelRun(run_id=run_id, period=period, output_dir=OUT_DIR, config=config)
    return modelrun, Path(modelrun())


def docker_available() -> bool:
    """Return True if the Docker daemon can be reached."""
    try:
        return subprocess.run(["docker", "info"], capture_output=True).returncode == 0
    except FileNotFoundError:
        return False
import shutil
import subprocess
from pathlib import Path

from rompy.backends import DockerConfig, LocalConfig
from rompy.core.data import DataBlob
from rompy.core.time import TimeRange
from rompy.logging import config as logging_config
from rompy.model import ModelRun

from rompy_schism.boundary_core import TidalDataset
from rompy_schism.config import SCHISMConfig
from rompy_schism.data import (
    BoundarySetupWithSource,
    SCHISMData,
    SCHISMDataBoundaryConditions,
)
from rompy_schism.grid import SCHISMGrid
from rompy_schism.namelists import NML, Param

logging_config.update(level="ERROR")

DATA_DIR = Path("../data")
OUT_DIR = Path("_output") / "running_schism"
shutil.rmtree(OUT_DIR, ignore_errors=True)
SCHISM_IMAGE = "ghcr.io/rom-py/schism:5.13.0"

config = SCHISMConfig(
    grid=SCHISMGrid(hgrid=DataBlob(source=DATA_DIR / "hgrid.gr3"), drag=0.0025),
    data=SCHISMData(
        boundary_conditions=SCHISMDataBoundaryConditions(
            tidal_data=TidalDataset(
                tidal_database=DATA_DIR / "tides",
                tidal_model="TPXO9-perth",
                constituents=["M2", "S2", "K1", "O1"],
            ),
            default_boundary=BoundarySetupWithSource(elev_type=3, vel_type=3),
        )
    ),
    nml=NML(
        param=Param(
            core={"ibc": 1, "ibtp": 0, "dt": 120.0, "nspool": 30, "ihfskip": 360},
            schout={"iof_hydro__1": 1, "iof_hydro__16": 1, "iof_hydro__26": 0},
        )
    ),
)
period = TimeRange(start="2023-01-01T00:00", end="2023-01-01T12:00", interval="1h")


def new_run(run_id: str) -> tuple[ModelRun, Path]:
    """Generate a workspace for this model under its own run id."""
    modelrun = ModelRun(run_id=run_id, period=period, output_dir=OUT_DIR, config=config)
    return modelrun, Path(modelrun())


def docker_available() -> bool:
    """Return True if the Docker daemon can be reached."""
    try:
        return subprocess.run(["docker", "info"], capture_output=True).returncode == 0
    except FileNotFoundError:
        return False

1. How SCHISM runs¶

SCHISM always runs under MPI, from inside the workspace, which must have an outputs/ folder (rompy-schism creates it):

mpirun -n 6 schism 2

The argument is the number of scribes, processes that only write output; the others compute. mpirun -n 6 schism 2 runs 4 compute processes and 2 scribes. SCHISM needs one scribe for all its 2D outputs (and the vertical coordinates), plus one per 3D output variable, two for vectors: 2 for the 2D model here, 4 with the default 3D velocity output. Output has the details.

2. With Docker¶

The public image ghcr.io/rom-py/schism has SCHISM v5.13.0 with MPI and NetCDF:

Command	What runs
`schism`	Hydrodynamics
`schism_wwm`	Hydrodynamics coupled with the WWM wave model (Waves with WWM)
`combine_hotstart7`	Merges the hotstart files of all processes (Hotstart and chained runs)

rompy's DockerConfig mounts the workspace in the container and runs mpiexec -n cpu executable there:

In [2]:

Copied!





docker = DockerConfig(image=SCHISM_IMAGE, executable="schism 2", mpiexec="mpirun", cpu=6)
if docker_available():
    modelrun, workspace = new_run("docker")
    returned = modelrun.run(docker, workspace_dir=workspace)
    print(f"run() returned {returned}")
else:
    print("Docker is not available, skipping the model run")
docker = DockerConfig(image=SCHISM_IMAGE, executable="schism 2", mpiexec="mpirun", cpu=6)
if docker_available():
    modelrun, workspace = new_run("docker")
    returned = modelrun.run(docker, workspace_dir=workspace)
    print(f"run() returned {returned}")
else:
    print("Docker is not available, skipping the model run")

run() returned True

The same from a terminal, in the workspace folder:

docker run --rm -v "$PWD":/data -w /data ghcr.io/rom-py/schism:5.13.0 mpirun -n 6 schism 2

More processes than the machine has cores are fine: the image allows it, since the scribes mostly wait.

3. Checking a run¶

SCHISM exits with success even when it stops on an error, so run(), which relies on the exit code, returns True either way. Check SCHISM's own files instead:

outputs/mirror.out, the log, ends with "Run completed successfully";
outputs/fatal.error holds the reason SCHISM stopped;
outputs/nonfatal_* hold warnings.

In [3]:

Copied!





def run_completed(workspace: Path) -> bool:
    """Check SCHISM's log: SCHISM exits with success even when it stops on an error."""
    mirror = workspace / "outputs" / "mirror.out"
    if mirror.exists() and "Run completed successfully" in mirror.read_text():
        return True
    fatal = workspace / "outputs" / "fatal.error"
    print(fatal.read_text().strip() if fatal.exists() else "SCHISM did not finish")
    return False


if docker_available():
    print(f"completed: {run_completed(workspace)}")
def run_completed(workspace: Path) -> bool:
    """Check SCHISM's log: SCHISM exits with success even when it stops on an error."""
    mirror = workspace / "outputs" / "mirror.out"
    if mirror.exists() and "Run completed successfully" in mirror.read_text():
        return True
    fatal = workspace / "outputs" / "fatal.error"
    print(fatal.read_text().strip() if fatal.exists() else "SCHISM did not finish")
    return False


if docker_available():
    print(f"completed: {run_completed(workspace)}")

completed: True

A failed run for comparison: too few scribes for the output. run() still returns True.

In [4]:

Copied!





if docker_available():
    modelrun, failed = new_run("too_few_scribes")
    one_scribe = DockerConfig(
        image=SCHISM_IMAGE, executable="schism 1", mpiexec="mpirun", cpu=6
    )
    print(f"run() returned {modelrun.run(one_scribe, workspace_dir=failed)}")
    print(f"completed: {run_completed(failed)}")
if docker_available():
    modelrun, failed = new_run("too_few_scribes")
    one_scribe = DockerConfig(
        image=SCHISM_IMAGE, executable="schism 1", mpiexec="mpirun", cpu=6
    )
    print(f"run() returned {modelrun.run(one_scribe, workspace_dir=failed)}")
    print(f"completed: {run_completed(failed)}")

run() returned True
3: ABORT: INIT: Too few scribes (1).  Please specify atleast equal to number of output variables (2)
completed: False

4. What SCHISM writes about the run¶

Besides the results, outputs/ has files about the run:

File	Contents
`mirror.out`	The log: setup, each time step, warnings, the final message
`param.out.nml`	The settings SCHISM used, including its own defaults
`fatal.error`, `nonfatal_*`	Errors and warnings
`local_to_global_*`, `global_to_local.prop`	How the mesh was split between compute processes
`maxelev_`, `maxdahv_`	Maximum water level and depth-averaged velocity at each node, per process
`total.out`, `total_TR.out`	Volume and tracer mass over time

In [5]:

Copied!





if docker_available():
    print(sorted(p.name for p in (workspace / "outputs").iterdir()))
    mirror = (workspace / "outputs" / "mirror.out").read_text().splitlines()
    steps = [line.strip() for line in mirror if "TIME STEP=" in line]
    print(f"{len(steps)} time steps, the last: {steps[-1]}")
    print(mirror[-1].strip())
if docker_available():
    print(sorted(p.name for p in (workspace / "outputs").iterdir()))
    mirror = (workspace / "outputs" / "mirror.out").read_text().splitlines()
    steps = [line.strip() for line in mirror if "TIME STEP=" in line]
    print(f"{len(steps)} time steps, the last: {steps[-1]}")
    print(mirror[-1].strip())

['JCG.out', 'coriolis.out', 'fatal.error', 'flux.out', 'global_to_local.prop', 'local_to_global_000000', 'local_to_global_000001', 'local_to_global_000002', 'local_to_global_000003', 'maxdahv_000000', 'maxdahv_000001', 'maxdahv_000002', 'maxdahv_000003', 'maxelev_000000', 'maxelev_000001', 'maxelev_000002', 'maxelev_000003', 'mirror.out', 'mirror.out.scribe', 'nonfatal_000000', 'nonfatal_000001', 'nonfatal_000002', 'nonfatal_000003', 'out2d_1.nc', 'param.out.nml', 'readme.md', 'subcycling.out', 'total.out', 'total_TR.out', 'zCoordinates_1.nc']
360 time steps, the last: TIME STEP=          360;  TIME=         43200.000000
Run completed successfully at 20260929, 020222.497

5. With your own SCHISM¶

SCHISM is compiled from source with the modules you need (the compilation guide). LocalConfig runs a command in the workspace on this machine. The executable name depends on how it was built, for example pschism_TVD-VL:

In [6]:

Copied!





executable = shutil.which("schism") or shutil.which("pschism_TVD-VL")
if executable and shutil.which("mpirun"):
    modelrun, workspace = new_run("local")
    modelrun.run(
        LocalConfig(command=f"mpirun -n 6 {executable} 2"), workspace_dir=workspace
    )
    print(f"completed: {run_completed(workspace)}")
else:
    print("SCHISM is not installed here, skipping the local run")
executable = shutil.which("schism") or shutil.which("pschism_TVD-VL")
if executable and shutil.which("mpirun"):
    modelrun, workspace = new_run("local")
    modelrun.run(
        LocalConfig(command=f"mpirun -n 6 {executable} 2"), workspace_dir=workspace
    )
    print(f"completed: {run_completed(workspace)}")
else:
    print("SCHISM is not installed here, skipping the local run")

SCHISM is not installed here, skipping the local run

6. From the command line¶

rompy run generates and runs a YAML configuration with a backend from a second YAML file, as in Tutorial 7:

type: docker
image: ghcr.io/rom-py/schism:5.13.0
executable: schism 2
mpiexec: mpirun
cpu: 6

Summary¶

SCHISM runs under MPI with the number of scribes as its argument; mpirun -n counts compute processes and scribes.
DockerConfig with ghcr.io/rom-py/schism needs only Docker; LocalConfig runs your own build.
run() cannot tell a failed SCHISM run from a successful one: check outputs/mirror.out and outputs/fatal.error.