SCHISM Backend Framework Tutorial

This tutorial guides you through using the SCHISM backend framework to execute SCHISM simulations with Docker containers, automatic image building, and comprehensive testing.

Getting Started

Prerequisites

Before starting, ensure you have:

• Docker installed and running
• ROMPY installed with backend framework support
• SCHISM boundary conditions examples (included with ROMPY)

Quick Start

The easiest way to get started is with the boundary conditions examples:

# Navigate to the SCHISM examples directory
cd rompy/notebooks/schism

# Run a single tidal example (dry run)
python run_boundary_conditions_examples.py --single basic_tidal --dry-run

# Run all tidal examples
python run_boundary_conditions_examples.py --tidal

# Run a complete test suite
python test_backend_examples.py

Basic Tutorial

Step 1: Understanding the Framework

The SCHISM backend framework consists of three main components:

1. ModelRun - ROMPY's model execution framework
2. DockerConfig - Backend configuration for Docker execution
3. Boundary Conditions - SCHISM-specific configuration system

Here's how they work together:

from rompy.model import ModelRun
from rompy.backends import DockerConfig
from rompy.core.time import TimeRange
from datetime import datetime
from pathlib import Path

# 1. Create a model run
model_run = ModelRun(
    run_id="my_schism_example",
    period=TimeRange(
        start=datetime(2023, 1, 1),
        end=datetime(2023, 1, 2),
        interval="1H"
    ),
    output_dir="schism_output",
    delete_existing=True
)

# 2. Configure Docker backend
docker_config = DockerConfig(
    dockerfile=Path("Dockerfile"),
    build_context=Path("docker/schism"),
    timeout=3600,
    cpu=8,
    memory="4g"
)

# 3. Generate configuration files
model_run.generate()

# 4. Execute simulation
success = model_run.run(backend=docker_config)

Step 2: Creating Your First SCHISM Configuration

Create a YAML configuration file for a simple tidal simulation:

# my_tidal_example.yaml
run_id: my_first_schism_run
period:
  start: 2023-01-01T00:00:00
  end: 2023-01-01T12:00:00
  interval: 1H
output_dir: my_schism_output
delete_existing: true

config:
  model_type: schism
  grid:
    grid_type: schism
    hgrid:
      model_type: data_blob
      source: tests/data/schism/hgrid.gr3
    drag: 2.5e-3

  data:
    data_type: schism
    boundary_conditions:
      data_type: boundary_conditions
      setup_type: tidal
      tidal_data:
        tidal_database: tests/data/schism/tides
        tidal_model: 'OCEANUM-atlas'
        constituents: [M2, S2, N2]
      boundaries:
        0:
          elev_type: 3  # HARMONIC
          vel_type: 3   # HARMONIC
          temp_type: 0  # NONE
          salt_type: 0  # NONE

  nml:
    param:
      core:
        dt: 150.0
        ibc: 1  # Barotropic
        ibtp: 0  # No tracer transport
        nspool: 24
        ihfskip: 1152
      schout:
        iof_hydro__1: 1  # elevation
        iof_hydro__26: 1  # velocity vector

Step 3: Running Your Configuration

Now create a Python script to execute your configuration:

# run_my_example.py
import yaml
from rompy.model import ModelRun
from rompy.backends import DockerConfig
from pathlib import Path

def main():
    # Load configuration
    with open("my_tidal_example.yaml", "r") as f:
        config = yaml.safe_load(f)

    # Create model run
    model_run = ModelRun(**config)

    # Configure Docker backend
    docker_config = DockerConfig(
        dockerfile=Path("Dockerfile"),
        build_context=Path("docker/schism"),
        timeout=3600,
        cpu=8,
        memory="4g",
        executable="bash -c 'cd /tmp/schism && mpirun --allow-run-as-root -n 8 schism_v5.13.0 4'",
        volumes=[f"{Path.cwd() / 'my_schism_output'}:/tmp/schism:rw"],
        env_vars={
            "OMPI_ALLOW_RUN_AS_ROOT": "1",
            "OMPI_ALLOW_RUN_AS_ROOT_CONFIRM": "1"
        },
        remove_container=True
    )

    try:
        # Generate configuration files
        print("Generating SCHISM configuration...")
        model_run.generate()

        # Execute simulation
        print("Running SCHISM simulation...")
        success = model_run.run(backend=docker_config)

        if success:
            print("✅ Simulation completed successfully!")

            # Check for output files
            output_dir = Path("my_schism_output")
            outputs = list(output_dir.glob("**/outputs/*.nc"))
            print(f"Generated {len(outputs)} output files")

        else:
            print("❌ Simulation failed")

    except Exception as e:
        print(f"❌ Error: {e}")

if __name__ == "__main__":
    main()

Step 4: Execute Your Example

Run your example:

python run_my_example.py

Expected output:

Generating SCHISM configuration...
Running SCHISM simulation...
✅ Simulation completed successfully!
Generated 1 output files

Intermediate Tutorial

Step 5: Adding Wave Coupling

Extend your configuration to include wave coupling:

# Add to your existing configuration
config:
  data:
    wave:
      buffer: 0.0
      coords:
        t: time
        x: lon
        y: lat
        z: depth
      id: wavedata
      source:
        catalog_uri: tests/data/catalog.yaml
        dataset_id: ausspec
        model_type: intake

  nml:
    param:
      opt:
        ihot: 0
        nstep_wwm: 4
      schout:
        iof_wwm__1: 1   # significant wave height
        iof_wwm__9: 1   # peak wave period
        iof_wwm__18: 1  # peak wave direction
    wwminput:
      proc:
        deltc: 600

Step 6: Hybrid Boundary Conditions

Configure hybrid boundaries that combine tidal and ocean data:

config:
  data:
    boundary_conditions:
      data_type: boundary_conditions
      setup_type: hybrid
      tidal_data:
        tidal_database: tests/data/schism/tides
        tidal_model: 'OCEANUM-atlas'
        constituents: [M2, S2, N2]
      boundaries:
        0:
          elev_type: 5  # HARMONICEXTERNAL
          vel_type: 5   # HARMONICEXTERNAL
          temp_type: 4  # EXTERNAL
          salt_type: 4  # EXTERNAL
          elev_source:
            data_type: boundary
            source:
              model_type: file
              uri: tests/data/schism/hycom.nc
            variables: [surf_el]
            coords:
              t: time
              x: xlon
              y: ylat
          vel_source:
            data_type: boundary
            source:
              model_type: file
              uri: tests/data/schism/hycom.nc
            variables: [water_u, water_v]
            coords:
              t: time
              x: xlon
              y: ylat
              z: depth

Step 7: Adding Hotstart Generation

Generate initial conditions from your ocean data:

config:
  data:
    boundary_conditions:
      hotstart_config:
        enabled: true
        temp_var: temperature
        salt_var: salinity
        output_filename: hotstart.nc

Advanced Tutorial

Step 8: Custom Docker Configuration

Create advanced Docker configurations for specific needs:

# High-performance configuration
docker_config = DockerConfig(
    dockerfile=Path("Dockerfile.optimized"),
    build_context=Path("docker/schism"),
    build_args={
        "SCHISM_VERSION": "v5.13.0",
        "ENABLE_OPTIMIZATION": "ON"
    },
    cpu=16,
    memory="16g",
    timeout=7200,
    volumes=[
        f"{output_dir}:/tmp/schism:rw",
        "/tmp:/tmp_host:rw"
    ],
    env_vars={
        "OMPI_ALLOW_RUN_AS_ROOT": "1",
        "OMPI_ALLOW_RUN_AS_ROOT_CONFIRM": "1",
        "OMP_NUM_THREADS": "16"
    }
)

Step 9: Multiple Boundary Types

Configure complex scenarios with multiple boundary types:

config:
  data:
    boundary_conditions:
      data_type: boundary_conditions
      setup_type: mixed
      tidal_data:
        tidal_database: tests/data/schism/tides
        tidal_model: 'OCEANUM-atlas'
        constituents: [M2, S2, N2]
      boundaries:
        0:  # Ocean boundary (tidal + external)
          elev_type: 5
          vel_type: 5
          temp_type: 4
          salt_type: 4
          # ... data sources
        1:  # River boundary (constant flow)
          elev_type: 0
          vel_type: 2
          temp_type: 2
          salt_type: 2
          const_flow: -100.0
          const_temp: 15.0
          const_salt: 0.1
        2:  # Nested boundary (relaxation)
          elev_type: 5
          vel_type: 7
          temp_type: 4
          salt_type: 4
          inflow_relax: 0.8
          outflow_relax: 0.2

Step 10: Automated Testing

Create automated testing for your configurations:

# test_my_configurations.py
import pytest
import yaml
from pathlib import Path
from rompy.model import ModelRun
from rompy.backends import DockerConfig

class TestSchismConfigurations:

    def test_basic_tidal_config(self):
        """Test basic tidal configuration."""
        config_file = Path("my_tidal_example.yaml")
        assert config_file.exists()

        with open(config_file, "r") as f:
            config = yaml.safe_load(f)

        model_run = ModelRun(**config)
        assert model_run.run_id == "my_first_schism_run"

    def test_docker_config_creation(self):
        """Test Docker configuration creation."""
        docker_config = DockerConfig(
            dockerfile=Path("Dockerfile"),
            build_context=Path("docker/schism"),
            cpu=8,
            memory="4g"
        )

        assert docker_config.dockerfile == Path("Dockerfile")
        assert docker_config.cpu == 8
        assert docker_config.memory == "4g"

    def test_dry_run_execution(self):
        """Test configuration generation without execution."""
        with open("my_tidal_example.yaml", "r") as f:
            config = yaml.safe_load(f)

        model_run = ModelRun(**config)

        # Test configuration generation
        try:
            model_run.generate()
            assert True, "Configuration generation succeeded"
        except Exception as e:
            pytest.fail(f"Configuration generation failed: {e}")

if __name__ == "__main__":
    pytest.main([__file__, "-v"])

Production Usage

Step 11: Batch Processing

Run multiple configurations in batch:

# batch_runner.py
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import yaml
from rompy.model import ModelRun
from rompy.backends import DockerConfig

def run_configuration(config_file):
    """Run a single configuration file."""
    try:
        with open(config_file, "r") as f:
            config = yaml.safe_load(f)

        model_run = ModelRun(**config)

        docker_config = DockerConfig(
            dockerfile=Path("Dockerfile"),
            build_context=Path("docker/schism"),
            timeout=3600,
            cpu=8,
            memory="4g"
        )

        model_run.generate()
        success = model_run.run(backend=docker_config)

        return config_file.name, success

    except Exception as e:
        return config_file.name, False

def main():
    # Find all configuration files
    config_files = list(Path(".").glob("*_example.yaml"))

    # Run configurations in parallel
    with ThreadPoolExecutor(max_workers=3) as executor:
        results = list(executor.map(run_configuration, config_files))

    # Report results
    for config_name, success in results:
        status = "✅" if success else "❌"
        print(f"{status} {config_name}")

if __name__ == "__main__":
    main()

Step 12: Monitoring and Logging

Add comprehensive monitoring to your runs:

# monitored_runner.py
import logging
import time
from pathlib import Path
from rompy.model import ModelRun
from rompy.backends import DockerConfig

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('schism_runs.log'),
        logging.StreamHandler()
    ]
)

logger = logging.getLogger(__name__)

def run_with_monitoring(config_file):
    """Run configuration with detailed monitoring."""
    start_time = time.time()

    try:
        logger.info(f"Starting run: {config_file}")

        with open(config_file, "r") as f:
            config = yaml.safe_load(f)

        model_run = ModelRun(**config)

        logger.info(f"Run ID: {model_run.run_id}")
        logger.info(f"Period: {model_run.period.start} to {model_run.period.end}")

        docker_config = DockerConfig(
            dockerfile=Path("Dockerfile"),
            build_context=Path("docker/schism"),
            timeout=3600,
            cpu=8,
            memory="4g"
        )

        # Generate configuration
        logger.info("Generating configuration files...")
        model_run.generate()

        # Execute simulation
        logger.info("Starting SCHISM simulation...")
        success = model_run.run(backend=docker_config)

        execution_time = time.time() - start_time

        if success:
            logger.info(f"✅ Run completed successfully in {execution_time:.1f}s")

            # Check output files
            output_dir = Path(config["output_dir"])
            outputs = list(output_dir.glob("**/outputs/*.nc"))
            logger.info(f"Generated {len(outputs)} output files")

            return True
        else:
            logger.error(f"❌ Run failed after {execution_time:.1f}s")
            return False

    except Exception as e:
        execution_time = time.time() - start_time
        logger.error(f"❌ Run crashed after {execution_time:.1f}s: {e}")
        return False

Troubleshooting

Common Issues and Solutions

Docker Build Failures:

# Check Docker daemon
docker version

# Verify Dockerfile exists
ls -la docker/schism/Dockerfile

# Test manual build
cd docker/schism
docker build -t test-schism .

Configuration Errors:

# Validate YAML syntax
import yaml
with open("my_config.yaml", "r") as f:
    config = yaml.safe_load(f)
print("✅ YAML is valid")

# Test ModelRun creation
from rompy.model import ModelRun
model_run = ModelRun(**config)
print("✅ ModelRun created successfully")

Resource Issues:

# Check system resources
import psutil
print(f"Available CPUs: {psutil.cpu_count()}")
print(f"Available Memory: {psutil.virtual_memory().available / 1024**3:.1f} GB")

# Adjust Docker configuration accordingly
docker_config = DockerConfig(
    cpu=min(8, psutil.cpu_count()),
    memory=f"{min(4, psutil.virtual_memory().available // 1024**3)}g"
)

File Path Issues:

# Use absolute paths for volume mounts
from pathlib import Path
output_dir = Path("my_output").absolute()

docker_config = DockerConfig(
    volumes=[f"{output_dir}:/tmp/schism:rw"]
)

Best Practices Summary

1. Start Simple - Begin with basic tidal configurations
2. Test Configurations - Use dry runs to validate before execution
3. Monitor Resources - Set appropriate CPU and memory limits
4. Use Absolute Paths - Avoid path resolution issues
5. Enable Logging - Track execution progress and errors
6. Clean Up - Remove containers after execution
7. Version Control - Track configuration changes
8. Document Runs - Record parameters and results

Next Steps

After completing this tutorial, you should be able to:

• Create and run basic SCHISM configurations
• Use the Docker backend framework effectively
• Configure complex boundary conditions
• Implement automated testing
• Monitor and troubleshoot runs

For more advanced features, see:

• Boundary Conditions - Comprehensive boundary conditions guide
• Hotstart - Initial conditions configuration
• Backend Framework - Complete backend framework reference
• ROMPY Backend System - ROMPY backend system documentation

Happy modeling with SCHISM and ROMPY! 🌊