Backend Systems#

ROMPY’s backend system provides flexible, type-safe execution environments for wave models. Whether you’re running simple local simulations or complex containerized workflows, the backend system handles execution, resource management, and result processing.

Overview#

Backend systems in ROMPY control how and where your models execute. They provide:

Execution Environments: Local system, Docker containers, HPC clusters
Resource Management: CPU, memory, and timeout controls
Type Safety: Pydantic-based configurations with validation
Reproducibility: Consistent execution across different environments
Scalability: From development to production deployments

Key Components#

Backend Configurations: Type-safe Pydantic models that define execution parameters. See rompy.backends.config.BaseBackendConfig and its subclasses.
Run Backends: Execution engines that handle the actual model runs. See rompy.run for implementations.
Postprocessors: Components that handle results after model execution. See rompy.backends.postprocessors for available processors.
CLI Integration: Command-line tools for configuration management and execution. See Command Line Interface for details.

Quick Start#

Get started with backends in three simple steps:

Create a Backend Configuration

# my_backend.yml
type: local
timeout: 3600
command: "python run_model.py"
env_vars:
  OMP_NUM_THREADS: "4"

Validate Your Configuration

rompy backends validate my_backend.yml

Run Your Model

from rompy.model import ModelRun
from rompy.backends import LocalConfig

# Load and run
model = ModelRun.from_file("model_config.yml")
config = LocalConfig.from_file("my_backend.yml")
success = model.run(backend=config)

Configuration Types#

All backend configurations inherit from rompy.backends.config.BaseBackendConfig and provide type-safe, validated execution parameters.

LocalConfig - Local System Execution#

Execute models directly on your local system using rompy.backends.config.LocalConfig.

Basic Usage:

from rompy.backends import LocalConfig

config = LocalConfig(
    timeout=3600,
    command="python run_simulation.py"
)

Advanced Configuration:

# local_advanced.yml
type: local
timeout: 7200  # 2 hours
command: "python run_simulation.py --verbose"
shell: true
capture_output: true
env_vars:
  OMP_NUM_THREADS: "8"
  PYTHONPATH: "/custom/path"
  MODEL_DEBUG: "true"

Key Parameters:

timeout: Maximum execution time in seconds (60-86400)
command: Shell command to execute (optional)
shell: Execute through shell (default: True)
capture_output: Capture stdout/stderr (default: True)
env_vars: Environment variables to set
working_dir: Working directory for execution

For complete parameter documentation, see rompy.backends.config.LocalConfig.

DockerConfig - Container Execution#

Execute models inside Docker containers for reproducible, isolated environments using rompy.backends.config.DockerConfig.

Basic Usage:

from rompy.backends import DockerConfig

config = DockerConfig(
    image="swan:latest",
    cpu=4,
    memory="2g",
    timeout=3600
)

Advanced Configuration:

# docker_advanced.yml
type: docker
image: "swan:latest"
timeout: 10800  # 3 hours
cpu: 8
memory: "4g"
mpiexec: "mpirun -np 8"
user: "modeluser"
volumes:
  - "/data/input:/app/input:ro"
  - "/data/output:/app/output:rw"
env_vars:
  MODEL_THREADS: "8"
  DATA_DIR: "/app/input"

Building from Dockerfile:

# docker_build.yml
type: docker
dockerfile: "./docker/Dockerfile"
build_context: "./docker"
build_args:
  MODEL_VERSION: "2.1.0"
cpu: 4
memory: "2g"

Key Parameters:

image: Docker image to use (mutually exclusive with dockerfile)
dockerfile: Path to Dockerfile to build (mutually exclusive with image)
cpu: Number of CPU cores (1-128)
memory: Memory limit (e.g., “2g”, “512m”)
timeout: Maximum execution time in seconds
volumes: Volume mounts in “host:container:mode” format
env_vars: Environment variables to set
executable: Path to executable inside container
mpiexec: MPI execution command for parallel runs

For complete parameter documentation, see rompy.backends.config.DockerConfig.

Using Backend Configurations#

With ModelRun#

Backend configurations integrate directly with ROMPY’s model execution via rompy.model.ModelRun.run():

from rompy.model import ModelRun
from rompy.backends import LocalConfig, DockerConfig

# Load your model
model_run = ModelRun.from_file("model_config.yml")

# Execute locally
local_config = LocalConfig(timeout=3600)
success = model_run.run(backend=local_config)

# Execute in Docker
docker_config = DockerConfig(
    image="rompy/swan:latest",
    cpu=4,
    memory="4g"
)
success = model_run.run(backend=docker_config)

From Configuration Files#

Load configurations from YAML or JSON files:

import yaml
from rompy.backends import LocalConfig, DockerConfig

# Load configuration
with open("backend_config.yml") as f:
    config_data = yaml.safe_load(f)
    backend_type = config_data.pop("type")

    if backend_type == "local":
        config = LocalConfig(**config_data)
    elif backend_type == "docker":
        config = DockerConfig(**config_data)

# Use configuration
success = model_run.run(backend=config)

Command Line Interface#

The CLI provides comprehensive backend management capabilities. See Command Line Interface for complete details.

Configuration Management#

# Validate configuration
rompy backends validate my_config.yml

# List available backends
rompy backends list

# Show configuration schema
rompy backends schema --backend-type docker

# Create configuration template
rompy backends create --backend-type local --output template.yml

Model Execution#

# Run model with backend configuration
rompy run model_config.yml --backend-config my_backend.yml

# Run pipeline with configuration
rompy pipeline --config pipeline_config.yml

Configuration Examples#

Environment-Specific Configurations#

Development Environment:

# dev_backend.yml
type: local
timeout: 1800  # 30 minutes
command: "python run_model.py --debug"
env_vars:
  ENV: "development"
  LOG_LEVEL: "DEBUG"
  PYTHONUNBUFFERED: "1"

Production Environment:

# prod_backend.yml
type: docker
image: "mymodel:production"
timeout: 14400  # 4 hours
cpu: 16
memory: "32g"
mpiexec: "mpirun -np 16"
volumes:
  - "/data/production:/app/data:ro"
  - "/results/production:/app/results:rw"
env_vars:
  ENV: "production"
  LOG_LEVEL: "INFO"

Resource-Based Configurations#

Small Models:

type: local
timeout: 3600
env_vars:
  OMP_NUM_THREADS: "4"

Large Models:

type: docker
image: "swan:hpc"
timeout: 86400  # 24 hours
cpu: 32
memory: "64g"
mpiexec: "mpirun -np 32"
env_vars:
  OMP_NUM_THREADS: "1"
  MODEL_PRECISION: "double"

Validation and Error Handling#

Type Safety#

Pydantic provides comprehensive validation with clear error messages:

from rompy.backends import LocalConfig, DockerConfig
from pydantic import ValidationError

try:
    # Invalid timeout (too short)
    config = LocalConfig(timeout=30)
except ValidationError as e:
    print(f"Validation error: {e}")

try:
    # Missing required image/dockerfile
    config = DockerConfig()
except ValidationError as e:
    print(f"Configuration error: {e}")

Configuration Validation#

Each configuration class validates fields according to execution environment requirements:

LocalConfig Validation: * Working directory must exist if specified * Environment variables must be string key-value pairs * Timeout must be between 60 and 86400 seconds

DockerConfig Validation: * Either image or dockerfile must be provided (not both) * CPU count must be between 1 and 128 * Memory format must match pattern (e.g., “2g”, “512m”) * Volume mounts must use “host:container:mode” format * Docker image names must follow valid naming conventions

Best Practices#

Configuration Management#

Version Control: Keep configuration files in version control
Environment Variables: Use environment variables for sensitive data:

config = LocalConfig(
    env_vars={"API_KEY": os.environ["API_KEY"]}
)

Validation: Always validate configurations before production use:

rompy backends validate my_config.yml

Documentation: Document your configurations with comments:

# Production SWAN model configuration
type: docker
image: "swan:2.1.0"  # Pin specific version
timeout: 14400  # 4 hours for typical runs
cpu: 16  # Match server capabilities

Resource Planning#

Start Small: Begin with conservative resource allocations
Monitor Usage: Track actual resource consumption
Scale Gradually: Increase resources based on measured needs
Set Realistic Timeouts: Base timeouts on model complexity

Security Considerations#

Container Security: Use appropriate user permissions:

config = DockerConfig(
    image="myapp:latest",
    user="appuser",  # Don't run as root
    volumes=["/data:/app/data:ro"]  # Read-only when possible
)

Environment Variables: Never hardcode sensitive data in configuration files
Volume Mounts: Use read-only mounts when possible

Troubleshooting#

Common Issues#

Configuration Validation Errors

# Check configuration syntax
rompy backends validate my_config.yml

# Show configuration schema
rompy backends schema --backend-type local

Docker Issues

# Verify Docker image exists
docker images | grep myimage

# Test Docker configuration
rompy backends validate --backend-type docker docker_config.yml

Timeout Issues

# Increase timeout for long-running models
type: local
timeout: 21600  # 6 hours

Memory Issues

# Increase memory allocation
type: docker
image: "swan:latest"
memory: "8g"

Debugging Configuration Issues#

Check Validation Errors:

from pydantic import ValidationError

try:
    config = DockerConfig(**config_data)
except ValidationError as e:
    for error in e.errors():
        print(f"Field {error['loc']}: {error['msg']}")

Verify Configuration Serialization:

config = LocalConfig(timeout=3600)
print(config.model_dump_json(indent=2))

Test with Simple Examples:

# Start with minimal configuration
config = LocalConfig(timeout=3600)
# Add complexity gradually

Getting Help#

# General help
rompy backends --help

# Command-specific help
rompy backends validate --help

# Show examples
rompy backends create --backend-type local --with-examples

Advanced Usage#

Custom Backend Configurations#

Extend the system with custom backend types by inheriting from rompy.backends.config.BaseBackendConfig:

from rompy.backends import BaseBackendConfig
from pydantic import Field

class HPCConfig(BaseBackendConfig):
    """Configuration for HPC cluster execution."""

    queue: str = Field(..., description="SLURM queue name")
    nodes: int = Field(1, ge=1, le=100)
    partition: str = Field("compute")

    def get_backend_class(self):
        from mypackage.backends import HPCRunBackend
        return HPCRunBackend

For detailed implementation guidance, see Backend Reference.

Postprocessors#

Handle results after model execution using postprocessor classes:

# Basic postprocessing
results = model_run.postprocess(processor="archive")

# Custom postprocessing
results = model_run.postprocess(
    processor="custom_analyzer",
    output_format="netcdf",
    compress=True
)

For available postprocessors, see rompy.backends.postprocessors.

Schema Generation#

Generate configuration schemas for external tools:

from rompy.backends import LocalConfig, DockerConfig
import json

# Generate JSON schema
local_schema = LocalConfig.model_json_schema()
docker_schema = DockerConfig.model_json_schema()

# Save schema for external validation
with open("local_schema.json", "w") as f:
    json.dump(local_schema, f, indent=2)

Integration Examples#

Complete Workflow Example#

from rompy.model import ModelRun
from rompy.backends import DockerConfig

# Load model
model = ModelRun.from_file("swan_model.yml")

# Configure backend
backend = DockerConfig(
    image="swan:latest",
    cpu=8,
    memory="8g",
    timeout=7200,
    volumes=[
        "/data/bathymetry:/app/bathy:ro",
        "/data/forcing:/app/forcing:ro",
        "/results:/app/results:rw"
    ]
)

# Execute
success = model.run(backend=backend)

if success:
    print("Model execution completed successfully")
else:
    print("Model execution failed")

Pipeline Integration#

from rompy.pipeline import Pipeline
from rompy.backends import LocalConfig, DockerConfig

# Create pipeline with different backends for different stages
pipeline = Pipeline([
    {
        "name": "preprocessing",
        "backend": LocalConfig(timeout=1800),
        "command": "python preprocess.py"
    },
    {
        "name": "simulation",
        "backend": DockerConfig(
            image="swan:latest",
            cpu=16,
            memory="32g",
            timeout=14400
        )
    },
    {
        "name": "postprocessing",
        "backend": LocalConfig(timeout=3600),
        "command": "python postprocess.py"
    }
])

# Execute pipeline
results = pipeline.run()

API Reference#

For complete API documentation, see:

rompy.backends.config.BaseBackendConfig - Base configuration class
rompy.backends.config.LocalConfig - Local execution configuration
rompy.backends.config.DockerConfig - Docker execution configuration
rompy.run - Run backend implementations
rompy.backends.postprocessors - Postprocessor implementations
Backend Reference - Comprehensive technical reference

The backend system provides a robust, type-safe foundation for model execution while maintaining flexibility for different deployment scenarios. From simple local development to complex containerized production environments, the backend system adapts to your needs while ensuring consistent, reproducible results.

Table of Contents

Backend Systems#

Overview#

Key Components#

Quick Start#

Configuration Types#

LocalConfig - Local System Execution#

DockerConfig - Container Execution#

Using Backend Configurations#

With ModelRun#

From Configuration Files#

Command Line Interface#

Configuration Management#

Model Execution#

Configuration Examples#

Environment-Specific Configurations#

Resource-Based Configurations#

Validation and Error Handling#

Type Safety#

Configuration Validation#

Best Practices#

Configuration Management#

Resource Planning#

Security Considerations#

Troubleshooting#

Common Issues#

Debugging Configuration Issues#

Getting Help#

Advanced Usage#

Custom Backend Configurations#

Postprocessors#

Schema Generation#

Integration Examples#

Complete Workflow Example#

Pipeline Integration#

API Reference#

This Page