Backend System Examples¶
This notebook demonstrates the usage of rompy's backend system for running models, processing outputs, and orchestrating complete workflows.
The backend system provides three types of components:
- Run Backends: Execute models in different environments
- Postprocessors: Process model outputs after execution
- Pipeline Backends: Orchestrate complete workflows
In [14]:
Copied!
from datetime import datetime
from pathlib import Path
import tempfile
from rompy.model import ModelRun
from rompy.core.config import BaseConfig
from rompy.core.time import TimeRange
from rompy.logging import get_logger
# Configure logging to see backend activity
logger = get_logger(__name__)
from datetime import datetime
from pathlib import Path
import tempfile
from rompy.model import ModelRun
from rompy.core.config import BaseConfig
from rompy.core.time import TimeRange
from rompy.logging import get_logger
# Configure logging to see backend activity
logger = get_logger(__name__)
Setup: Create a Basic Model Run¶
First, let's create a basic model run that we can use to demonstrate the different backends.
In [15]:
Copied!
# Create a temporary directory for outputs
output_dir = tempfile.mkdtemp(prefix="rompy_backend_demo_")
print(f"Output directory: {output_dir}")
# Create a model run
model = ModelRun(
run_id="backend_demo",
period=TimeRange(
start=datetime(2023, 1, 1, 0),
end=datetime(2023, 1, 1, 6),
interval="1H"
),
output_dir=output_dir,
config=BaseConfig(arg1="demo", arg2="backend_test"),
delete_existing=True
)
print(f"Model run created: {model.run_id}")
print(f"Period: {model.period.start} to {model.period.end}")
# Create a temporary directory for outputs
output_dir = tempfile.mkdtemp(prefix="rompy_backend_demo_")
print(f"Output directory: {output_dir}")
# Create a model run
model = ModelRun(
run_id="backend_demo",
period=TimeRange(
start=datetime(2023, 1, 1, 0),
end=datetime(2023, 1, 1, 6),
interval="1H"
),
output_dir=output_dir,
config=BaseConfig(arg1="demo", arg2="backend_test"),
delete_existing=True
)
print(f"Model run created: {model.run_id}")
print(f"Period: {model.period.start} to {model.period.end}")
Output directory: /tmp/rompy_backend_demo_v01nrn0z Model run created: backend_demo Period: 2023-01-01 00:00:00 to 2023-01-01 06:00:00
Discovering Available Backends¶
Let's see what backends are available in the current environment.
In [16]:
Copied!
from rompy.model import RUN_BACKENDS, POSTPROCESSORS, PIPELINE_BACKENDS
print("Available Run Backends:")
for name, backend_class in RUN_BACKENDS.items():
print(f" - {name}: {backend_class.__name__}")
print("\nAvailable Postprocessors:")
for name, processor_class in POSTPROCESSORS.items():
print(f" - {name}: {processor_class.__name__}")
print("\nAvailable Pipeline Backends:")
for name, pipeline_class in PIPELINE_BACKENDS.items():
print(f" - {name}: {pipeline_class.__name__}")
from rompy.model import RUN_BACKENDS, POSTPROCESSORS, PIPELINE_BACKENDS
print("Available Run Backends:")
for name, backend_class in RUN_BACKENDS.items():
print(f" - {name}: {backend_class.__name__}")
print("\nAvailable Postprocessors:")
for name, processor_class in POSTPROCESSORS.items():
print(f" - {name}: {processor_class.__name__}")
print("\nAvailable Pipeline Backends:")
for name, pipeline_class in PIPELINE_BACKENDS.items():
print(f" - {name}: {pipeline_class.__name__}")
Available Run Backends: - docker: DockerRunBackend - local: LocalRunBackend Available Postprocessors: - noop: NoopPostprocessor Available Pipeline Backends: - local: LocalPipelineBackend
Run Backends¶
Run backends execute the model in different environments. Let's demonstrate the available backends.
Local Backend¶
The local backend runs the model directly on the current system.
In [17]:
Copied!
from rompy.backends.config import LocalConfig
# Run with local backend
print("Running with local backend...")
local_config = LocalConfig(
command="echo 'Hello from local backend'", # Replace with your actual command
timeout=3600 # Optional: timeout in seconds
)
success = model.run(backend=local_config)
print(f"Run successful: {success}")
# Check what files were created
output_path = Path(output_dir) / model.run_id
if output_path.exists():
files = list(output_path.rglob("*"))
print(f"\nFiles created: {len(files)}")
for file in files[:5]: # Show first 5 files
print(f" - {file.relative_to(output_path)}")
if len(files) > 5:
print(f" ... and {len(files) - 5} more")
from rompy.backends.config import LocalConfig
# Run with local backend
print("Running with local backend...")
local_config = LocalConfig(
command="echo 'Hello from local backend'", # Replace with your actual command
timeout=3600 # Optional: timeout in seconds
)
success = model.run(backend=local_config)
print(f"Run successful: {success}")
# Check what files were created
output_path = Path(output_dir) / model.run_id
if output_path.exists():
files = list(output_path.rglob("*"))
print(f"\nFiles created: {len(files)}")
for file in files[:5]: # Show first 5 files
print(f" - {file.relative_to(output_path)}")
if len(files) > 5:
print(f" ... and {len(files) - 5} more")
2025-08-25 22:37:23 [INFO] rompy.run : Starting local execution for run_id: backend_demo 2025-08-25 22:37:23 [WARNING] rompy.run : ! No workspace_dir provided, generating files (this may cause double generation in pipeline)
2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┃ MODEL RUN CONFIGURATION ┃ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : Run ID : backend_demo 2025-08-25 22:37:23 [INFO] rompy.model : Model Type : BaseConfig 2025-08-25 22:37:23 [INFO] rompy.model : Start Time : 2023-01-01T00:00:00 2025-08-25 22:37:23 [INFO] rompy.model : End Time : 2023-01-01T06:00:00 2025-08-25 22:37:23 [INFO] rompy.model : Duration : 6 hours 2025-08-25 22:37:23 [INFO] rompy.model : Time Interval : 1:00:00 2025-08-25 22:37:23 [INFO] rompy.model : Output Directory : /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : 2025-08-25 22:37:23 [INFO] root : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] root : ┃ MODEL CONFIGURATION (BaseConfig) ┃ 2025-08-25 22:37:23 [INFO] root : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] root : 2025-08-25 22:37:23 [INFO] rompy.model : BaseConfig: 2025-08-25 22:37:23 [INFO] rompy.model : model_type: base 2025-08-25 22:37:23 [INFO] rompy.model : template: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:23 [INFO] rompy.model : checkout: main 2025-08-25 22:37:23 [INFO] rompy.model : arg1: demo 2025-08-25 22:37:23 [INFO] rompy.model : arg2: backend_test 2025-08-25 22:37:23 [INFO] rompy.model : 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┃ STARTING MODEL GENERATION ┃ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : Preparing input files in /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:23 [INFO] rompy.model : Processing model configuration... 2025-08-25 22:37:23 [INFO] rompy.model : Running configuration callable... 2025-08-25 22:37:23 [INFO] rompy.model : Rendering model templates to /tmp/rompy_backend_demo_v01nrn0z/backend_demo... 2025-08-25 22:37:23 [INFO] rompy.core.render : Template source: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:23 [INFO] rompy.core.render : Output directory: /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:23 [INFO] rompy.core.render : Using template version: main 2025-08-25 22:37:23 [INFO] rompy.core.render : • Locating template repository... 2025-08-25 22:37:23 [INFO] rompy.core.render : Template repository located at: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:23 [INFO] rompy.core.render : • Generating files from template... 2025-08-25 22:37:23 [INFO] rompy.core.render : • Rendering time: 0.01 seconds 2025-08-25 22:37:23 [INFO] rompy.core.render : • Total process time: 0.01 seconds 2025-08-25 22:37:23 [INFO] rompy.core.render : • Files created: 3 2025-08-25 22:37:23 [INFO] rompy.core.render : • Output location: /tmp/rompy_backend_demo_v01nrn0z/backend_demo 2025-08-25 22:37:23 [INFO] rompy.model : 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┃ MODEL GENERATION COMPLETE ┃ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : Model files generated at: /tmp/rompy_backend_demo_v01nrn0z/backend_demo 2025-08-25 22:37:23 [INFO] rompy.run : Model inputs generated in: /tmp/rompy_backend_demo_v01nrn0z/backend_demo 2025-08-25 22:37:23 [INFO] rompy.run : Executing command: echo 'Hello from local backend' 2025-08-25 22:37:23 [INFO] rompy.run : Command stdout: Hello from local backend 2025-08-25 22:37:23 [INFO] rompy.run : Local execution completed successfully for run_id: backend_demo
Running with local backend... Run successful: True Files created: 5 - INPUT - outputs - datasets - outputs/readme.md - datasets/readme.md
Docker Backend (if available)¶
The Docker backend runs the model inside a Docker container. This requires Docker to be installed and running.
In [18]:
Copied!
from rompy.backends.config import DockerConfig
# Check if Docker backend is available
if "docker" in RUN_BACKENDS:
print("Docker backend is available!")
try:
print("\nAttempting to run with Docker backend...")
print("(This may fail if Docker is not available or configured)")
# Create a DockerConfig instance
docker_config = DockerConfig(
image="ubuntu:20.04", # Simple base image for demonstration
executable="/bin/echo 'Hello from Docker container'", # Simple command that will succeed
env_vars={"TEST_VAR": "backend_demo"},
volumes=[f"{str(Path(output_dir).absolute())}:/data"] # Mount the output directory
)
# Pass the config instance to model.run()
success = model.run(backend=docker_config)
print(f"Docker run successful: {success}")
except Exception as e:
print(f"Docker run failed: {e}")
else:
print("Docker backend is not available in this environment.")
from rompy.backends.config import DockerConfig
# Check if Docker backend is available
if "docker" in RUN_BACKENDS:
print("Docker backend is available!")
try:
print("\nAttempting to run with Docker backend...")
print("(This may fail if Docker is not available or configured)")
# Create a DockerConfig instance
docker_config = DockerConfig(
image="ubuntu:20.04", # Simple base image for demonstration
executable="/bin/echo 'Hello from Docker container'", # Simple command that will succeed
env_vars={"TEST_VAR": "backend_demo"},
volumes=[f"{str(Path(output_dir).absolute())}:/data"] # Mount the output directory
)
# Pass the config instance to model.run()
success = model.run(backend=docker_config)
print(f"Docker run successful: {success}")
except Exception as e:
print(f"Docker run failed: {e}")
else:
print("Docker backend is not available in this environment.")
2025-08-25 22:37:23 [WARNING] rompy.run.docker : ! No workspace_dir provided, generating files (this may cause double generation in pipeline) 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┃ MODEL RUN CONFIGURATION ┃ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : Run ID : backend_demo 2025-08-25 22:37:23 [INFO] rompy.model : Model Type : BaseConfig 2025-08-25 22:37:23 [INFO] rompy.model : Start Time : 2023-01-01T00:00:00 2025-08-25 22:37:23 [INFO] rompy.model : End Time : 2023-01-01T06:00:00 2025-08-25 22:37:23 [INFO] rompy.model : Duration : 6 hours 2025-08-25 22:37:23 [INFO] rompy.model : Time Interval : 1:00:00 2025-08-25 22:37:23 [INFO] rompy.model : Output Directory : /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : 2025-08-25 22:37:23 [INFO] root : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] root : ┃ MODEL CONFIGURATION (BaseConfig) ┃ 2025-08-25 22:37:23 [INFO] root : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] root : 2025-08-25 22:37:23 [INFO] rompy.model : BaseConfig: 2025-08-25 22:37:23 [INFO] rompy.model : model_type: base 2025-08-25 22:37:23 [INFO] rompy.model : template: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:23 [INFO] rompy.model : checkout: main 2025-08-25 22:37:23 [INFO] rompy.model : arg1: demo 2025-08-25 22:37:23 [INFO] rompy.model : arg2: backend_test 2025-08-25 22:37:23 [INFO] rompy.model : 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┃ STARTING MODEL GENERATION ┃ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : Preparing input files in /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:23 [INFO] rompy.model : Processing model configuration... 2025-08-25 22:37:23 [INFO] rompy.model : Running configuration callable... 2025-08-25 22:37:23 [INFO] rompy.model : Rendering model templates to /tmp/rompy_backend_demo_v01nrn0z/backend_demo... 2025-08-25 22:37:23 [INFO] rompy.core.render : Template source: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:23 [INFO] rompy.core.render : Output directory: /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:23 [INFO] rompy.core.render : Using template version: main 2025-08-25 22:37:23 [INFO] rompy.core.render : • Locating template repository... 2025-08-25 22:37:23 [INFO] rompy.core.render : Template repository located at: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:23 [INFO] rompy.core.render : • Generating files from template... 2025-08-25 22:37:23 [INFO] rompy.core.render : • Rendering time: 0.01 seconds 2025-08-25 22:37:23 [INFO] rompy.core.render : • Total process time: 0.01 seconds 2025-08-25 22:37:23 [INFO] rompy.core.render : • Files created: 3 2025-08-25 22:37:23 [INFO] rompy.core.render : • Output location: /tmp/rompy_backend_demo_v01nrn0z/backend_demo 2025-08-25 22:37:23 [INFO] rompy.model : 2025-08-25 22:37:23 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:23 [INFO] rompy.model : ┃ MODEL GENERATION COMPLETE ┃ 2025-08-25 22:37:23 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:23 [INFO] rompy.model : Model files generated at: /tmp/rompy_backend_demo_v01nrn0z/backend_demo 2025-08-25 22:37:23 [INFO] rompy.run.docker : Using provided Docker image: ubuntu:20.04 2025-08-25 22:37:23 [INFO] rompy.run.docker : Executing: docker run --rm --user root -e TEST_VAR=backend_demo -v /tmp/rompy_backend_demo_v01nrn0z/backend_demo:/app/run_id:Z -v /tmp/rompy_backend_demo_v01nrn0z:/data ubuntu:20.04 bash -c cd /app/run_id && echo 'Directory contents:' && ls -la && echo 'Executing model...' && /bin/echo 'Hello from Docker container'
Docker backend is available! Attempting to run with Docker backend... (This may fail if Docker is not available or configured)
2025-08-25 22:37:23 [INFO] rompy.run.docker : Docker stdout: Directory contents: total 4 drwxr-xr-x. 4 1001 1001 100 Aug 25 12:37 . drwxr-xr-x. 1 root root 12 Aug 25 12:37 .. -rw-r--r--. 1 1001 1001 354 Aug 25 12:37 INPUT drwxr-xr-x. 2 1001 1001 60 Aug 25 12:37 datasets drwxr-xr-x. 2 1001 1001 60 Aug 25 12:37 outputs Executing model... Hello from Docker container 2025-08-25 22:37:23 [INFO] rompy.run.docker : Model run completed successfully with exit code 0
Docker run successful: True
Postprocessors¶
Postprocessors handle model outputs after execution.
No-op Postprocessor¶
The no-op postprocessor is a placeholder that performs no operations.
In [19]:
Copied!
# Use the no-op postprocessor
print("Running no-op postprocessor...")
results = model.postprocess(processor="noop")
print(f"Postprocessing results: {results}")
# Use the no-op postprocessor
print("Running no-op postprocessor...")
results = model.postprocess(processor="noop")
print(f"Postprocessing results: {results}")
2025-08-25 22:37:23 [INFO] rompy.postprocess : Starting no-op postprocessing for run_id: backend_demo 2025-08-25 22:37:23 [INFO] rompy.postprocess : Found 3 output files in /tmp/rompy_backend_demo_v01nrn0z/backend_demo 2025-08-25 22:37:23 [INFO] rompy.postprocess : No-op postprocessing completed for run_id: backend_demo
Running no-op postprocessor...
Postprocessing results: {'success': True, 'message': 'No postprocessing requested - validation only', 'run_id': 'backend_demo', 'output_dir': '/tmp/rompy_backend_demo_v01nrn0z/backend_demo', 'validated': True}
Custom Postprocessor Example¶
Let's create a simple custom postprocessor to demonstrate how to extend the system.
In [20]:
Copied!
from typing import Dict, Any
class FileCountPostprocessor:
"""A simple postprocessor that counts files in the output directory."""
def process(self, model_run, **kwargs) -> Dict[str, Any]:
"""Count files in the model output directory."""
output_path = Path(model_run.output_dir) / model_run.run_id
if not output_path.exists():
return {
"success": False,
"message": "Output directory does not exist",
"file_count": 0
}
# Count files
file_count = sum(1 for f in output_path.rglob("*") if f.is_file())
total_size = sum(f.stat().st_size for f in output_path.rglob("*") if f.is_file())
return {
"success": True,
"message": f"Found {file_count} files",
"file_count": file_count,
"total_size_bytes": total_size,
"output_directory": str(output_path)
}
# Use our custom postprocessor
print("Running custom file count postprocessor...")
custom_processor = FileCountPostprocessor()
results = custom_processor.process(model)
print("Custom postprocessing results:")
for key, value in results.items():
print(f" {key}: {value}")
from typing import Dict, Any
class FileCountPostprocessor:
"""A simple postprocessor that counts files in the output directory."""
def process(self, model_run, **kwargs) -> Dict[str, Any]:
"""Count files in the model output directory."""
output_path = Path(model_run.output_dir) / model_run.run_id
if not output_path.exists():
return {
"success": False,
"message": "Output directory does not exist",
"file_count": 0
}
# Count files
file_count = sum(1 for f in output_path.rglob("*") if f.is_file())
total_size = sum(f.stat().st_size for f in output_path.rglob("*") if f.is_file())
return {
"success": True,
"message": f"Found {file_count} files",
"file_count": file_count,
"total_size_bytes": total_size,
"output_directory": str(output_path)
}
# Use our custom postprocessor
print("Running custom file count postprocessor...")
custom_processor = FileCountPostprocessor()
results = custom_processor.process(model)
print("Custom postprocessing results:")
for key, value in results.items():
print(f" {key}: {value}")
Running custom file count postprocessor... Custom postprocessing results: success: True message: Found 3 files file_count: 3 total_size_bytes: 468 output_directory: /tmp/rompy_backend_demo_v01nrn0z/backend_demo
Pipeline Backends¶
Pipeline backends orchestrate the complete workflow from model generation through execution to postprocessing.
Local Pipeline¶
The local pipeline backend executes the complete workflow locally.
In [21]:
Copied!
# Create a fresh model run for the pipeline demo
pipeline_model = ModelRun(
run_id="pipeline_demo",
period=TimeRange(
start=datetime(2023, 1, 1, 0),
end=datetime(2023, 1, 1, 3),
interval="1H"
),
output_dir=output_dir,
config=BaseConfig(arg1="pipeline", arg2="demo"),
delete_existing=True
)
# Run the complete pipeline
print("Running complete pipeline...")
results = pipeline_model.pipeline(
pipeline_backend="local",
run_backend="local",
processor="noop"
)
print("\nPipeline results:")
for key, value in results.items():
print(f" {key}: {value}")
# Create a fresh model run for the pipeline demo
pipeline_model = ModelRun(
run_id="pipeline_demo",
period=TimeRange(
start=datetime(2023, 1, 1, 0),
end=datetime(2023, 1, 1, 3),
interval="1H"
),
output_dir=output_dir,
config=BaseConfig(arg1="pipeline", arg2="demo"),
delete_existing=True
)
# Run the complete pipeline
print("Running complete pipeline...")
results = pipeline_model.pipeline(
pipeline_backend="local",
run_backend="local",
processor="noop"
)
print("\nPipeline results:")
for key, value in results.items():
print(f" {key}: {value}")
2025-08-25 22:37:24 [INFO] rompy.pipeline : Starting pipeline execution for run_id: pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.pipeline : Pipeline configuration: run_backend='local', processor='noop' 2025-08-25 22:37:24 [INFO] rompy.pipeline : Stage 1: Generating input files for pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ MODEL RUN CONFIGURATION ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Run ID : pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.model : Model Type : BaseConfig 2025-08-25 22:37:24 [INFO] rompy.model : Start Time : 2023-01-01T00:00:00 2025-08-25 22:37:24 [INFO] rompy.model : End Time : 2023-01-01T03:00:00 2025-08-25 22:37:24 [INFO] rompy.model : Duration : 3 hours 2025-08-25 22:37:24 [INFO] rompy.model : Time Interval : 1:00:00 2025-08-25 22:37:24 [INFO] rompy.model : Output Directory : /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] root : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] root : ┃ MODEL CONFIGURATION (BaseConfig) ┃ 2025-08-25 22:37:24 [INFO] root : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] root :
2025-08-25 22:37:24 [INFO] rompy.model : BaseConfig: 2025-08-25 22:37:24 [INFO] rompy.model : model_type: base 2025-08-25 22:37:24 [INFO] rompy.model : template: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.model : checkout: main 2025-08-25 22:37:24 [INFO] rompy.model : arg1: pipeline 2025-08-25 22:37:24 [INFO] rompy.model : arg2: demo 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ STARTING MODEL GENERATION ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Preparing input files in /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.model : Processing model configuration... 2025-08-25 22:37:24 [INFO] rompy.model : Running configuration callable... 2025-08-25 22:37:24 [INFO] rompy.model : Rendering model templates to /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo... 2025-08-25 22:37:24 [INFO] rompy.core.render : Template source: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.core.render : Output directory: /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.core.render : Using template version: main 2025-08-25 22:37:24 [INFO] rompy.core.render : • Locating template repository... 2025-08-25 22:37:24 [INFO] rompy.core.render : Template repository located at: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.core.render : • Generating files from template... 2025-08-25 22:37:24 [INFO] rompy.core.render : • Rendering time: 0.01 seconds 2025-08-25 22:37:24 [INFO] rompy.core.render : • Total process time: 0.01 seconds 2025-08-25 22:37:24 [INFO] rompy.core.render : • Files created: 3 2025-08-25 22:37:24 [INFO] rompy.core.render : • Output location: /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ MODEL GENERATION COMPLETE ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Model files generated at: /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.pipeline : Input files generated successfully in: /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.pipeline : Stage 2: Running model using local backend 2025-08-25 22:37:24 [INFO] rompy.run : Starting local execution for run_id: pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.run : Using provided workspace directory: /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo 2025-08-25 22:37:24 [WARNING] rompy.run : ! Model config does not have a run method. Nothing to execute. 2025-08-25 22:37:24 [INFO] rompy.run : Local execution completed successfully for run_id: pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.pipeline : Model run completed successfully 2025-08-25 22:37:24 [INFO] rompy.pipeline : Stage 3: Postprocessing with noop 2025-08-25 22:37:24 [INFO] rompy.postprocess : Starting no-op postprocessing for run_id: pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.postprocess : Found 3 output files in /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.postprocess : No-op postprocessing completed for run_id: pipeline_demo 2025-08-25 22:37:24 [INFO] rompy.pipeline : Postprocessing completed 2025-08-25 22:37:24 [INFO] rompy.pipeline : Pipeline execution completed successfully for run_id: pipeline_demo
Running complete pipeline...
Pipeline results:
success: True
run_id: pipeline_demo
stages_completed: ['generate', 'run', 'postprocess']
run_backend: local
processor: noop
staging_dir: /tmp/rompy_backend_demo_v01nrn0z/pipeline_demo
run_success: True
postprocess_results: {'success': True, 'message': 'No postprocessing requested - validation only', 'run_id': 'pipeline_demo', 'output_dir': '/tmp/rompy_backend_demo_v01nrn0z/pipeline_demo', 'validated': True}
message: Pipeline completed successfully
Error Handling¶
The backend system provides clear error messages when backends are not found.
In [22]:
Copied!
from rompy.backends.config import LocalConfig
# Try to use a non-existent run backend
try:
# Instead of passing a string, we need to create a BackendConfig instance
# with an invalid backend type
class InvalidBackendConfig:
pass
model.run(backend=InvalidBackendConfig())
except TypeError as e:
print(f"Run backend error: {e}")
# Try to use a non-existent postprocessor
try:
model.postprocess(processor="invalid")
except ValueError as e:
print(f"Postprocessor error: {e}")
# Try to use a non-existent pipeline backend
try:
model.pipeline(pipeline_backend="unknown")
except ValueError as e:
print(f"Pipeline backend error: {e}")
from rompy.backends.config import LocalConfig
# Try to use a non-existent run backend
try:
# Instead of passing a string, we need to create a BackendConfig instance
# with an invalid backend type
class InvalidBackendConfig:
pass
model.run(backend=InvalidBackendConfig())
except TypeError as e:
print(f"Run backend error: {e}")
# Try to use a non-existent postprocessor
try:
model.postprocess(processor="invalid")
except ValueError as e:
print(f"Postprocessor error: {e}")
# Try to use a non-existent pipeline backend
try:
model.pipeline(pipeline_backend="unknown")
except ValueError as e:
print(f"Pipeline backend error: {e}")
Run backend error: Backend must be a subclass of BaseBackendConfig, got InvalidBackendConfig Postprocessor error: Unknown postprocessor: invalid. Available processors: noop Pipeline backend error: Unknown pipeline backend: unknown. Available backends: local
Complete Workflow Example¶
Let's put it all together with a complete workflow that demonstrates the full capability of the backend system.
In [23]:
Copied!
from rompy.backends.config import LocalConfig
# Create a complete workflow model
workflow_model = ModelRun(
run_id="complete_workflow",
period=TimeRange(
start=datetime(2023, 1, 1, 0),
end=datetime(2023, 1, 1, 6),
interval="1H"
),
output_dir=output_dir,
config=BaseConfig(arg1="complete", arg2="workflow"),
delete_existing=True
)
print("1. Generating model inputs...")
workflow_model.generate()
print(" ✓ Inputs generated")
# Create a LocalConfig instance
local_config = LocalConfig(
command="echo 'Running model simulation'", # Replace with your actual command
working_dir=str(Path(output_dir) / "complete_workflow")
)
# Step 2: Run the model
print("\n2. Running model...")
run_success = workflow_model.run(backend=local_config)
print(f" ✓ Model run: {'Success' if run_success else 'Failed'}")
# Step 3: Process outputs
print("\n3. Processing outputs...")
process_results = workflow_model.postprocess(processor="noop")
print(" ✓ Output processing complete")
# Step 4: Show results
print("\n4. Results summary:")
print(f" - Run ID: {workflow_model.run_id}")
print(f" - Output directory: {Path(output_dir) / workflow_model.run_id}")
print(f" - Run successful: {run_success}")
print(f" - Process results: {process_results}")
from rompy.backends.config import LocalConfig
# Create a complete workflow model
workflow_model = ModelRun(
run_id="complete_workflow",
period=TimeRange(
start=datetime(2023, 1, 1, 0),
end=datetime(2023, 1, 1, 6),
interval="1H"
),
output_dir=output_dir,
config=BaseConfig(arg1="complete", arg2="workflow"),
delete_existing=True
)
print("1. Generating model inputs...")
workflow_model.generate()
print(" ✓ Inputs generated")
# Create a LocalConfig instance
local_config = LocalConfig(
command="echo 'Running model simulation'", # Replace with your actual command
working_dir=str(Path(output_dir) / "complete_workflow")
)
# Step 2: Run the model
print("\n2. Running model...")
run_success = workflow_model.run(backend=local_config)
print(f" ✓ Model run: {'Success' if run_success else 'Failed'}")
# Step 3: Process outputs
print("\n3. Processing outputs...")
process_results = workflow_model.postprocess(processor="noop")
print(" ✓ Output processing complete")
# Step 4: Show results
print("\n4. Results summary:")
print(f" - Run ID: {workflow_model.run_id}")
print(f" - Output directory: {Path(output_dir) / workflow_model.run_id}")
print(f" - Run successful: {run_success}")
print(f" - Process results: {process_results}")
2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ MODEL RUN CONFIGURATION ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Run ID : complete_workflow 2025-08-25 22:37:24 [INFO] rompy.model : Model Type : BaseConfig 2025-08-25 22:37:24 [INFO] rompy.model : Start Time : 2023-01-01T00:00:00 2025-08-25 22:37:24 [INFO] rompy.model : End Time : 2023-01-01T06:00:00 2025-08-25 22:37:24 [INFO] rompy.model : Duration : 6 hours 2025-08-25 22:37:24 [INFO] rompy.model : Time Interval : 1:00:00 2025-08-25 22:37:24 [INFO] rompy.model : Output Directory : /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] root : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] root : ┃ MODEL CONFIGURATION (BaseConfig) ┃ 2025-08-25 22:37:24 [INFO] root : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] root : 2025-08-25 22:37:24 [INFO] rompy.model : BaseConfig: 2025-08-25 22:37:24 [INFO] rompy.model : model_type: base 2025-08-25 22:37:24 [INFO] rompy.model : template: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.model : checkout: main 2025-08-25 22:37:24 [INFO] rompy.model : arg1: complete 2025-08-25 22:37:24 [INFO] rompy.model : arg2: workflow 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ STARTING MODEL GENERATION ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Preparing input files in /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.model : Processing model configuration... 2025-08-25 22:37:24 [INFO] rompy.model : Running configuration callable... 2025-08-25 22:37:24 [INFO] rompy.model : Rendering model templates to /tmp/rompy_backend_demo_v01nrn0z/complete_workflow... 2025-08-25 22:37:24 [INFO] rompy.core.render : Template source: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.core.render : Output directory: /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.core.render : Using template version: main 2025-08-25 22:37:24 [INFO] rompy.core.render : • Locating template repository... 2025-08-25 22:37:24 [INFO] rompy.core.render : Template repository located at: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.core.render : • Generating files from template... 2025-08-25 22:37:24 [INFO] rompy.core.render : • Rendering time: 0.01 seconds 2025-08-25 22:37:24 [INFO] rompy.core.render : • Total process time: 0.01 seconds 2025-08-25 22:37:24 [INFO] rompy.core.render : • Files created: 3 2025-08-25 22:37:24 [INFO] rompy.core.render : • Output location: /tmp/rompy_backend_demo_v01nrn0z/complete_workflow 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ MODEL GENERATION COMPLETE ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Model files generated at: /tmp/rompy_backend_demo_v01nrn0z/complete_workflow 2025-08-25 22:37:24 [INFO] rompy.run : Starting local execution for run_id: complete_workflow 2025-08-25 22:37:24 [WARNING] rompy.run : ! No workspace_dir provided, generating files (this may cause double generation in pipeline) 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ MODEL RUN CONFIGURATION ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Run ID : complete_workflow 2025-08-25 22:37:24 [INFO] rompy.model : Model Type : BaseConfig 2025-08-25 22:37:24 [INFO] rompy.model : Start Time : 2023-01-01T00:00:00 2025-08-25 22:37:24 [INFO] rompy.model : End Time : 2023-01-01T06:00:00 2025-08-25 22:37:24 [INFO] rompy.model : Duration : 6 hours 2025-08-25 22:37:24 [INFO] rompy.model : Time Interval : 1:00:00 2025-08-25 22:37:24 [INFO] rompy.model : Output Directory : /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] root : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] root : ┃ MODEL CONFIGURATION (BaseConfig) ┃ 2025-08-25 22:37:24 [INFO] root : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] root : 2025-08-25 22:37:24 [INFO] rompy.model : BaseConfig: 2025-08-25 22:37:24 [INFO] rompy.model : model_type: base 2025-08-25 22:37:24 [INFO] rompy.model : template: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.model : checkout: main 2025-08-25 22:37:24 [INFO] rompy.model : arg1: complete 2025-08-25 22:37:24 [INFO] rompy.model : arg2: workflow 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ STARTING MODEL GENERATION ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Preparing input files in /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.model : Processing model configuration... 2025-08-25 22:37:24 [INFO] rompy.model : Running configuration callable... 2025-08-25 22:37:24 [INFO] rompy.model : Rendering model templates to /tmp/rompy_backend_demo_v01nrn0z/complete_workflow... 2025-08-25 22:37:24 [INFO] rompy.core.render : Template source: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.core.render : Output directory: /tmp/rompy_backend_demo_v01nrn0z 2025-08-25 22:37:24 [INFO] rompy.core.render : Using template version: main 2025-08-25 22:37:24 [INFO] rompy.core.render : • Locating template repository... 2025-08-25 22:37:24 [INFO] rompy.core.render : Template repository located at: /home/tdurrant/source/rompy/rompy/rompy/templates/base 2025-08-25 22:37:24 [INFO] rompy.core.render : • Generating files from template... 2025-08-25 22:37:24 [INFO] rompy.core.render : • Rendering time: 0.01 seconds 2025-08-25 22:37:24 [INFO] rompy.core.render : • Total process time: 0.01 seconds 2025-08-25 22:37:24 [INFO] rompy.core.render : • Files created: 3 2025-08-25 22:37:24 [INFO] rompy.core.render : • Output location: /tmp/rompy_backend_demo_v01nrn0z/complete_workflow 2025-08-25 22:37:24 [INFO] rompy.model : 2025-08-25 22:37:24 [INFO] rompy.model : ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ 2025-08-25 22:37:24 [INFO] rompy.model : ┃ MODEL GENERATION COMPLETE ┃ 2025-08-25 22:37:24 [INFO] rompy.model : ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 2025-08-25 22:37:24 [INFO] rompy.model : Model files generated at: /tmp/rompy_backend_demo_v01nrn0z/complete_workflow 2025-08-25 22:37:24 [INFO] rompy.run : Model inputs generated in: /tmp/rompy_backend_demo_v01nrn0z/complete_workflow 2025-08-25 22:37:24 [INFO] rompy.run : Executing command: echo 'Running model simulation' 2025-08-25 22:37:24 [INFO] rompy.run : Command stdout: Running model simulation 2025-08-25 22:37:24 [INFO] rompy.run : Local execution completed successfully for run_id: complete_workflow 2025-08-25 22:37:24 [INFO] rompy.postprocess : Starting no-op postprocessing for run_id: complete_workflow 2025-08-25 22:37:24 [INFO] rompy.postprocess : Found 3 output files in /tmp/rompy_backend_demo_v01nrn0z/complete_workflow 2025-08-25 22:37:24 [INFO] rompy.postprocess : No-op postprocessing completed for run_id: complete_workflow
1. Generating model inputs...
✓ Inputs generated
2. Running model...
✓ Model run: Success
3. Processing outputs...
✓ Output processing complete
4. Results summary:
- Run ID: complete_workflow
- Output directory: /tmp/rompy_backend_demo_v01nrn0z/complete_workflow
- Run successful: True
- Process results: {'success': True, 'message': 'No postprocessing requested - validation only', 'run_id': 'complete_workflow', 'output_dir': '/tmp/rompy_backend_demo_v01nrn0z/complete_workflow', 'validated': True}
Summary¶
This notebook demonstrated the key features of rompy's backend system:
- Run Backends: Execute models in different environments (local, Docker)
- Postprocessors: Process model outputs with custom logic
- Pipeline Backends: Orchestrate complete workflows
- Extensibility: Easy to create and register custom backends
- Error Handling: Clear error messages for debugging
The entry point-based architecture makes it easy to extend rompy with custom backends without modifying the core library. Simply implement the required interface and register your backend through Python entry points.
In [24]:
Copied!
# Cleanup: Remove temporary files
import shutil
try:
shutil.rmtree(output_dir)
print(f"Cleaned up temporary directory: {output_dir}")
except Exception as e:
print(f"Could not clean up {output_dir}: {e}")
# Cleanup: Remove temporary files
import shutil
try:
shutil.rmtree(output_dir)
print(f"Cleaned up temporary directory: {output_dir}")
except Exception as e:
print(f"Could not clean up {output_dir}: {e}")
Cleaned up temporary directory: /tmp/rompy_backend_demo_v01nrn0z
In [ ]:
Copied!