Data sources¶

What this shows: every source type rompy-xbeach can read, and which forcing classes use each one.

Prerequisites: Tutorial 3: Bathymetry from your data.

You will learn:

• how to open gridded data from GeoTIFF, NetCDF, XYZ point clouds, intake catalogues and in-memory datasets
• which sources hold wave spectra, tidal constituents and plain timeseries
• which sources come from rompy and which from rompy-xbeach

Data used: several files from the data/ folder.

Setup¶

In [1]:

from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd
import xarray as xr
from rompy.logging import config as logging_config

logging_config.update(level="WARNING")  # rompy logs every step, show warnings only

DATA_DIR = Path("../data")

from pathlib import Path import matplotlib.pyplot as plt import pandas as pd import xarray as xr from rompy.logging import config as logging_config logging_config.update(level="WARNING") # rompy logs every step, show warnings only DATA_DIR = Path("../data")

How sources work¶

A source describes where data lives and how to open it. open() returns an xarray dataset. rompy defines the base sources. rompy-xbeach extends them to always carry a coordinate reference system (CRS), which is needed to interpolate onto a projected XBeach grid. In YAML, each source is selected by its model_type.

Source	Package	model_type	Data
SourceGeotiff	rompy-xbeach	geotiff	GeoTIFF rasters
SourceCRSFile	rompy-xbeach	file	NetCDF, Zarr or anything xarray opens
SourceXYZ	rompy-xbeach	xyz	x, y, z point clouds
SourceCRSIntake	rompy-xbeach	intake	datasets in an intake catalogue
SourceCRSDataset	rompy-xbeach	dataset	an xarray dataset in memory
SourceCRSWavespectra	rompy-xbeach	wavespectra	wave spectra files
SourceCRSOceantide	rompy-xbeach	oceantide	gridded tidal constituents
SourceTideConsPointCSV	rompy-xbeach	tide_cons_point_csv	constituents at one site
SourceTimeseriesCSV	rompy	csv	a timeseries in a CSV file
SourceTimeseriesDataFrame	rompy-binary-datasources	dataframe	a pandas dataframe

1. Gridded data¶

These sources feed XBeachBathy and the ...Grid and ...Station forcing classes.

GeoTIFF¶

The CRS is read from the file. band picks the raster band.

In [2]:

from rompy_xbeach.source import SourceGeotiff

source = SourceGeotiff(filename=DATA_DIR / "bathy.tif")
ds = source.open()
print(ds.rio.crs, dict(ds.sizes))

from rompy_xbeach.source import SourceGeotiff source = SourceGeotiff(filename=DATA_DIR / "bathy.tif") ds = source.open() print(ds.rio.crs, dict(ds.sizes))

EPSG:4326 {'x': 176, 'y': 180}

NetCDF and other xarray formats¶

kwargs are passed to xarray.open_dataset. NetCDF files rarely declare a CRS, so you state it, together with the names of the x and y dimensions.

In [3]:

from rompy_xbeach.source import SourceCRSFile

source = SourceCRSFile(
    uri=DATA_DIR / "bathy.nc",
    crs=4326,
    x_dim="x",
    y_dim="y",
    kwargs={"engine": "netcdf4"},
)
ds = source.open()
print(ds.rio.crs, dict(ds.sizes))

from rompy_xbeach.source import SourceCRSFile source = SourceCRSFile( uri=DATA_DIR / "bathy.nc", crs=4326, x_dim="x", y_dim="y", kwargs={"engine": "netcdf4"}, ) ds = source.open() print(ds.rio.crs, dict(ds.sizes))

EPSG:4326 {'y': 180, 'x': 176}

XYZ point clouds¶

Survey or LiDAR points are gridded at resolution res (in CRS units) when opened. read_csv_kwargs and griddata_kwargs control how the file is read and gridded.

In [4]:

from rompy_xbeach.source import SourceXYZ

source = SourceXYZ(
    filename=DATA_DIR / "bathy_xyz.zip",
    crs=4326,
    res=0.0005,
    xcol="easting",
    ycol="northing",
    zcol="elevation",
    read_csv_kwargs={"sep": "\t"},
)
ds = source.open()
ds.data.plot(cmap="terrain", vmin=-20, vmax=20, figsize=(5, 4))
plt.show()

from rompy_xbeach.source import SourceXYZ source = SourceXYZ( filename=DATA_DIR / "bathy_xyz.zip", crs=4326, res=0.0005, xcol="easting", ycol="northing", zcol="elevation", read_csv_kwargs={"sep": "\t"}, ) ds = source.open() ds.data.plot(cmap="terrain", vmin=-20, vmax=20, figsize=(5, 4)) plt.show()

Intake catalogues¶

An intake catalogue lists datasets by name, so configurations can refer to data without hard-coding file paths.

In [5]:

from rompy_xbeach.source import SourceCRSIntake

print((DATA_DIR / "catalog.yaml").read_text())

from rompy_xbeach.source import SourceCRSIntake print((DATA_DIR / "catalog.yaml").read_text())

sources:
    bathy_netcdf:
        driver: netcdf
        description: Test bathy in NetCDF format
        args:
            urlpath: '{{CATALOG_DIR}}/bathy.nc'
            engine: "netcdf4"
    bathy_geotiff:
        driver: netcdf
        description: Test bathy in GeoTIFF format
        args:
            urlpath: '{{CATALOG_DIR}}/bathy.tif'
            xarray_kwargs:
                engine: rasterio

In [6]:

source = SourceCRSIntake(
    catalog_uri=DATA_DIR / "catalog.yaml", dataset_id="bathy_netcdf", crs=4326
)
print(dict(source.open().sizes))

source = SourceCRSIntake( catalog_uri=DATA_DIR / "catalog.yaml", dataset_id="bathy_netcdf", crs=4326 ) print(dict(source.open().sizes))

{'y': 180, 'x': 176}

An xarray dataset in memory¶

Useful when data has been downloaded or pre-processed in the same script. This source cannot be written to YAML, since the data lives only in memory.

In [7]:

from rompy_xbeach.source import SourceCRSDataset

in_memory = xr.open_dataset(DATA_DIR / "bathy.nc")
source = SourceCRSDataset(obj=in_memory, crs=4326)
print(source.open().rio.crs)

from rompy_xbeach.source import SourceCRSDataset in_memory = xr.open_dataset(DATA_DIR / "bathy.nc") source = SourceCRSDataset(obj=in_memory, crs=4326) print(source.open().rio.crs)

EPSG:4326

2. Specialised sources¶

Wave spectra¶

SourceCRSWavespectra opens spectra with a wavespectra reader such as read_ww3, read_swan or read_era5. It is used by the BoundaryStationSpectra... classes.

In [8]:

from rompy_xbeach.source import SourceCRSWavespectra

source = SourceCRSWavespectra(
    uri=DATA_DIR / "ww3-spectra-20230101-short.nc", reader="read_ww3"
)
spectra = source.open()
print(dict(spectra.sizes))

from rompy_xbeach.source import SourceCRSWavespectra source = SourceCRSWavespectra( uri=DATA_DIR / "ww3-spectra-20230101-short.nc", reader="read_ww3" ) spectra = source.open() print(dict(spectra.sizes))

{'time': 9, 'site': 19, 'freq': 31, 'dir': 24}

Tidal constituents¶

SourceCRSOceantide reads tide model constituents with an oceantide reader, for TideConsGrid. SourceTideConsPointCSV reads amplitude and phase per constituent at one site, for TideConsPoint.

In [9]:

from rompy_xbeach.source import SourceCRSOceantide, SourceTideConsPointCSV

cons_dir = DATA_DIR / "swaus_tide_cons"
source = SourceCRSOceantide(
    reader="read_otis_binary",
    kwargs={
        "gfile": cons_dir / "grid_m2s2n2k2k1o1p1q1mmmf",
        "hfile": cons_dir / "h_m2s2n2k2k1o1p1q1mmmf",
        "ufile": cons_dir / "u_m2s2n2k2k1o1p1q1mmmf",
    },
    crs=4326,
)
print("Gridded constituents:", source.open().con.values)

source = SourceTideConsPointCSV(filename=DATA_DIR / "tide_cons_station.csv")
source.open()

from rompy_xbeach.source import SourceCRSOceantide, SourceTideConsPointCSV cons_dir = DATA_DIR / "swaus_tide_cons" source = SourceCRSOceantide( reader="read_otis_binary", kwargs={ "gfile": cons_dir / "grid_m2s2n2k2k1o1p1q1mmmf", "hfile": cons_dir / "h_m2s2n2k2k1o1p1q1mmmf", "ufile": cons_dir / "u_m2s2n2k2k1o1p1q1mmmf", }, crs=4326, ) print("Gridded constituents:", source.open().con.values) source = SourceTideConsPointCSV(filename=DATA_DIR / "tide_cons_station.csv") source.open()

Gridded constituents: ['M2' 'S2' 'N2' 'K2' 'K1' 'O1' 'P1' 'Q1' 'MM' 'MF']

Out[9]:

<xarray.Dataset> Size: 320B
Dimensions:  (con: 10)
Coordinates:
  * con      (con) <U4 160B 'M2' 'S2' 'N2' 'K2' 'K1' 'O1' 'P1' 'Q1' 'MM' 'MF'
Data variables:
    h        (con) complex128 160B (0.018999999842540936+0.03799999832062555j...

xarray.Dataset

• Dimensions:
  • con: 10
• Coordinates: (1)
  • con
    (con)
    <U4
    'M2' 'S2' 'N2' ... 'Q1' 'MM' 'MF'

    array(['M2', 'S2', 'N2', 'K2', 'K1', 'O1', 'P1', 'Q1', 'MM', 'MF'], dtype='<U4')

• Data variables: (1)
  • h
    (con)
    complex128
    (0.018999999842540936+0.03799999...

    array([ 1.89999998e-02+0.038j     ,  2.30000016e-02+0.03999999j,
           -6.99999779e-03+0.015j     ,  9.00000109e-03+0.012j     ,
           -1.82000002e-01-0.01400001j, -1.28999990e-01+0.00600002j,
           -5.70000015e-02+0.00100001j, -2.79999990e-02+0.005j     ,
            1.22464680e-19+0.002j     ,  1.22464680e-19+0.002j     ])

3. Timeseries without coordinates¶

The ...Point forcing classes take a single timeseries, such as a buoy record. These sources come from rompy and need no CRS.

In [10]:

from rompy.core.source import SourceTimeseriesCSV

source = SourceTimeseriesCSV(filename=DATA_DIR / "wind.csv", tcol="time")
source.open()

from rompy.core.source import SourceTimeseriesCSV source = SourceTimeseriesCSV(filename=DATA_DIR / "wind.csv", tcol="time") source.open()

Out[10]:

<xarray.Dataset> Size: 1kB
Dimensions:  (time: 25)
Coordinates:
  * time     (time) datetime64[us] 200B 2023-01-01 ... 2023-01-02
Data variables:
    u10      (time) float64 200B -3.634 -3.476 -3.083 ... -5.894 -6.366 -6.693
    v10      (time) float64 200B 6.261 6.246 6.411 6.792 ... 6.306 6.253 6.247
    wspd     (time) float64 200B 7.239 7.148 7.113 7.274 ... 8.632 8.923 9.155
    wdir     (time) float64 200B 149.9 150.9 154.3 159.0 ... 136.9 134.5 133.0

xarray.Dataset

• Dimensions:
  • time: 25
• Coordinates: (1)
  • time
    (time)
    datetime64[us]
    2023-01-01 ... 2023-01-02

    array(['2023-01-01T00:00:00.000000', '2023-01-01T01:00:00.000000',
           '2023-01-01T02:00:00.000000', '2023-01-01T03:00:00.000000',
           '2023-01-01T04:00:00.000000', '2023-01-01T05:00:00.000000',
           '2023-01-01T06:00:00.000000', '2023-01-01T07:00:00.000000',
           '2023-01-01T08:00:00.000000', '2023-01-01T09:00:00.000000',
           '2023-01-01T10:00:00.000000', '2023-01-01T11:00:00.000000',
           '2023-01-01T12:00:00.000000', '2023-01-01T13:00:00.000000',
           '2023-01-01T14:00:00.000000', '2023-01-01T15:00:00.000000',
           '2023-01-01T16:00:00.000000', '2023-01-01T17:00:00.000000',
           '2023-01-01T18:00:00.000000', '2023-01-01T19:00:00.000000',
           '2023-01-01T20:00:00.000000', '2023-01-01T21:00:00.000000',
           '2023-01-01T22:00:00.000000', '2023-01-01T23:00:00.000000',
           '2023-01-02T00:00:00.000000'], dtype='datetime64[us]')

• Data variables: (4)
  • u10
    (time)
    float64
    -3.634 -3.476 ... -6.366 -6.693

    array([-3.6337004 , -3.4760983 , -3.082727  , -2.6036313 , -2.1391146 ,
           -1.7158526 , -1.4180747 , -1.0972018 , -0.88136923, -0.8445983 ,
           -0.41188955, -0.7986343 , -1.3572783 , -2.0408142 , -2.6733668 ,
           -3.2876465 , -3.8719795 , -4.4412546 , -5.003958  , -5.5272417 ,
           -5.91878   , -6.2821198 , -5.894191  , -6.3657856 , -6.6932287 ])

  • v10
    (time)
    float64
    6.261 6.246 6.411 ... 6.253 6.247

    array([ 6.260819 ,  6.2456193,  6.4107513,  6.792392 ,  7.2329392,
            7.7844214,  8.242932 ,  8.703077 ,  9.128032 ,  9.506232 ,
            9.770763 , 10.055317 , 10.0995035, 10.046434 ,  9.802656 ,
            9.4612255,  9.092444 ,  8.637854 ,  8.0594425,  7.4669037,
            6.935783 ,  6.5578074,  6.3057675,  6.2526135,  6.246578 ])

  • wspd
    (time)
    float64
    7.239 7.148 7.113 ... 8.923 9.155

    array([ 7.238897 ,  7.147798 ,  7.1134334,  7.274303 ,  7.542627 ,
            7.971284 ,  8.364022 ,  8.771967 ,  9.170484 ,  9.543678 ,
            9.779442 , 10.086983 , 10.190298 , 10.251623 , 10.160658 ,
           10.016157 ,  9.882549 ,  9.712737 ,  9.486527 ,  9.290051 ,
            9.117951 ,  9.081292 ,  8.631581 ,  8.9229145,  9.155274 ])

  • wdir
    (time)
    float64
    149.9 150.9 154.3 ... 134.5 133.0

    array([149.86966, 150.90118, 154.31857, 159.02739, 163.52463, 167.56956,
           170.23866, 172.8146 , 174.48483, 174.92279, 177.58612, 175.45888,
           172.34586, 168.5173 , 164.74535, 160.83836, 156.93349, 152.7895 ,
           148.16461, 143.48997, 139.5236 , 136.23001, 136.9322 , 134.48615,
           133.02307])

SourceTimeseriesDataFrame wraps a pandas dataframe with a time index. It is provided by the rompy-binary-datasources package, installed with pip install rompy-xbeach[extra].

In [11]:

from rompy_binary_datasources.source import SourceTimeseriesDataFrame

df = pd.read_csv(DATA_DIR / "ssh.csv", index_col="time", parse_dates=True)
source = SourceTimeseriesDataFrame(obj=df)
source.open()

from rompy_binary_datasources.source import SourceTimeseriesDataFrame df = pd.read_csv(DATA_DIR / "ssh.csv", index_col="time", parse_dates=True) source = SourceTimeseriesDataFrame(obj=df) source.open()

Out[11]:

<xarray.Dataset> Size: 400B
Dimensions:  (time: 25)
Coordinates:
  * time     (time) datetime64[us] 200B 2023-01-01 ... 2023-01-02
Data variables:
    ssh      (time) float64 200B -0.6052 -0.4818 -0.3333 ... -0.997 -0.849

xarray.Dataset

• Dimensions:
  • time: 25
• Coordinates: (1)
  • time
    (time)
    datetime64[us]
    2023-01-01 ... 2023-01-02

    array(['2023-01-01T00:00:00.000000', '2023-01-01T01:00:00.000000',
           '2023-01-01T02:00:00.000000', '2023-01-01T03:00:00.000000',
           '2023-01-01T04:00:00.000000', '2023-01-01T05:00:00.000000',
           '2023-01-01T06:00:00.000000', '2023-01-01T07:00:00.000000',
           '2023-01-01T08:00:00.000000', '2023-01-01T09:00:00.000000',
           '2023-01-01T10:00:00.000000', '2023-01-01T11:00:00.000000',
           '2023-01-01T12:00:00.000000', '2023-01-01T13:00:00.000000',
           '2023-01-01T14:00:00.000000', '2023-01-01T15:00:00.000000',
           '2023-01-01T16:00:00.000000', '2023-01-01T17:00:00.000000',
           '2023-01-01T18:00:00.000000', '2023-01-01T19:00:00.000000',
           '2023-01-01T20:00:00.000000', '2023-01-01T21:00:00.000000',
           '2023-01-01T22:00:00.000000', '2023-01-01T23:00:00.000000',
           '2023-01-02T00:00:00.000000'], dtype='datetime64[us]')

• Data variables: (1)
  • ssh
    (time)
    float64
    -0.6052 -0.4818 ... -0.997 -0.849

    array([-0.6052159 , -0.48183319, -0.3332678 , -0.16083648,  0.03371079,
            0.24590418,  0.46658058,  0.68095331,  0.86922664,  1.00887239,
            1.07823838,  1.06076514,  0.94885449,  0.74643286,  0.46949626,
            0.14436276, -0.19611194, -0.51761095, -0.78965724, -0.99000416,
           -1.10719092, -1.14078199, -1.09937557, -0.99701314, -0.84900647])

Summary¶

• Gridded sources carry a CRS, so data in any projection can be interpolated onto the XBeach grid.
• Wave spectra and tidal constituents have dedicated sources.
• Plain timeseries use the rompy CSV or dataframe sources.

See also: Data selection options controls which part of a source is used.