Changelog for 1.2#

1.2.7#

Release date: 30 October 2026

  • Fixed a previously known issue, where labels that were the same as a result type name (e.g. giving a PO point a label "H") would cause issues when extracting data such as time_series(). It is still recommended to avoid these naming conflicts, however PyTUFLOW should be more robust at handling it.

1.2.6#

Release date: 13 October 2026

  • Further optimised long plot connectivity when using the section() method for a time series result (e.g. TPC). The optimisation is related to when more than one ID is passed in; if the IDs are passed in in order from upstream to downstream, then this change has no effect (should already be quick). If the IDs are passed in from downstream to upstream, then previously this could cause the connectivity to be slow because it tries to connect the branches in the wrong order i.e. it won’t find a connection and checks every branch (which can be slow if there are 1,000s of branches). Now it limits an initial search pass to the branch_limit parameter in both directions before moving onto a wider search if no connection is found.

1.2.5#

Release date: 12 October 2026

  • Fixed a bug that could cause GPKG time series results to fail to plot GPKG1D.time_series results due to duplicate entries in the GPKG Timeseries_info table.

  • Improved GPKG time series loading times. Testing on a dataset with a lot of timesteps reduced the load time from about 8.5 sec down to 4 sec.

1.2.4#

Release date: 06 October 2026

  • Fixed a bug where TPC files that were created from a 2D only model (no 1d or hidden 1d nodes) caused the TPC to throw an error stating that the result looks empty.

  • Added a branch limit argument to the section() method signature to prevent the method from hanging due to millions of (legitimate) paths being found and processed. The number of paths can balloon due to many parallel channels in the section (e.g. culvert / weir combinations). The number of paths follows a power relationship to the number of parallel channels instances and as a consequence can become exceedingly large.

  • Fixed a bug that occurred when extracting results that are typically 3D results (such as velocity) from a 2D result file for the NCMesh results. Previously, the time_series() or section() methods incorrectly returned NaN values.

  • Version name extraction from a folder name is now more robust, allowing for version searching among other text e.g. TUFLOW.2011-09-AE will now correctly find the version name. Previously, the TUFLOW. at before the version caused an error extracting the version.

1.2.2#

Release date: 10 September 2026

  • Older TUFLOW versions that sit within a w64 subdirectory are correctly handled when checking binary versions.

1.2.1#

Release date: 07 September 2026

  • Encoding errors are now ignored when querying TUFLOW versions.

1.2.0#

Release date: 28 August 2026

TUFLOW FV#

Support has been added for TUFLOW FV control files. The workflow and data structures are very similar to Classic/HPC with a few minor differences. See Working with TUFLOW FV for examples.

Project Creation#

Note

This is an experimental feature. This means that the CLI is not considered final and can change in future versions without provision for backward compatibility.

PyTUFLOW now supports project initialisation and creation of a skeleton directory and control files via pytuflow-project. Alternatively, via the classes HPCProject and FVProject.

Flux#

The ability to extract flux from map output results has been added. The flux method takes line(s) and extracts the resulting flux by using the unit flow, or alternatively, the velocity and depth. By default, the flux() method returns the volume of water passing over the line. However, it also has the option to pass in the name of constituent/tracer data types. In these cases, the routine will return the mass flux passing over the line.

The routine is available for map output result formats that contain vector velocity or vector unit flow data. This includes the XMDF, NCGrid, NCMesh, and CATCHJson classes. Flux calculation from 3D results will include the summation from each vertical layer and will not perform depth averaging.

Warning

The flux() method should be used with care as for most result formats, it will be calculating the flux from an interpolated surface. From limited testing on real world models, the XMDF.flux() method will typically estimate flow peak within 5% of the equivalent PO flow line. This holds provided there is enough cell resolution across the flow path and SGS is turned off. If SGS is turned on, then it is recommended to use unit flow rather than depth and velocity as depth at cell corners can be a poor representation of the flow depth within a cell using an SGS curve. Note, using unit flow is recommended with SGS on or off, however it is especially important with SGS on.

The flux() routine can be expensive, mostly due to the number of I⁠/⁠O operations it has to perform (especially if it has to query both depth and velocity). This can be greatly sped up if performing multiple flux() calls on the same result by loading the relevant result types into memory. This can be done using the load_into_memory() method. See Load into Memory below for more information.

Example:

>>> from pytuflow import XMDF
>>> res = XMDF('/path/to/hpc-result.xmdf')
>>> df = res.flux('/path/to/flux-line.shp')
         line/flux (q)
time
80.0          0.000000
82.0          0.000000
84.0          0.000000
86.0          0.000000
88.0          0.000000
...                ...
342.0        13.754197
344.0        13.498373
346.0        13.158218
348.0        13.268435
350.0        12.799280

[136 rows x 1 columns]

Flux Integral#

The ability to extract the flux integral from map output results has been added. The flux integral method takes line(s) and extracts the resulting flux integral by using the unit flow, or alternatively, the velocity and depth. By default, the flux_integral() method returns the volume of water passing over the line integrated over time. However, it also has the option to pass in the name of constituent/tracer data types. In these cases, the routine will return the mass flux passing over the line integrated over time.

Note

Similar caveats and recommendations apply to the flux_integral() method as for the flux() method. See the Flux section above for more information.

Load into Memory#

A method to load data into memory has been added to MapOutput classes e.g. XMDF.load_into_memory() and NCGrid.load_into_memory(). This method loads the entire dataset, including all timesteps, for given data type(s) into memory. This can greatly improve the speed of subsequent queries by removing the need to perform relatively expensive I⁠/⁠O operations.

The process of loading an entire dataset into memory itself can be relatively expensive, however can move all I⁠/⁠O operation costs to a single upfront call. This can then greatly improve the speed of subsequent calls that would usually make a lot of I⁠/⁠O operations, or if making many calls that use I⁠/⁠O operations. A great example of when it could be beneficial to load data into memory upfront is when using the flux() call. The flux method can be more expensive than other extraction methods, especially if it has to query both depth and velocity results. A single flux call will most likely be cheaper than loading a dataset into memory, however it can quickly become beneficial to load into memory up front if making multiple flux calls on the same result.

As an example, a test was carried out by extracting flux on a medium sized XMDF result (~1 GB). The result did not contain unit flow, so both depth and velocity had to be queried to obtain the flux. The XMDF.flux() method took ~2.5 s to execute. Loading both the depth and velocity into memory took ~4.5 s. Subsequent XMDF.flux() calls took ~0.01 s. So in this example case, it would be beneficial to load depth and velocity into memory if extracting flux from two or more locations. This can scale very rapidly, so it is almost always a good idea to load the relevant result types into memory when batch exporting flux from a single result. The exact ratio will vary depending on the results and other variables such as how many cells the flux line intersects and how many timesteps are in the results etc.

Example:

>>> from pytuflow import XMDF
>>> res = XMDF('/path/to/hpc-result.xmdf')
>>> res.load_into_memory(['depth', 'vector velocity'])  # make sure to use 'vector velocity' and not 'velocity' for Classic/HPC results
>>> df = res.flux('/path/to/flux-lines.shp')
>>> df
      38_1/flux (d.v)  38_2/flux (d.v)  37_1/flux (d.v)  ...  TMR_001/flux (d.v)  SRC_073/flux (d.v)  BCC_077/flux (d.v)
time                                                      ...
80.0          0.000000         0.000000         0.000000  ...            0.000000            0.000000            0.000000
82.0          0.000000         0.000000         0.000000  ...          510.361820            0.000000            0.003746
84.0          0.000000         0.000000         0.000000  ...        -1140.110520            0.000000           -0.004497
86.0          0.000000         0.000000         0.000000  ...         -176.114340            0.000000           -0.011052
88.0          0.000000         0.000000         0.000000  ...         1446.423141            0.000000           -0.032609
...                ...              ...              ...  ...                 ...                 ...                 ...
342.0        13.754197         4.655368         6.469491  ...         3636.175415         3202.797150         3052.134145
344.0        13.498373         4.438060         6.097150  ...         3243.619695         3208.747937         3048.996453
346.0        13.158218         4.238195         5.743462  ...         3412.564508         3203.476447         3048.696176
348.0        13.268435         4.044892         5.398321  ...         3906.436114         3208.706137         3050.309653
350.0        12.799280         3.846815         5.082798  ...         4107.193577         3197.010855         3050.993464

[136 rows x 122 columns]

Note

QGIS drivers, specifically the result extraction drivers, are much slower than h5py and netcdf4 for extracting entire data types results. The same example, which takes ~4.5 s in h5py takes ~60 s with QGIS drivers. This can still be beneficial if exporting hundreds of flux locations, however it is much better to use a different driver if possible. Note, it is possible to use QGIS for geometry and netcdf4 for result extraction which will negate this problem. In fact, the TUFLOW Viewer V2 (part of the TUFLOW plugin in QGIS) uses this capability and preferences netcdf4 if it is available and QGIS for the geometry.

This is not intended to disparage QGIS. Libraries such as h5py have been specifically optimised for getting entire datasets from hdf5 files into Python. So naturally they are very quick at this task. General data extraction in QGIS, given single timesteps, are comparable in speed to h5py.

Add Dataset#

A method to add additional datasets has been added to Mesh classes e.g. XMDF.add_dataset() and NCMesh.add_dataset(). This routine will take an additional dataset and add it to an existing mesh class as long as they use the same geometry and are of the same type (i.e. a .xmdf can only be loaded onto an XMDF class even if the mesh files share the same geometry).

Adding a dataset to an existing mesh output instance will be cheaper than loading into its own instance since it will not have to load its own mesh geometry (which is typically the slowest part of loading a mesh result). It can also be useful to have result types together in a single result instance as some routines can combine results to give more information. An example of this is the flux() method which can take additional tracer or constituent data types to provide the resulting mass flux across a line. Results from the TUFLOW FV WQ module is a good example of when this is useful. The WQ module outputs into a separate NetCDF mesh to the hydrodynamic results. The WQ results can be added to the hydrodynamics to combine them to calculate mass flux of various constituents. This is not possible with the WQ results by themselves.

Example:

>>> from pytuflow import NCMesh
>>> res = NCMesh('/path/to/hydrodynamic-results.nc')
>>> res.add_dataset('/path/to/wq-results.nc')
>>> df = res.flux('/path/to/flux-line.shp', 'wq_ammonium_mg_l')
>>> df
          line/flux wq_ammonium_mg_l (d.v)
time
289272.00                          0.000000
289272.25                          0.000000
289272.50                          0.000009
289272.75                          0.000009
289273.00                          0.000011
...                                     ...
289439.00                          0.002076
289439.25                         -0.000039
289439.50                         -0.001020
289439.75                         -0.001708
289440.00                         -0.000621

[673 rows x 1 columns]
>>> df.plot()
>>> import matplotlib.pyplot as plt
>>> plt.show()
../../_images/wq_ammonium_flux.png

Note

This functionality is not intended as a way to quickly reload results onto an already instantiated mesh. The class will return the first instance of a given result type name, which will be the original mesh result. This means that if a dataset is added with data type names that exist already in the mesh instance, it will not overwrite the original instance’s datasets.

Note

QGIS drivers do not allow a NetCDF mesh to be appended to another NetCDF mesh. PyTUFLOW will allow it and the subsequent behaviour in PyTUFLOW in respect to interacting with the mesh output will be identical regardless of this. However, the mesh geometry will be required to be loaded again so the performance gain will be lost in this situation.

Mesh Dataset#

Mesh results now has a convenience function to obtain a pyvista.PolyData mesh object. This can be done using mesh_dataset(), which will return a geometry based on a dataset such as bed level or water level.

A pyvista.PolyData object can be plotted in 3D space using the pyvista.Plotter() class. Results can be rendered to a movie, or a time slider can be added to allow interaction as well. See the mesh_dataset() documentation for examples.

Example using pyvista to render an animation with vectors

LP2D#

The 2d_lp output from TUFLOW Classic/HPC is now supported as a new output class - LP2D. This class supports the section() method as well as the time_series() method at automatically generated points at each distance value in the CSV file.

Minor New Features#

  • Point and line locations used for result extraction (e.g. XMDF.time_series() and XMDF.section()) will now accept shapely geometry and geopandas.GeoDataFrame objects.

  • The NetCDF4 driver for XMDF and NetCDF map ouput formats is now more performant when extracting non-contiguous data.

  • Added the ability to refresh the tuflow binary locations if the cache (json settings file) is modified by a separate pytuflow process using the TuflowBinaries.refresh_from_settings method.

Bug Fixes#

  • Fixed a bug in the MapOutput classes where reading in an empty GIS file would cause an unexpected Python error. When an empty GIS file is provided, an exception is still thrown, but gives a more informative message about the issue.

  • Fixed a bug where if a .prj file was used in a TUFLOW projection command, and the accompanying shapefiles were missing (i.e. the .shp, .dbf etc), then the projection was not read in correctly and produced a warning. This is now fixed and the .prj is no opened and the projection string is read directly from the file.