pytuflow.NCGrid.load_into_memory

NCGrid.load_into_memory(data_types)

Load the given data types into memory. This loads the entire dataset, including all timesteps, which can greatly improve the speed of queries by removing the need for frequent and relatively expensive I⁠/⁠O operations. Any accompanying data will also be loaded, e.g. the active (wet/dry) flags.

Loading data types into memory can be a relatively slow process, however it will speed up subsequent queries. The speed-up is relative to how many I⁠/⁠O operations a given method makes. e.g. The flux() method makes many I⁠/⁠O calls and it can typically be beneficial to load the relevant results into memory if making more than two flux() calls on a result. On the other hand, the section() method will typically only make two queries, one for the data and one for the active flag. As a consequence, it may not be worth loading the data into memory for section() calls as the cost of loading into memory outweighs the subsequent speed up. It will vary depending on the results, the calls being made, and where the results are located (e.g. locally or network drive).

Note

The maximum and temporal datasets are considered separate datasets e.g. loading water level into memory will not load maximum water level and vice versa. It is also not supported in v1.0 mesh drivers.

Note

QGIS drivers, specifically the result extraction drivers, are much slower than h5py and netcdf4 for extracting entire data types results. In one test case, h5py took ~4.5 s to load both vector velocity and depth, whereas QGIS drivers took ~60 s. This can still be beneficial if exporting hundreds of flux locations, however it is much better to use a different driver if possible. Note, it is possible to use QGIS for geometry and netcdf4 for result extraction which will negate this problem. In fact, the TUFLOW Viewer V2 (part of the TUFLOW plugin in QGIS) uses this capability and preferences netcdf4 if it is available and QGIS for the geometry.

This is not intended to disparage QGIS. Libraries such as h5py have been specifically optimised for getting entire datasets from hdf5 files into Python. So naturally they are very quick at this task. General data extraction in QGIS, given single timesteps, are comparable to speeds in h5py.

Parameters:

data_types (str | list[str]) – The result type(s) to load into memory.

Examples

Load a result type into memory to speed up flux() calls.

>>> res = ... # assume res is a loaded mesh or grid result file
>>> res.load_into_memory('vector unit flow')
>>> df = res.flux('/path/to/flux_line.shp')

If unit flow is not available, the depth and velocity will be needed. Note, just velocity is needed for NCMesh results as the depth is stored within the vertical layer elevation data.

>>> res.load_into_memory(['depth', 'vector velocity'])
>>> df = res.flux('/path/to/flux_line.shp')