Section 7 Outputs and Check Files
7.1 Additional Options for Record Gauge Data Outputs
The 2023-03 Release now allows the user to choose whether to automatically include reporting location lines, reporting location points, and gauge locations (PO ‘G_’ type) when processing gauge receptor objects.
The following command can be used to specify which location types to include:
Or alternatively the following command can be used to exclude types:
The default is to include all reporting locations lines, points, and gauges.
7.2 Time of Max Water Level Switch
The 2023-03 Release adds an output switch in the .tcf that gives the user the ability to turn off the “TMax_h” output. This command builds on an existing command:
7.3 Time of Max Velocity Off by Default
For the 2023-03 Release, the “TMax_V” output has been switched off by default for TUFLOW Classic outputs to be consistent with TUFLOW HPC. The “TMax_V” output can be turned back on using the following command (available for TUFLOW Classic only):
7.4 File Access Timeout
Improvements have been made to how TUFLOW handles an .xf or .xmdf file that is locked whilst being written to. For both file types, TUFLOW will now repeatedly attempt to access the file.
XF File access
When multiple instances of TUFLOW are running from the same .tcf file (e.g. different events/scenario combinations), it is possible for these separate TUFLOW simulations to try to access the same .xf file at the same time. This could cause a “race condition” where only one simulation is able to open the file. Previously, TUFLOW would issue a dialog with a message that the file is locked. The user could continue once the XF file was no longer in use by another TUFLOW simulation. The 2023-03 Release will automatically wait for the file to become free and continue without user intervention.
XMDF File access
When running TUFLOW, XMDF files may become locked due to external use of the file, such as during file backups. The 2023-03 Release will wait for the file to become freed up to the maximum number of retries.
The .tcf commands below can be used to modify the number of retries and time to wait between retries. The maximum number of retries is capped to prevent and infinite loops if the file is permanently locked. The commands control behaviour for both .xf and .xmdf files.
7.5 Additional Check Grid Formats
All raster output format types are now supported for the check DEMs (DEM_M and DEM_Z). The type can be set using:
7.6 High Resolution Map Output Includes Additional File Formats
All raster outputs now support a high resolution (HR) option, available when using SGS Method C. The below lists the available high resolution outputs which can be added as map outputs types in the .tcf:
- HRASC
- HRFLT
- HRTIF
- HRGPKG
- HRNC
- HRTGO
- HRWRR
Customisation of the HR outputs is also possible for each type:
Note, only depth and water level data types are supported with high resolution map outputs.
7.6.1 Viewing HRNC in TUFLOW Viewer
The TUFLOW Viewer supports NetCDF raster viewing (NC and HRNC). This allows users to easily view and query high resolution temporal results using the same functionality available for XMDF. Please see the following wiki page for more information:
https://wiki.tuflow.com/TUFLOW_Viewer_-_Load_Results_-_NetCDF_Grid
7.7 New Map Output Data Types
7.7.1 HPC Control Numbers
For HPC simulations the three timestep control numbers (Nu, Nc, and Nd) are now available as map output data types. These control number outputs are:
- Nu: Courant Number
- Nc: Shallow Wave Celerity Number
- Nd: Diffusion Number
Note, these are only valid map output data types for HPC simulations, these are not available for Classic simulations. For TUFLOW Classic Courant Number “Cr” output is available. “Cr” map output is not valid for HPC simulations.
7.7.2 New Hazard Output – Auckland Council Hazard NZ
Build 2023-03-AB introduces a new hazard category used by Auckland Council, New Zealand. This is specified with the “ZNZ2” map output data type. The categories are shown in Figure 7.1 and defined as:
Category 1 (Low Safety Risk):
- D*V < 0.4
- V < 3.0 m/s
- D < 0.5 m
Category 2 (Moderate Safety Risk)
- D*V < 0.8
- V < 3.0 m/s
- D < 1.2 m
Category 3 (High Safety Risk)
- D*V ≥ 0.8
- V ≥ 3.0 m/s and D ≥ 0.1m
- D ≥ 1.2 m
Dry points are assigned Category 0.
Figure 7.1: Auckland Council Hazard Output
7.8 Changes to Simulation Computational and Clock Times
For build 2023-03 extra information is output to the simulations.log file and .tlf regarding simulation clock and processor time. Note that the processor time is summed across all threads and includes GPU hardware. If parallel processing is used in model initialisation or solver this number may be larger than clock time.
Prior to the 2023 release the simulations.log file reported, shown in Figure 7.2:
- Total processor time at the end of the simulation – highlighted in red
- Hardware type and number of processor used (CPU or GPU) - highlighted in blue
- Clock time (wall clock or elapsed real time) at the end of the simulation – highlighted in green
Figure 7.2: 2020-10-AF Simulation Log
For the 2023 release this has been expanded to include the clock and processor time after initialisation (the start of the solution) as well as the clock and processor time at the end of the simulation, shown in Figure 7.3. From left to right these are:
- Clock time after initialisation - yellow
- Processor time after initialisation – light blue
- Clock time at end of simulation – green
- Processor time at end of simulation – red
- Hardware type and number of processor used - dark blue
Figure 7.3: 2023-03-AA Simulation Log
The clock and processor times are also reported at the end of the .tlf, shown in Figure 7.4.
Figure 7.4: 2023-03-AA Simulation Log
7.8.1 Changes to Simulation Log Files for 2023-03-AB
Computational processor time is being reported very differently for Windows 11 compared to prior versions of Windows. For earlier version of Windows, the processor time included time spent in GPU computer. For Windows 11, the processor time appears to exclude time spent in GPU calculations. From Build 2023-03-AB if running on GPU hardware the processor time is not reported to the simulations.log file. The processor time is still reported to the .tlf with a note that:
“Processor times may be misleading/inaccurate when using GPU hardware.”
Build 2023-03-AB also includes some minor changes to the formatting of the simulation log file, which includes:
- Extra width allowed for dongle serial numbers
- Better alignment of columns
7.9 Minor Enhancements and Bug Fixes
The 2023-03 Release includes a range of minor enhancements and bug fixes relating to output and check files.
7.9.1 Checks for Valid Geometries for Plot Outputs
For the 2023-03-AA release, additional checks are performed for plot outputs depending on the GIS geometry. For example, plot output type “Q” (flow) is valid on a line but not a point. Incompatible plot output types will generate either a warning (which is the default) or an error. To set this the following .tcf command can be used:
Note: Build 2023-03-AB fixes an issue which prevented plot output types Qin, Qout and Vol from working with PO regions.
7.9.2 HPC dt.csv Interval Can Now Be Set
The .hpc.dt.csv contains information on the timestep history for HPC (including Quadtree) simulations. Prior to the 2023-03 Release this was output at each timestep. For the 2023-03 Release the output interval can be set with the .tcf command:
If not set, the default is to use the Screen/Log Display Interval, if set to zero or less the file is supressed and not written.
7.9.3 Bug Fix for Map Output Zones Using Different Map Output Intervals
The 2023-03 Release fixes an issue with map output zones that use different map output intervals. Previously, when the output zones had different map output intervals (from other map output zones or the global settings), TUFLOW could fail to write all output times. This could also occasionally cause a ‘Should not be here’ to occur and the model to crash.
7.9.4 SGS Sample Distance Reporting in TLF
The 2023-03 Release fixes an issue with the SGS sample distance reporting in the .tlf when the sample distance is less than 0.00001. Previously this would be reported as 0.000.
7.9.5 Limit Number of WARNING 2550 Messages
The 2023-03 Release limits the number of WARNING 2550 messages (written at the end of an HPC simulation) to 100. The total number of cells with timestep corrections is still reported.
This limit can be disabled by using the “
7.9.6 2D_WRF Empty File
The 2023-03 Release will now write 2d_wrf empty files as part of the following command:
7.9.7 High Resolution Grid Outputs Not Created with Map Output Zones
The 2023-03 Release fixes a bug with high resolution grid outputs. Previously, they were not being written when the model contained map output zones.
7.9.8 Time Output Cutoff Logging Fix
The 2023-03 Release fixes a bug that could incorrectly report the hazard being used in the “
7.9.9 Mass Error Reporting During Simulation Showing Asterisks
The 2023-03 Release fixes the mass error reporting in the simulation window when the mass error equals negative 100%. Previously this would appear as asterisks (******%).
7.9.10 Snapping Settings Added to .qgs File
The 2023-03 Release adds snapping settings to the .qgs file (QGIS project workspace) that is written to the log folder as part of a TUFLOW simulation. This fixes an issue that is present in QGIS version 3.26 or later when loading the .qgs file written by TUFLOW. Without snapping settings in the .qgs file, none of the layers written to the workspace would allow snapping.
7.9.11 Duplicate Messages in the messages_P GIS File
The 2023-03 Release removes duplicate messages that were written to the messages_P file. For example, CHECK 1206 and CHECK 1625 could be duplicated.
7.9.12 Quadtree zsh_zpt_check Points in Non-Refined Areas
The 2023-03 Release fixes an issue in Quadtree models where the points in the zsh_zpt_check_P check file were at spacing that did not match the level of mesh refinement. This issue was limited to Z-shape polygons and only occurred if the polygon was within the hidden domains used by the Quadtree pre-processing engine. This issue did not affect the mesh generation or the results.
7.9.13 Bug Fix for SGS Model Outputs When Cell Averaged Depth Option is Used
The 2023-03 Release fixes an issue where a SGS model could output zero depth at the first simulation timestep when “
7.9.14 Fixes an Issue with Viscosity Output Written as Zero
The 2023-03 Release fixes an issue with the viscosity output (data type “T”) being written as zero unless the data type “T” was specified for the default map output data types (e.g. Map Output Data Types without format prefix). For example, the following would cause an output of zero viscosity:
The following output settings would correctly output the viscosity.
For the 2023-03 Release, if viscosity output is requested for any file formats, this will be correctly pulled from the solver.
7.9.15 HPC Control Number Map Output
The 2023-03 Release supports three new output data types in the HPC Solver (including Quadtree):
Nu is the Courant Number, Nc is the Shallow Wave Celerity Number, and Nd is the Diffusion Number. Please refer to https://wiki.tuflow.com/HPC_Adaptive_Timestepping for how the control numbers are calculated. The 2023-03 Release will also produce WARNING 2386 if the Classic Courant Number (Cr) output is used in a HPC model.
7.9.16 _zsh_zpt Check File for SGS Models
The 2023-03 Release fixes a check file inconsistency between SGS Approach Method B and C (see Section 2.2.2). Previously, SGS Method C writes _zsh_zpt check points only if a cell centre, a face mid-points or a corner point is located inside a 2d_zsh polygon. On the other hand, SGS Method B writes _zsh_zpt check points at all the updated SGS cells, faces and corners including the partially updated SGS points. The 2023-03 Release now writes _zsh_zpt check points for all the updated SGS points in both SGS method B and C.
In addition, two extra attributes are added to the SGS _zsh_zpt check file. The new attributes include:
- dZ: the change in elevation at the exact location of cell centres, face mid-points or corner points.
- ZExact: the new elevation at the exact location of cell centres, face mid-points or corner points.
- dZmin: the change in minimum elevation inside cells, along faces or at corners.
- Zmin: the new minimum elevation inside cells, along faces or at corners.
- Source: the source file that edits the SGS points.
7.9.17 Extra Checks on File Format Specific Map Output Commands
Build 2023-03-AA introduces checks on file format specific Map Output commands, to ensure that only one file format is specified. For example, the following command Will now produce ERROR 2590:
To apply the same settings to more than one format, the command needs to be repeated for each format, e.g.:
7.9.18 Changes to Processing of Map Output Data Types Command
When processing “
The recommended syntax would be:
7.10 Minor Enhancements and Bug Fixes for 2023-03-AB
7.10.1 Fix for Quadtree High Resolution Outputs and WLL
Build 2023-03-AB fixes an issue with high resolution output in quadtree models with SGS and water level lines. Previously this could cause the simulation to crash without error.
7.10.2 Reduced Log File when Processing Quadtree Polygons
Build 2023-03-AB reduces the amount of output which goes to the .tlf when processing Quadtree nesting level polygons. Previously, if a significant number of nesting polygons was used then a large amount of output to the log file was included. For 2023-03-AB this output is still displayed to the console, but not to the .tlf unless “
7.10.3 Bug Fix for Integer (Hazard) Grid Outputs
Build 2023-03-AB now correctly outputs integer grids (e.g. certain hazard outputs) as integers rather than decimal numbers. In the 2023-03-AA build this was broken for the FLT format and did not work for the new raster formats (GeoTIFF, GPKG raster). The ASC format worked in the 2023-03-AA build.
7.10.4 Consistent Reporting Location Dry Output
Build 2023-03-AB will now output consistent RL point values across engines for dry cells / nodes. These will now be output with a null value (default value -99999, however can be user set) whenever the output location is dry. Previously Classic and Quadtree would initially report the null value for the first timestep then a zero value for subsequent dry timesteps; HPC and ESTRY would report the bed level.
7.10.5 Bug Fix MIF PLOT output
Build 2023-03-AB fixes a bug introduced in the 2023-03-AA build that would not correctly write the GIS PLOT output layer for the MIF format. This only affected the GIS PLOT output layer (the CSV outputs were correct) which meant that the results could not be viewed in TUFLOW Viewer.
7.10.6 Scientific Notation for PO Outputs
Build 2023-03-AB uses scientific notation for CSV PO outputs when values are very small (i.e. close to zero, not negative values), e.g. groundwater flow outputs.
7.10.7 Scientific Notation for ASC Outputs
Build 2023-03-AB uses scientific notation for ASC grid outputs when values are very small (i.e. close to zero, not negative values) e.g. groundwater unit flow output.
7.10.8 Bug Fix Geometry Type Groundwater Flow Output in TPC
Build 2023-03-AB fixes a bug when writing groundwater PO flow line output to the TPC which was previously listing it as a ‘point output’ (e.g.
7.10.9 Bug Fix Groundwater Vector Grid Outputs
Build 2023-03-AB fixes a bug that could cause groundwater vector grid outputs to write out only null values.
7.10.10 Bug Fix Velocity GIS Map Output
Build 2023-03-AB fixes the velocity output for the GIS map output which was creating an empty output.
7.10.11 Extra Precision in Quadtree uvpt Check File for Form Loss
Build 2023-03-AB allows extra width for the form loss attribute, this has been increased to 4 decimal places in line with the Classic/HPC check file. The form loss values are stored as floating-point numbers, this change only affects the value written to the check files.
7.10.12 TMO Output Changes for HPC Model with SGS
Build 2023-03-AB fixes an issue with the TMO format output for HPC models (without Quadtree) when using SGS. For these models the depth output was being reported relative to the cell minimum elevation (
7.10.13 Duplicated HR Output Message in Log File
Build 2023-03-AB removes duplicated log file messages related to HR Output.
7.10.14 Updated Links to TUFLOW Wiki
An update on the software used to host the TUFLOW wiki website has changed URL links. For example:
https://wiki.tuflow.com/index.php?title=TUFLOW_Message_2570
has now become:
https://wiki.tuflow.com/TUFLOW_Message_2570
A re-direct has been setup for existing links / pages, however, messages output for Build 2023-03-AB onwards will use the new wiki URL naming convention.
The TUFLOW wiki update was competed to provide improvements including the ability to have animations within pages.
7.10.15 Warning 3551 now Check 3551
Build 2023-03-AB has changed “WARNING 3551 - SGS elevations have changes, reprocessing”, from a WARNING to a CHECK. The message remains unchanged.
7.10.16 Check 0008 now ERROR 0008
Build 2023-03-AB escalates message “CHECK 0008 -
7.10.17 Check 2099 now Reported to Log and Console
For build 2023-03-AB “CHECK 2099 - Ignored repeat application of boundary to 2D cell” is now output to the .tlf and console, prior to this build only the GIS message layer was written.
7.10.18 Bug Fix for Warning 4020
Build 2023-03-AB fixes a bug that could cause “WARNING 4020 - Should not be here - TUFLOW HPC unused AD BC” to be issued spuriously.
7.10.19 Buffered File Write for .2dm
Build 2023-03-AB uses a buffered file write for the .2dm (2D Mesh file associated with the .xmdf and .dat map output files). This aims to address an issue which could randomly show missing lines from the .2dm file causing it to be incompatible with the .xdmf / .dat file. This issue was quite rare.
With a buffered file write, instead of each line being written to the filesystem as it is processed, these are accumulated in an internal buffer (with a user definable size). As well is increasing robustness, there may also be a small performance improvement with this change. This behaviour can be controlled with the .tcf command:
If this is set to a value of 0 (or less), then buffered writing of the .2dm file is disabled.
7.11 Minor Enhancements and Bug Fixes for 2023-03-AC
7.11.1 Fix to TMax Output in Output Zone
Build 2023-03-AC fixes an issue where the time of maximum output (Tmax) was not correctly reported for output zones. This issue happened prior to 2023-03-AC if the time of maximum output was requested for output zones only, but not for the entire domain.
7.11.2 Enhancement to Plot Output Memory Allocation
Build 2023-03-AC increased the upper limit of the output data size requested by plot output, which is the number of cells selected by plot output objects (points/lines/regions) multiplied by the number of plot output timesteps. Prior to build 2023-03-AC, memory access violation occurred if the requested output data size exceeds 2,147,483,647, because a 4 byte integer was used for indexing. An 8 byte integer is now used for indexing, and users can selected upto 2,147,483,647 plot output cells and request upto 2,147,483,647 plot output timesteps.
7.11.3 Bug Fix to SGS High-Res Output below Sea Surface Level
Build 2023-03-AC fixes an issue with SGS High-Res water level output below sea surface level. The output value for the cell corner water level has been initiliased as 0.0m, which caused error in water level output for model located below the elevation of 0.0m.
7.11.4 Quadtree PO Header Rows Swapped
Build 2023-03-AC swaps the first two header rows in the _PO.csv for Quadtree. This change aligns the Quadtree output with Classic and HPC.
