Section 13 Managing and Starting Simulations
13.1 Introduction
This chapter provides guidance on model naming conventions and how TUFLOW’s powerful Variables, Events and Scenarios functionalities may be used to simulate any number of scenarios and events from a single control (.tcf) file.
The latter part of the chapter discusses the installation of TUFLOW, the different TUFLOW dongle types and how to start a simulation, both as a standalone TUFLOW simulation and in conjunction with third-party software.
Finally, there is a section on the reproducibility of results when using different hardware.
13.2 File Naming
Each hydraulic modelling study can easily generate hundreds, if not thousands, of model input files in addition to a large number of check and result files. Devising a sound naming convention as part of the modelling process is key to modelling that is easily interpreted, logical, efficient, quick to error check and quality control, and provide traceability for quality assurance.
The examples below are presented as guidance only. They demonstrate the progression of a simple model naming convention to a more complex version that incorporates different flood events and scenarios. The examples focus on the name of the .tcf control file, as this determines the prefix assigned to both the 1D and 2D check and result files.
Tip: A general recommendation is to avoid long filenames and use acronyms where possible. For example, for a simulation of the Brisbane River for the existing topography for the 1 in 100, 12 hour duration flood event, “BR_Exg_100yr_12hr_001.tcf” is preferable to “Brisbane_River_Existing_100year_12hour_001.tcf”.
Example 1: Basic
MODEL_001.tcf
In its simplest form, the names of the majority of TUFLOW models consist of a few characters denoting the study name and a version number. The characters denoting the study name are typically included in all input files to specify that the files are unique or created for this study. The numbering is used to denote different versions of the model, where each time a change is made and the model re-simulated, the version number is incremented. Use of a model version numbering system ensures it is clear which model input files generated which model output files. This is particularly important when troubleshooting or quality controlling a model.
Example 2: Event Naming
MODEL_0100F_001.tcf
MODEL_1000F_001.tcf
Most hydraulic modelling studies require the simulation of more than one event. In these cases, it is preferable to include the name of the event within the simulation name rather than simply incrementing the model version number. Note that the same number of characters is retained for the 100-year (0100F) and 1000-year (1000F) events and, in this case, the use of ‘F’ to denote the simulation is a fluvial flood event. Retaining the same number of characters (for example, by using preceding zeroes) ensures when the files are viewed in Windows Explorer, they are presented in ascending order. The model version number is the same in this example and tells the user that the same version of the model has been simulated for two different flood events.
Refer to Section 13.3.1 which presents a method to model numerous hydraulic events using a single .tcf control file.
Example 3: Scenario Naming
MODEL_0100F_EXG_001.tcf
MODEL_0100F_PRP_001.tcf
Many hydraulic modelling studies require the simulation of multiple scenarios, such as pre- and post-development topography scenarios or sensitivity testing where one or more model parameters are varied. Incorporating the name of the modelled scenario into the .tcf filename easily differentiates the output files associated with each scenario. The characters used to denote the scenario are typically also used for model input files specific to the scenario. For example, the post-development scenario ‘PRP’ may involve the raising of flood defences, hence the GIS layer used to raise the defences might be named ‘2d_zsh_MODEL_PRP_defence_001.shp’. The presence of this layer in a model simulation of the pre-development scenario ‘EXG’, will immediately highlight to the user that a mistake has been made.
Refer to Section 13.3.2 which presents a method to model one or more scenarios using a single .tcf control file.
13.3 Simulation Management
TUFLOW incorporates powerful functionality to manage the simulation of multiple events and multiple scenarios. The flexibility offered allows the user the ability to initiate all simulations using the one .tcf file should they so desire. Rather than create a new .tcf file for every simulation and a new .tgc and/or .ecf file for every different model scenario, it is possible to just have one of each of these files, or at least markedly reduce the number of these files. An example application using a single set of control files to simulate 328,986 combinations of different model domains, events and scenarios is published in the TUFLOW Insights Library.
Numerous tutorial examples (with associated data downloads) are available which demonstrate these powerful features:- eLearning - Bulk Simulation Management Scenarios, Events and Variables
- TUFLOW Wiki Tutorial 8 - Scenario Management
- TUFLOW Wiki Tutorial 9 - Event Management
- TUFLOW Wiki Example Models - Scenarios, Events and Variables
The following sections of the manual discuss the available options.
13.3.1 Events
Most hydraulic modelling studies require the simulation of different probabilistic events. For example, a flood study may have to consider the 2, 5, 10, 20, 50, 100, 200 and 500 year Average Recurrence Interval events, and one or two extreme floods. Each, or most of these, may have to be simulated using a range of different duration rainfalls. A downstream boundary such as the ocean may also need to be varied to model different probability storm tides or climate change scenarios. The final number of simulations can easily be in the hundreds, if not thousands.
The ability to simulate multiple events using a single .tcf control file not only makes management of the model easier, it also ensures consistency between the simulations and supports better project quality control.
Multiple events are set up through the creation of a TUFLOW Event File (recommended extension .tef) containing .tcf and .ecf commands that are particular to a specific event. The event specific commands are contained between Define Event and End Define commands. Global .tcf and .ecf commands can also be used by placing these commands outside the Define Event blocks (normally at the top). The .tef file is referenced in the .tcf file using the Event File command. The TUFLOW Event File is typically the last line in a .tcf file. Placing it in this location ensures it will be the final .tcf command that is read, and any global .tcf and .ecf commands within it will take precedence over conflicting commands in the parent .tcf and .ecf files.
Up to nine (9) different event groups can be specified for a single simulation. Unlimited events can be defined within an event group. Example event groups and associated events could be:
- Flood event magnitude: 2, 5, 10, 20, 50, 100, 200 and 500 year Average Recurrence Interval storms.
- Storm Duration: 1hr, 2hr, 3hr, 4hr, 6hr, 9hr, 12hr etc.
- Design event temporal pattern: TP01, TP02, TP03, TP04, TP05, TP06, TP07, TP08, TP09, TP10.
- Downstream boundary condition: LAT, HAT, 5 year, 10 year.
- Climate change time horizon: 2025, 2050, 2100.
Since the .tef file typically lists all available event options, the file can be viewed as a database of all available event possibilities. This is useful for model peer review purposes.
To automatically insert the event name(s) into the output filenames, making them unique, place “~e~” or “~e<X>~” (where <X> is from 1 to 9) into the .tcf filename.
Note ~e~ is effectively the same as ~e1~ and is typically used if there is only one event type being varied.
For example,
MODEL_~e1~_~e2~_EXG_001.tcf can be configured to run any number of events within the e1 and e2 event groups, such as:
- e1=0100F and e2=12h results in MODEL_0100F_12h_EXG_001
- e1=0100F and e2=24h results in MODEL_0100F_24h_EXG_001
- e1=1000F and e2=12h results in MODEL_1000F_12h_EXG_001
- e1=1000F and e2=24h results in MODEL_1000F_24h_EXG_001
Two options are available to define the event choice for a model simulation:
- Using a batch file (.bat): Specify the –e or –eX options (see Table
13.2) to run the multiple events using the same .tcf file – refer to the examples below. This option is best suited to executing bulk simulations.
- Using the Model Events command: Change the events manually before each simulation within the .tcf file. Note that if using the –e or -ex option, this will override any events defined using Model Events in the .tcf file. This option is useful for carrying out quick what-if or sensitivity simulations. It is not practical for bulk simulation.
The examples below demonstrate two approaches for running simulations of a 20 and 50 year flood and 1 and 2 hour duration storms.
- Example 1 uses a single event group.
- Example 2 achieves the same outcome as Example 1, though by using two event groups (~e1~ and ~e2~) it avoids some of the command repetition required for Example 1.
In the examples, note how the same command (in this case End
Time) is used within the event blocks to assign a
different simulation end time depending on the duration of the event. Also note the use of BC Event Source, which is a much more powerful alternative to using BC Event Text and BC Event Name, in that it can be used to define
up to 100 different event text / name combinations instead of being
limited to one. The key text field linkages with the BC Database file
are highlighted in yellow.
Events Example 1: One Event Group
Within the BC Database file:
Name,Source,Column 1,Column 2,Add Col 1,Mult Col 2,Add Col 2,
C001,..\Inflows\~ARI~_~DUR~.csv,Time,C001
C002,..\Inflows\~ARI~_~DUR~.csv,Time,C002
…
Include “~e~” in the .tcf filename, for example “Nile_~e~.tcf”, and add the following line at the bottom of the file:
The “Nile_Events.tef” file contains lines such as the below:
Multiple events can be simulated from a single .tcf file using a batch
file. Refer to Section 13.5.2 for details how to create a batch
file. Below is an events batch file example in its simplest form, without looping.
set RUN=start “TUFLOW” /low /wait /min “%TUFLOWEXE%” -b
%RUN% –e1 020y_01h Nile_~e~.tcf
%RUN% –e1 020y_02h Nile_~e~.tcf
%RUN% –e1 050y_01h Nile_~e~.tcf
%RUN% –e1 050y_02h Nile_~e~.tcf
Events Example 2: Two Event Groups
Within the BC Database file:
Name,Source,Column 1,Column 2,Add Col 1,Mult Col 2,Add Col 2,
C001,..\Inflows\Nile\~ARI~_~DUR~.csv,Time,C001
C002,..\Inflows\Nile\~ARI~_~DUR~.csv,Time,C002
…
Include “~e1~” and “~e2” in the .tcf filename, for example “Nile_~e1~_~e2~.tcf”, and add the following line:
The “Nile_Events.tef” file would contain lines such as the below:
To run multiple events from the one .tcf file a .bat file may be used, such as below (see Section 13.5.2):
set RUN=start “TUFLOW” /low /wait /min “%TUFLOWEXE%” -b
%RUN% –e1 020y -e2 01h Nile_~e1~_~e2~.tcf
%RUN% –e1 020y -e2 02h Nile_~e1~_~e2~.tcf
%RUN% –e1 050y -e2 01h Nile_~e1~_~e2~.tcf
%RUN% –e1 050y -e3 02h Nile_~e1~_~e2~.tcf
If Event logic blocks are also permitted in the same manner as the If Scenario command. Refer to Section 13.3.2 for further information. Also, events are automatically defined as variables and can be used, for example, to set the output folder name. See Section 13.3.3 for information on “Variables”.
Reminder: The location of Event File within the .tcf
file and the location of the commands within the .tef file are
important. If, for example End Time occurs in the
.tcf file after where Event File occurs, this latter
occurrence of End Time will prevail over those specified
in the .tef file. Essentially, the commands from the .tef file (for the
events specified using the –e or -e
Tip: To distinguish between 1D and 2D commands in a .tef file, prefix .ecf commands by “1D” followed by a space. In the examples below, the .ecf Output Folder command is used to set different folders for the 1D results depending on the event.
13.3.2 Scenarios
Using If Scenario is similar to using If…Else If… Else…End If constructs in a programming or macro language. This feature enables control of the commands to include or exclude depending on the scenario or combination of scenarios specified by the user at the time of simulation execution.
Scenarios can be added into most of TUFLOW’s control files. The only control files not supporting scenarios are the TUFLOW Rainfall Control File (.trcf), TUFLOW External Stress File (.tesf), Advection Control File (.adcf) or within a Define Control command inside a TUFLOW Operational Control (.toc) file. Similar to “events”, 9 scenario groups can be specified. Unlimited scenario variations can be defined within each scenario group.
In Example 1 below, an If Scenario logic block has
been inserted into a .tgc file:
Scenarios Example 1:
TUFLOW will carry out the following steps:
- Set all material values over the entire 2D domain to a value of 1.
- Process the 2d_mat_existing.shp layer to assign material values from
a layer of land-use polygons representing the existing situation.
- One of the following will then occur:
- If either “–s1 opA” was specified as a run selection in a batch file, or “Model Scenarios
== opA” was specified in the .tcf file, then the 2d_mat_opA.shp layer would be processed to modify the material values affected by the Option A scenario. If “~s1~” occurs within the .tcf filename, it is replaced by “opA” in the output filenames, otherwise “opA” is appended to the end of the .tcf filename for the output filenames.
- If opA was not specified as the chosen scenario for a simulation, 2d_mat_opA.shp would be ignored. For example, that option would be required if modelling the existing scenario.
- If either “–s1 opA” was specified as a run selection in a batch file, or “Model Scenarios
Example 1 can be extended to include an Option B, where Option B is an extension of Option A, and also Option C, which is a complete alternative to Option A and the combined Option A and B design. The If Scenario logic block for this situation is shown in Example 2a below:
Scenarios Example 2a:
In the above example:
- If “–s1 opA” is selected as the scenario for simulation, only 2d_mat_opA.shp will be read.
- If “–s1 opB” is selected as the scenario for simulation, 2d_mat_opA.shp and 2d_mat_opB.shp will be read.
- If “–s1 opC” is selected as the scenario for simulation, only 2d_mat_opC.shp will be read.
Example 2b has been expanded further to include an Else and Pause command in 2c. The command in the Else section will apply if any scenario except opA, opB or opC is called. The Pause command will halt the simulation initialisation and display the specified message in the DOS window. The user then has the option to continue or discontinue the simulation. It is good practice to include Pause commands as a way to avoid accidental use of unspecified scenarios.
Scenarios Example 2c:
In the example above, the batch file used to run the Option A, Option B and Option C scenarios may look like this:
set RUN=start “TUFLOW” /low /wait /min “%TUFLOWEXE%” -b
%RUN% –b -s1 opA Nile_~s1~_001.tcf
%RUN% –b –s1 opB Nile_~s1~_001.tcf
%RUN% –b –s1 opC Nile_~s1~_001.tcf
As demonstrated above, it is recommended that scenario group placeholders be used (~s1~, ~s2~ etc.) in the .tcf file name. This provides the modeller with increased output file name control. Above, ~s1~ in the .tcf file name will be substituted with the chosen scenario in all result, log and check files. For example, for the first simulation in the queue, Nile_~s1~__001.tcf would become M01_opA_001 for all output files.
The If Scenario command can be nested up to 10 levels. An example is provided in Scenario Example 3.
Scenarios Example 3:
Scenarios are automatically defined as “Variables” and can be used, for example, to set the output folder name. See Section 13.3.3 for information on Variables.
13.3.3 Variables
The Set Variable command can be used to assign different values for the same variable in different parts of the various TUFLOW control files. It is an efficient way to centralise information. It is regularly used to consolidate the definition of multiple variables in a single logic block rather than having multiple If..Else If.. End If command blocks in different locations or control files.
The Set Variable command can only be specified in the .tcf or within a read file (.trd) referenced from the .tcf. The variables can then be used in all control files (.tcf, .tgc, .tbc, .ecf, etc). The variables are referenced within the control files by being enclosing the character string within “<<” and “>>”.
An example is provided below. Variables associated with modelling at different cell sizes are defined in the .tcf within a single logic block.
Along with this variable definition, the following is used in the respective control files:
- .tcf: Timestep
== <<2D_TIMESTEP>> - .tcf: Grid Output Cell Size
== <<2D_CELL_SIZE>> - .tgc: Cell Size
== <<2D_CELL_SIZE>>
Refer to TUFLOW Wiki Example Models - Variables for various examples of variables in a working model.
Note: Set Variable scenario blocks are often contained within “read files”, referenced in the .tcf using the Read File command.
All events and scenarios are automatically set as a variable that can be used within your control files. For example, if your model results are to be output to different folders depending on Scenario 1 (~s1~), enter the following into the .tcf file noting the use of << and >> to delineate the variable name.
Output Folder
In the case above, if Scenario ~s1~ is set to “OpA”, TUFLOW automatically sets a variable named “~s1~” to a value of “OpA”, and the output will be directed to ..\results\OpA.
As an extension to the example above, if the output folder is to also include the first event name, which, for example, is the return period of the flood, the following could be used:
Output Folder
If Event ~e1~ is set to “Q100”, the output will be directed to ..\results\Q100_OpA.
13.4 TUFLOW Executable Download
13.4.1 Overview and Where to Install
The TUFLOW release consists of two different versions of the executable as follows:
- TUFLOW_iSP_w64.exe
- TUFLOW_iDP_w64.exe
The naming convention for each executable is explained in Table 13.1. Each version has associated TUFLOW .dll and .ptx files, and several system .dll files.
For each version all .exe, .dll and .ptx files must be placed in the same folder and kept together at all times. Older versions of TUFLOW may not have any .ptx files (these are associated with GPU based simulations. When replacing with a new build, archive the files by creating a folder of the same name as the Build ID (e.g. 2023-03-AD), and place all files in this folder.
The Build ID includes the acronym and appears in the top bar of the Console window, in the .tlf file and elsewhere. For example, if running the single precision, Windows 64-bit version, the Build ID would be “2023-03-AD-iSP-w64”.
Older builds can always be setup in a similar manner by placing the build in its own folder, if they are needed for running legacy models.
Model TUFLOW Build or a combination of Model TUFLOW Release, Model Precision and Model Platform can be used to force a simulation to use a specific TUFLOW build, release, or precision (SP or DP). This can be very useful for ensuring a consistent TUFLOW build/release/version is used for a project.
It is recommended that a folder structure such as that shown in Figure 13.1 is used. If an update (patch) is issued, archive the old build by moving all files to a separate folder or .zip file, and copy in the new TUFLOW files. This way, batch files, text editors and GIS packages that are used to start TUFLOW simulations will access the latest build. Previous versions can still be used if required.
Figure 13.1: TUFLOW Engine Files
| Version | Description |
|---|---|
|
Single Precision (iSP) |
Compiled using single precision (4-byte) real numbers that typically have around 7 significant figures. This version is recommended for the majority of TUFLOW simulations. See Section 13.4.2 for discussion on precision. |
|
Double Precision (iDP) |
Compiled using double precision (8-byte) real numbers that typically have around 16 significant figures. See Section 13.4.2 for discussion on precision. For TUFLOW Classic, this version should be used for models with high ground elevations (greater than 100 to 1,000m) and for direct rainfall models. If a model falls into these categories and is experiencing mass errors when using the single precision version, re-run using the double precision version and should the mass errors reduce to acceptable levels, continue to use double precision, otherwise contact support@tuflow.com. For TUFLOW HPC, due to its depth based formulation (rather than using elevation as required for implicit schemes like Classic), it double precision rarely needs to be used. However, for long-running simulations involving millions of timesteps, subsurface water levels can be subject to round-off errors during the integration process. Should mass volume errors occur, or if you are concerned about the accuracy of ground water levels, run in double precision and compare with single precision. Should the mass errors reduce to acceptable levels, continue to use double precision, otherwise contact support@tuflow.com. Model Precision can be used to ensure that a simulation is started using the single or double precision version. For example, a single precision version of TUFLOW will generate an error if used to start the simulation if the command below is included in the .tcf: Model Precision |
|
Windows 32-bit (w32) |
Legacy builds that were compiled as a 32-bit process. w32 TUFLOW builds can be run on Windows 32-bit and 64-bit platforms. All builds/releases prior to the 2010-10 release (except for the 2009-07-XX-iDP-64 build) are 32-bit. 32-bit programs are unable to access as much memory (RAM) as 64-bit versions of TUFLOW. For large models TUFLOW may generate an error that it is unable to allocate enough memory. Typically, when the memory requirement is ~1.5GB per simulation the 32-bit version of TUFLOW will not be able to allocate enough RAM. The 32-bit version of TUFLOW is only available for pre 2017-09 releases. Users should use the 64-bit versions where possible. |
|
Windows 64-bit (w64) |
Compiled as a 64-bit process. w64 TUFLOW builds can only be run on Windows 64-bit platforms. w64 2010-10 builds have been found to reduce simulation times by 20 to 50% compared with the 2009-07 (32-bit) release. Slightly different results (mostly fractions of a mm) will occur between w32 and w64 for the same model. Model Platform can be used to force a simulation to use a w32 or w64 version. Note that the 64-bit versions of TUFLOW may only be run using a WIBU dongle . |
Running TUFLOW is carried out by initiating the TUFLOW_X.exe file using one of the approaches discussed in Section 13.5.2. The system .dll files are required for the following purposes:
- TUFLOW_LINK_X.dll allows other schemes such as 12D DDA, FloodModeller 1D, EPA SWMM and XPSWMM to dynamically link with TUFLOW.
- TUFLOW_USER_DEFINED_X.dll allows users to customise TUFLOW to suit
their purposes (see Section 13.4.3).
- TUFLOW_AD_X.dll contains the AD (advection-dispersion) module
algorithms.
- TUFLOW_MORPHOLOGY_X.dll contains the morphology module algorithms. These are deprecated and retained for backward compatibility.
- Other supplied .dlls are system DLLs required by TUFLOW.
A number of .ptx files are required to run TUFLOW HPC/Quadtree simulations on a GPU card, these are:
- hpcKernels_nSP.ptx
- hpcKernels_nDP.ptx
- qpcKernels_nSP.ptx
- qpcKernels_nDP.ptx
As a general rule, once a build is over three years old, it is no longer supported. It can still be used, but there is no guarantee that older builds will recognise newer dongles or are updated with bug fixes.
13.4.2 Single and Double Precision
When storing floating point values on a computer, a certain number of bytes per value is needed. Single precision numbers use 4 bytes and double precision 8 bytes. This will yield from 6 to 9 digits of precision for single precision and 15 to 17 digits for double. TUFLOW Classic and TUFLOW HPC are available in both single precision and double precision, however, the reasons why you would use double precision instead of single are somewhat different for the two solvers.
13.4.2.1 TUFLOW Classic
Being an implicit matrix based solution, TUFLOW Classic solves for water levels rather than depths. The affect of this is best illustrated by an example. Consider a model where the highest ground level is under 10mAD. If a tiny amount of rainfall falls on the cell, this might raise the water level in the cell to 5.000001mAD. Both single and double precision will handle the accuracy needed as the resulting water level falls within the acceptable number of significant figures for both precisions.
However, consider another model with much higher elevations, and a cell that has a water level of 1000.000mAD. If this cell receives 0.000001m of rainfall in one timestep, for single precision the resulting water level will be 1000.000mAD (i.e. the added rainfall has disappeared resulting in an incorrect calculation and associated mass error). However, for double precision the resulting water level is 1000.00000100000, which is the correct water level and mass is conserved.
The need to use double precision for Classic also depends on other factors such as cell size and timestep. As there is no simple rule-of-thumb, the recommended action is to run the model in single and double precision modes. If the results are unacceptably different, and especially if the mass error is significantly lower in double precision, then double precision should be used.
Note that the choice of single or double precision also impacts on simulation times and RAM allocation. iDP versions of TUFLOW Classic will take approximately 25% longer to run and require up to twice as much memory, limiting the ability to run concurrent simulations. Therefore, if the results of a model run in both iSP and iDP versions are consistent, the iSP version of TUFLOW is recommended to take advantage of the faster simulation times and smaller memory footprint.
13.4.2.2 TUFLOW HPC
Being an explicit solution, the calculation method in TUFLOW HPC uses depth, unlike TUFLOW Classic which uses water level as discussed above. This means that precision issues associated with, for example, applying a very small rainfall to a high elevation are not applicable in TUFLOW HPC, resulting in most models being accurately solved using single precision. However, carrying out simulations in single and double precision and checking for consistency is still a recommended task, especially if the simulation is using for example, very small inflows or other metrics that are unusually fine-scale.
Unless testing shows otherwise, use the single precision version for all TUFLOW HPC simulations. Note: Double precision solutions on GPU cards can be four times slower than single precision on some GPU devices! If the TUFLOW HPC simulation is started with the single precision version of TUFLOW, the HPC solver will utilise a single precision version. If the simulation is started with the double precision version of TUFLOW, error message ERROR 2420 will be output by default stating that the single precision version should be used. This can be overridden by using the HPC DP Check
13.4.3 Customising TUFLOW using TUFLOW_USER_DEFINED.dll
The TUFLOW_USER_DEFINED.dll is a legacy feature for TUFLOW Classic that was primarily used for customising hazard category outputs and is no longer supported as the same service cannot be offered for HPC given its GPU code base.
Note: For customising hazard category outputs, see Section 11.2.3.1.
13.5 Running Simulations
13.5.1 Dongle Types and Setup
BMT dongles are generally distributed as a WIBU Codemeter (metal) dongle or software licence. These dongles were introduced in 2010. The older ‘SoftLok’ dongles are no longer supported.
Two types of licences are currently provided:
- Local or Standalone Licence – Allows up to a specified limit the
number of TUFLOW simulations to be run from the one computer. For
example, a Local 1 licence will allow a single simulation to be
performed on the machine. A local 4 licence allows up to 4 concurrent
simulations. Local 1 to Local 16 licences can be configured.
- Network Licence – allows for multiple TUFLOW processes running at any one time across multiple machines within the limitations of the TUFLOW EULA. For example, a Network 5 licence allows up to 5 concurrent simulations to be performed. This could be 5 simulations on a single computer, 5 simulations with each one on a different computer, or anything in between.
Refer to the installation instructions on the TUFLOW Wiki for both types of dongles.
13.5.1.1 Protocols for Accessing Dongles
If more than one type of dongle is available the protocols for taking and checking licences are:
- WIBU Codemeter licences are searched for, and if a licence is free
it is taken.
- If no licence is available, you can optionally set for TUFLOW to
continue to try and find an available WIBU dongle licence. This is
achieved using the “C:\BMT\TUFLOW_Dongle_Settings.dcf” file
described below. This is useful if there are no free licences and
you wish to start a simulation (the simulation will start once a
free licence becomes available).
- Once a simulation is under way, and if the licence is lost, TUFLOW will try to regain a licence, but it cannot switch to a different dongle type / provider. For example, a WIBU dongle can’t be used to finish a simulation if it is started with another WIBU dongle which is removed.
There are five varieties of licence type / vendor:
- BMT physical USB lock (dongle)
- BMT software lock (tied to a particular machine)
- Aquaveo (SMS) USB lock
- Jacobs (Flood Modeller) USB lock
- Jacobs (Flood Modeller) software lock
With numerous licencing options available, setting the preferred licence type can (slightly) speed the simulation start-up. The licence search order can be set via a licence control file “TUFLOW_licence_settings.lcf”, this replaces the TUFLOW_Dongle_Settings.dcf. Note that this file can occur in several locations. When looking for a licence setting file TUFLOW searches in the following locations:
- A “TUFLOW_licence_settings.lcf” in the same location as the TUFLOW executable.
- A “TUFLOW_licence_settings.lcf” in a TUFLOW folder in the “ProgramData” environment variable location. By default this will resolve to C:\ProgramData\TUFLOW\TUFLOW_licence_settings.lcf.
Entering %programdata% into the windows explorer path will take you to the location.
- C:\BMT\TUFLOW_Licence_Settings.lcf
- C:\BMT\TUFLOW_Dongle_Settings.dcf
If no licence settings files are found, TUFLOW defaults to the order listed above. WIBU Firm Code Search Order can be used to control the search order in the TUFLOW_licence_settings.lcf file.
Note: Use of C:\BMT\ above is for legacy reasons and is not recommended as read/write access to local drives is now often blocked for protection against hackers.
13.5.1.2 TUFLOW_Licence_Settings.lcf File
The TUFLOW Licence Settings (.lcf) file can be used to set WIBU settings, such as retry time interval and count. It has the same form and notation as a TUFLOW .tcf file.
This file is optional and if not found the settings below are the default:
In the above, the:
Simulations Log Folder sets the folder path or URL to a folder for logging all simulations. If the keywords “DO NOT USE” occur within the folder path or URL, this feature is disabled. Also see Simulations Log Folder.
WIBU Retry Time sets the interval in seconds for retrying to take a licence or regain a lost licence. The default is 60 and values less than 3 are set to 3.
WIBU Retry Count sets the number of times to retry for a licence at the start of a simulation. By default, if a licence is lost during the simulation, TUFLOW tries indefinitely to regain a licence so as not to lose the simulation.
WIBU Dongles Only if set to ON will force TUFLOW to only search for WIBU dongles.
13.5.1.3 Dongle Failure during a Simulation
If TUFLOW fails to recognise a network lock during a simulation (e.g. the network dongle server computer is down) it enters a holding pattern and continues trying until a license is found.
For local, standalone locks, TUFLOW prompts with a message that the lock could not be found. If it’s a USB lock try a different USB port, and press Enter to continue.
13.5.2 Starting a Simulation
There are a number of ways TUFLOW can be started, in each case the TUFLOW executable is started with the TUFLOW control file (.tcf) as the input argument. There are a number of optional switches that can be specified when starting a simulation; these are outlined in Table 13.2.
Starting a TUFLOW simulation can be carried out in a multitude of methods:
- From within a text editor such as Notepad++
- Through a batch file
- From within your GIS software:
- QGIS via the QGIS TUFLOW Plugin
- ArcMap via the use of the ArcTUFLOW toolbox
- MapInfo (through the MiTools add-on)
- QGIS via the QGIS TUFLOW Plugin
- From a Console (DOS) Window
- Via the right mouse button in Microsoft Explorer
- Via a runner such as the TUFLOW Runner or TRIM
This section of the manual only provides detailed instructions on initiating simulations through batch files as it is the most effective method to run several or many simulations. Instructions on all methods have been described on the TUFLOW Wiki.
13.5.2.1 Batch File Example and Run Options (Switches)
A batch file is a text file that contains a series of commands or instructions that is read by the Windows Operating System. The batch file requires the extension .bat to be recognised by Windows.
The simplest format of a TUFLOW batch file is to specify a single simulation. This is of the format:
For example:
The .bat file is run or opened by double clicking on it in Windows Explorer. This opens a Console Window and then executes each line of the .bat file. The above batch file will start the TUFLOW executable with the control file “MR_H99_C25_Q100.tcf” as the input file. As the control file argument does not have a file path, the batch file must be located in the same folder as the .tcf file. This could also include a relative or absolute file path, for example in the batch file line below, the batch file could be stored in any folder as the absolute file path is used. As outlined in Section 4.2.2, the relative file path is recommended for TUFLOW modelling inputs.
To start multiple simulations one after the other, these can be listed in succession.
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe –b MR_H99_C25_Q050.tcf
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe –b MR_H99_C25_Q020.tcf
pause
Note the use of the –b (batch) switch, which suppresses the need to press the return key at the end of a simulation. This ensures that one simulation proceeds on to the next without any need for user input. The pause at the end stops the Console window from closing automatically after completion of the last simulations so you can check the status of the simulations.
The –t (test) switch is very useful for testing the data input without running the simulation. It is good practice to use this switch before carrying out the simulations, as this will tell you whether there are any data input problems. The –t switch runs TUFLOW to just before it starts the hydrodynamic computations.
Using the example above, the recommended approach is to first run the following batch file:
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe -b –t
MR_H99_C25_Q050.tcf
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe -b –t
MR_H99_C25_Q020.tcf
pause
This will indicate any input problems (note some WARNINGS do not require a “press return key”, but they can be located in the .tlf file).
From TUFLOW version 2018-03-AA, it is possible to use the test model option without using a TUFLOW licence noting no diagnostics are reported so don’t use this option whilst you are developing/building a model. To utilise this licence free test, the -nlc (No Licence Check) input argument must be specified, e.g. the command line below would run the model in test model (-t) without needing a licence (-nlc):
To carry out the simulations the –t can be removed (and -nlc flag if used) or replaced with the –x (execute) switch. The -x switch is optional, but is useful when editing the .bat file to quickly change between –t and –x switches.
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe -b –x MR_H99_C25_Q050.tcf
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe -b –x MR_H99_C25_Q020.tcf
pause
Comment lines are specified in a .bat file using “REM” (remark) in the first column. Alternatively, “::” has a similar effect. The ss64 website has further guidance on using comment lines in batch files. For example, if you want to re-run only the first simulation in the examples above, edit the file as follows:
:: C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe -b –x MR_H99_C25_Q050.tcf
:: C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe -b –x MR_H99_C25_Q020.tcf
pause
Table 13.2 below describes other switches that are available. The switches are also displayed to the console if TUFLOW.exe is executed without any arguments (i.e. double click on TUFLOW.exe in Windows Explorer).
Any of the options can be prefixed by a:
- “-” (short dash or minus sign);
- “-” (long dash); or
- “/” (forward slash)
For example, to start a model in batch mode, the following all perform the exact same function:
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe –b MR_H99_C25_Q100.tcf
C:\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe /b MR_H99_C25_Q100.tcf
| Switch | Description |
|---|---|
| -acf | Automatically create folders (the default). Prevents the dialog prompt from appearing when encountering non-existent folders (ie. results folders), and creates these folders automatically. If for any reason the folder can’t be created, a dialog will appear. |
| -b | Batch mode. Used when running two or more simulations in succession from a .bat file (see Section 13.5.2.1). Suppresses display of the TUFLOW “simulation has finished” dialogue window, or a request to press Enter at the end of a simulation, so that the .bat file can continue onto the next simulation. |
|
-c Optional additional flags of “a” , “L”, “p” or “ncf”. (e.g. -c, -cap, -cpncf, -cncf) |
A copy of a TUFLOW model can be created as described below. Making a copy of a model is useful for transferring a model to another site or for making an archive of the input data. Also see the -pm (package model) option as an alternative. To copy a TUFLOW model, the -c switch must be included on the TUFLOW command line, as a minimum. The -c switch copies only the files read by TUFLOW. Therefore, for MapInfo users, the .mif and .mid files read by TUFLOW will be copied. To copy the remaining MapInfo format files (.tab, .id, .dat and .map) use the -ca option described further below. Additional optional flags can be added to the base -c switch, in any combination, including:
The addition of the “a” flag (e.g. -ca) copies all files of the same name for all input files (i.e. same name, but different extensions). This option is particularly useful if the .tab and other associated files of a GIS layer need to be archived or delivered. The addition of the “L” flag will output the files used by TUFLOW into a .tcl (TUFLOW Copy List) file but not copy the files to a destination folder. This can be useful if scripting the copying of models. To run the copy list the character “L” needs to be specified after the -c input argument. This works for all copy options, for example the following are all valid; -cL, -caL -capL. The .tcl file produced is output in the same directory as the .tcf and takes the simulation name. The addition of the “p” flag (e.g. -cp) allows the user to specify an alternate path in which to copy the model. Without this flag, the location defaults to the .tcf’s location. For example, specifying the following, will place a copy of the model into a folder C:\put_model_here: “TUFLOW.exe” -cp “C:\put_model_here” “C:\TUFLOW\runs\M01_5m_003.tcf” The addition of the “ncf” flag (e.g. -cncf) copies the essential input files and excludes all check files. Note that these optional flags can be added in any combination to the base -c switch (e.g. -c, -ca, -cp, -cncf, -cap, -cancf, -cpncf, -capncf). Specifying -c on the TUFLOW command line creates a folder “\(\lt\).tcf filename\(\gt\)_copy” (or “\(\lt\).tcf filename\(\gt\)_copy_all” if the “a” flag is added) in the same location as the .tcf file. Under the folder, input files are copied (including the full folder structure), and any check files and output folders created. For example, specifying: TUFLOW.exe -c “C:\tuflow_models\my model.tcf” will make a copy of the TUFLOW model based on the file “my model.tcf” in a folder “my model.tcf_copy”, or “my model.tcf_copy_all” if using the “a” flag. Notes:
|
| -e<name> -e{1-9}<name> |
Specifies an event name to be used by Define Event in a .tef Event File to customise inputs for an event. There must be a space between –e and <name>. <name> may itself contain spaces, but if it does the event name must be enclosed in quotes. More than one (up to a maximum of nine) events per simulation may be specified by placing a number after –e. –e and –e1 are treated the same (don’t specify both of them otherwise indeterminate results may occur). Also see Section 13.3.1 and Model Events. Examples:–e Q100 Run the Q100 event. –e1 Q100 –e2 02h Run the Q100, 2 hour storm event. –e Q100 –e2 02h Same as above. –e “100 yr” Run the event name “100 yr” noting quotes required as there is a space in the name. |
| -et<time_in_hours> | The end time for a simulation can be specified using the run option -et<time_in_hours>. Any end time specified via this run option argument is given the highest priority and overrides the End Time settings in the .tcf, event files (.tef) and override files. |
| -nc |
The use of the -nc (no console) switch suppresses the appearance (hides) the console window (Section 14.2). You won’t be able to see the simulation running, however there will be a TUFLOW process visible in Windows Task Manager. The -nc switch automatically invokes the -nmb and -b switches. The “start” command included in the Advanced Batch File examples in Section 13.5.2.3 should not be used with the -nc command. It is strongly recommended to redirect standard console window output to a text file. It’s recommended that this file be given a unique name for each run, matching your simulation. “…TUFLOW.exe” –nc my_run.tcf > my_run.txt Build 2018-03-AB included some enhancements when running with the -nc (no console option). These were designed to remove any need for user input to TUFLOW as follows:
|
| -nlc | For Build 2018-03-AA or later it is possible to use the copy model (-c option) or test model (-t option) without using a licence. To utiltise this licence free copy / test, the -nlc (No Licence Check) input argument must be specified. If running without a TUFLOW licence no diagnostic output is generated (e.g. messages layer). If these are required, the -nlc option must be removed. |
| -nmb | No Message Boxes. Suppress use of windows message boxes to prompt the user. All prompts will be via the console window. |
| -nq | The use of the -nq (no queries) switch prevents the termination query dialog from displaying when Ctrl+C is pressed to terminate a simulation cleanly. If –nq is specified and Ctrl-C is pressed, the simulation terminates cleanly without a query dialog to check you are certain, so be careful! |
| -nt | Sets the number of CPU threads (cores) to use for a TUFLOW HPC simulation. Noting that the number of threads requested is limited to the maximum number of CPU cores available on the machine, and the available TUFLOW Thread licences. |
| -nwk | Force TUFLOW to search for a network licence (i.e. skip the search for a local licence). |
| -od<drive> | The Output Drive for a simulation can be specified with the –od command line option. For example –odC will redirect all outputs to the C:\ drive. |
| -oz<name> |
Map output includes Output Zone <name>. For more information on Output Zones, please refer to Section 11.2.5. For example “-oz ZoneA” would include output for Zone A, similar to the .tcf command: Model Output Zones |
|
-pm Optional additional flags of “All” , “L” or “ini”. |
The package model (-pm) function copies all input files for all events and scenarios defined in the model. Unlike the -c option, -pm does not read and process the data during the file copy. As such it is substantially faster than -c. The package model function does not require access to a TUFLOW licence. Optional switches are:
Combinations of the above are also valid, with the order of the optional switches not being important (-pmAllL would be treated the same as -pmLAll). Three options are also available for handling of the binary processed files created by TUFLOW to speed up the simulation start. These options are:
For example, the command line below will package the model but not include any .xf files. TUFLOW_iSP_w64.exe -pm -xf0 PR_~s1~_~e1~_~e2~_001.tcf When using package model the default destination folder is created in the same directory as the .tcf file, with the prefix “pm_”. For example, C:\Projects\Modelling\TUFLOW\runs\Run_001.tcf will create a package in the folder “C:\Projects\Modelling\TUFLOW\runs\pm_Run_001”. A .ini file can be used to overwrite the default base and destination folders. A .ini file is specified by including the optional ini argument after -pm followed by the ini file name. For example: TUFLOW_iSP_w64.exe -pmini package.ini PR_~s1~_~e1~_~e2~_001.tcf Valid commands in the .ini file are:
|
| -pu<id> |
Used to select which processing units to direct the simulation towards. At present this only applies to the HPC GPU solver where a simulation is to be directed to a particular GPU card or cards. –pu must be specified once for each device. For example, to direct the simulation to GPU devices 0 and 2, specify –pu0 –pu2. Can be used in place of the .tcf command GPU Device IDs. Note that the numbering starts at 0 for GPU devices. Override files (see Section 4.2.16) can be used to control the device ID’s used by a range of computers accessing the same control file. |
| -qcf | Query the creation of a folder. If you would prefer to have the create folder query dialog to appear (rather than TUFLOW automatically creating folders - see -acf), you can specify the –qcf run time option. |
|
-s<name> -s{1-9}<name> |
Specifies a scenario name to be used by If Scenario blocks to include/exclude commands. There must be a space between –s and <name>. <name> may itself contain spaces, but if it does the scenario name must be enclosed in quotes. More than one (maximum of nine) scenarios per simulation maybe specified by placing a number after –s. –s and –s1 are treated the same, so don’t specify both of them otherwise indeterminate results may occur. Also see Section 13.3.2 and Model Scenarios. Examples:–s exg Process all exg scenario commands. –s1 opA –s2 opB Process all opA and opB scenarios. –s opA –s2opB Same as above. –s “Option A” Quotes required as there is a space in the scenario name. |
| -slp | Simulation Log Path is a legacy option for Softlok (blue) dongles to set the path to a folder on the intranet to log all simulations initiated from the lock. Refer to the 2018 manual or earlier for details. (See Section 13.5.1.2 for equivalent option for WIBU Codemeter licences, and also Simulations Log Folder.) |
| -st<time_in_hours> | The start time for a simulation can be specified using the run option -st<time_in_hours>. Any start time specified via this run option argument is given the highest priority and overrides the Start Time settings in the .tcf, event files (.tef) and override files. |
| -t | Test input only. Processes all input data including writing of check files, but does not start the simulation. Useful for checking that simulations in a .bat file all initialise without error, prior to carrying out the simulations (especially when the runs will take all night or all weekend and you made a change you forgot to test!). |
| -wibu | Search for a WIBU Codemeter licence only. |
| -x | eXecute the simulation (the default). |
13.5.2.2 Copy/Package Model from Batch Files
TUFLOW can be run in copy mode, which can be useful for transferring a model to another site or for making an archive of the input data. To copy a TUFLOW model, the -c switch must be included on the TUFLOW command line. The -nlc (no licence check) flag can be used to copy a model without the need for a TUFLOW licence. Further details can be found here.
A licence free package model function that copies all input files for all events and scenarios defined in the model is also available using the -pm switch. If only a single set of events and scenarios is required, -c is preferable. Unlike the -c option, -pm does not read and process the data during the file copy, as such it is substantially faster than -c. The package model function does not require access to a TUFLOW licence. More information on the Package model function can be found here.
13.5.2.3 Advanced Batch Files
Batch files can be set up so that they are more generic and easily customised when moving from one project to another. Within the example below, a variable, TUFLOWEXE, is used to define the path to the TUFLOW executable, and a variable RUN is used to incorporate options such as starting the simulation on a low priority and to minimise the console window when the process starts (/min option).
set RUN=start “TUFLOW” /low /wait /min “%TUFLOWEXE%” -b
%RUN% MR_H99_C25_Q100.tcf
%RUN% MR_H99_C25_Q050.tcf
%RUN% MR_H99_C25_Q020.tcf
The advantage of using variables in batch files is if the path to the TUFLOW executable changes, or to run a different version of TUFLOW, it is just a simple single line change in the batch file. In the above, note the use of quotes around %TUFLOWEXE% in the definition for the RUN variable – quotes are needed around file path names whenever they contain a space. Also note there is no space between the variable name and the equals sign.
For examples of batch files used for running multiple events and different scenarios from the same .tcf file, see Sections 13.3.1 and 13.3.2.
It is also possible to create control logic such as loops in a batch file. For example, this could be used to loop through a large number of events and scenarios such as the example here.
SetLocal
:: Set up variables
set TUFLOWEXE=C:\Program Files\TUFLOW\Releases\2023-03\TUFLOW_iSP_w64.exe
set RUN=start “TUFLOW” /low /wait /min “%TUFLOWEXE%” -b
set A=Q010 Q020 Q050 Q100 Q200
set B=10min 30min 60min 120min 270min
:: Loop through each simulation
FOR %%a in (%A%) do (
FOR %%b in (%B%) DO (
%RUN% -e1 %%a -e2 %%b filename_~e1~_~e2~.tcf
)
)
pause
Further guidance on advanced batch files, including looping examples can be found on this page of the TUFLOW Wiki.
13.5.3 Running TUFLOW HPC
The front end of TUFLOW Classic and TUFLOW HPC are identical. No modification of the model input is needed to utilise the TUFLOW HPC solver (in CPU or GPU compute mode). Similarly, the output data is written by TUFLOW in the same output formats and data types irrespective of whether Classic or HPC solvers are used.
The .tcf Solution Scheme command is used to switch the solution scheme from TUFLOW Classic (the default) to TUFLOW HPC.
Hardware is used to access GPU hardware (CPU is the default). As such, to convert a model from TUFLOW Classic to TUFLOW HPC and run using GPU hardware only requires two additional TCF commands in most cases:
With TUFLOW HPC it is possible to use multi-GPU cards to split the simulation across multiple cards to reduce run times or access more GPU memory. This is carried out either by using the GPU Device IDs command or with the processing unit command line switch “-pu
TUFLOW HPC simulations are started in the same manner as a standard TUFLOW simulation and explained in Section 13.5.2.
13.5.3.1 TUFLOW HPC and GPU Module Commands
The following .tcf commands (apart from Timestep) are commands specific to the TUFLOW HPC solver:
| Command | Description |
|---|---|
| Control Number Factor | The default HPC Courant, shallow wave celerity and diffusion control number limits can be reduced to effectively underclock the simulation. Using the above command factors all three control numbers. For example, a value of 0.8 reduces the default limits by 20%. Reducing the control number limits may be useful if the simulation is exhibiting erratic behaviour or numerical “noise”, although testing has found this is rare in real-world models, and if occurring is more likely to be a sign of poor data or poor model schematisation. |
| Hardware |
TUFLOW HPC can be run using CPU or GPU hardware. This command defines the hardware to be used for the compute. CPU is the default and will run a simulation using the Central Processing Unit (CPU). GPU will run the simulation using the Graphics Processor Unit (GPU). When running TUFLOW HPC using the GPU hardware module, the pre (reading of data) and post processing (writing outputs) is managed by the standard TUFLOW CPU engine. This allows the user to utilise the extensive range of GIS input functionality available in TUFLOW. Simulation using GPU hardware requires the GPU Hardware Module in addition to a standard TUFLOW licence. |
| HPC DP Check |
TUFLOW HPC is available in both single precision and double precision. If the simulation is started with the single precision version of TUFLOW, the HPC solver will utilise a single precision version. If the simulation is started with the double precision version of TUFLOW, error message ERROR 2420 will be output by default stating that the single precision version should be used. The calculation method in TUFLOW HPC uses depth due to its explicit nature, unlike TUFLOW Classic that uses water level due to its implicit scheme. This means that precision issues when using the Classic solver, such as when applying a very small rainfall to a high elevation, are not applicable in HPC. Unless testing shows that single and double precision produce significantly different results, use the single precision version of TUFLOW for all TUFLOW HPC simulations. Note: Double precision solutions on GPU cards can be four times slower than single precision! Also, NVIDIA Cards with a compute capability of 1.2 or less are only able to run single precision versions. The HPC DP Check command allows the user to disable the default setting of issuing ERROR 2420 should double precision be required. |
| HPC Temporal Scheme |
Sets the order of the temporal solution. The default is the recommended 4th order temporal solution, therefore this command is usually not specified. We recommend the use of the 4th order temporal scheme as it is unconditionally stable with adaptive timestepping turned on and has been found to give accurate results and is results are not affected due to changing the timestepping. Lower order schemes save a little on memory requirements but are more prone to instability and in some cases unreliable results that are sensitive to the timestepping intervals. |
| GPU Device IDs | Controls the GPU device or devices to be used for the simulation if multiple CUDA enabled GPU cards are available in the computer or on the GPU itself. |
| Solution Scheme | This command is used to select the desired solution scheme for the compute. Use “HPC” to call TUFLOW HPC’s finite volume 2nd order solver (recommended over the 1st order alternative). |
| Timestep |
If adaptive timestepping is active (the default), the timestep value is only used for the very first step. To allow the same command to be used for either a TUFLOW Classic or a HPC simulation the timestep value is divided by 10 for the initial HPC timestep, therefore, enter a timestep value similar to that that you would use for TUFLOW Classic. If adaptive timestepping is off, sets the fixed timestep. The timestep will always be much smaller than TUFLOW Classic’s timesteps as the scheme is explicit (TUFLOW uses an implicit scheme). As a general rule of thumb specify a timestep that is around one tenth of the TUFLOW Classic timestep you would use. |
13.5.3.2 Compatible Graphics Cards
TUFLOW HPC’s GPU hardware module requires an NVIDIA CUDA enabled GPU card. A list of CUDA enabled GPUs can be found on the following website: https://developer.nvidia.com/cuda-gpus.
To check if your computer has an NVIDIA GPU and if it is CUDA enabled:
- Right click on the Windows desktop.
- If you see “NVIDIA Control Panel” or “NVIDIA Display” in the pop-up
dialogue, the computer has an NVIDIA GPU.
- Click on “NVIDIA Control Panel” or “NVIDIA Display” in the pop-up
dialogue.
- The GPU model should be displayed in the graphics card information.
- Check to see if the graphics card is listed on the following website: https://developer.nvidia.com/cuda-gpus.
The following screen images show the steps outlined above, this may vary slightly between NVIDIA GPU card models.
Figure 13.2: Accessing NVIDIA Control Panel from the Desktop
Figure 13.3: NVIDIA GPU Model
Figure 13.4: Check the Website for your NVIDIA Card
More information on the card can be found in the “System Information” section, which is accessed from the NVIDIA Control Panel. The system information contains more details on the following:
- The number of CUDA cores.
- Frequency of the graphics, processors and memory.
- Available memory including dedicated graphics and shared memory.
Figure 13.5: NVIDIA System Information
On the NVIDIA website, each CUDA enabled graphics card has a “Compute Capability” listed. For cards with a compute capability of 1.2 or less, only the single precision version of the GPU Module can be utilised. However, benchmarking has indicated that the double precision version, except in rare situations, is NOT required and that for nearly all GPU simulations, TUFLOW_iSP engine versions should suffice. Refer to Section 13.4.2.
Extensive GPU hardware benchmarking has been undertaken to assist users who are upgrading hardware for TUFLOW modelling, with numerous hardware options tested for their speed performance. The results are provided on the TUFLOW Wiki.
13.5.3.3 Updating NVIDIA Drivers
It is likely that the NVIDIA drivers will need to be updated periodically to the latest version as the drivers shipped with Windows Operating Systems are not always up-to-date. To update, open the NVIDIA Control Panel (by right clicking on the desktop and selecting NVIDIA Control Panel or NVIDIA Display). Once the control panel has loaded, select Help >> Updates from the menu items. For further guidance see the Updating NVIDIA Drivers Wiki Page.
If new drivers are available, please download and install these by following the prompts.
NOTE: Even if not prompted by the system, a restart is recommended to ensure the new drivers are correctly detected prior to running any simulations.
Figure 13.6: Accessing Driver Updates from the NVIDIA Control Panel
13.5.3.4 NVLink – Multi-GPU Performance (HPC Only)
NVLink is a connection (cable) for NVIDIA GPU devices providing peer-to-peer (direct) access between GPUs, rather than having to communicate via the CPU, giving faster communication. For example, data transfer rate for the RTX 2080 Ti is 32GB/s on the CPU PCIe and 100 GB/s via an NVLink. The TUFLOW 2020-01 and later releases automatically recognises and utilises any peer-to-peer (p2p) access between GPUs that is possible according to the hardware setup. Peer-to-peer access typically requires an NVLink connector between GPUs, or in some cases peer-to-peer access can occur via the PCI bus if all cards are placed into TCC driver mode.
Whilst testing thus far has not produced a huge jump in performance, it is expected greater gains in the future will arise as it is increasingly possible to connect to large numbers of GPU devices, especially Cloud-based instances. The user can choose to disable peer-to-peer GPU access by specifying the .tcf command:
GPU Peer to Peer Access
13.5.3.5 Troubleshooting
If you receive the following error when trying to run the TUFLOW GPU model:
TUFLOW GPU: Error: Non-CUDA Success Code returned
Please try the following steps:
- Check the compatibility of your GPU card and whether the latest
drivers are installed (see instructions in Section 13.5.3.2).
- Test with a user account that has administrator privileges as these
may be required for running computations on the GPU.
- If multiple monitors are running from the video card, try running with only a single monitor.
If the above steps fail to get the simulation to run, please email the NVIDIA system information (see Section 13.5) and TUFLOW log file (.tlf) to support@tuflow.com.
13.5.4 Running TUFLOW 1D Only Simulations
TUFLOW is set up to run as either a 2D model or as a 1D/2D model. It is, however, possible to run a TUFLOW 1D only model by including a small ‘dummy’ 2D domain within the model files. The 2D domain can be set up with the .tgc file and can be small, contain few cells of uniform material type and elevation values. The .tbc file will be set up so that the 2D domain does not receive any flows and therefore will not impact simulation runtimes. The 1D model can then be constructed within the .ecf and the simulation set up within the .tcf as per usual. An example model exists on the TUFLOW User Gitlab Page.
13.6 Using TUFLOW with Flood Modeller, SWMM, XP-SWMM, 12D or from SMS
TUFLOW supports the linking with third-party external 1D schemes such as Flood Modeller (Jacobs), SWMM (USA EPA SWMM Version 5), XP-SWMM (Autodesk), 12D (12D Solutions) as well as integration within the SMS GUI (Aquaveo). Flood Modeller (formerly known as ISIS), XP-SWMM and 12D executables access the TUFLOW hydraulic computational engine via the TUFLOW_LINK.dll. EPA SWMM (Version 5) is built into the standard TUFLOW release (2023-03-AF build or later). Flood Modeller, XP-SWMM, 12D and SMS are generally distributed with the latest supported version of TUFLOW at the time of release. It is possible to utilise different versions of TUFLOW with external 1D schemes using the processes described in the following sections.
13.6.1 Using TUFLOW with EPA SWMM
When running a TUFLOW - SWMM model, TUFLOW controls the execution of the simulation. SWMM input commands are specified within a SWMM Control File (.tscf). The SWMM control file is referenced in the TUFLOW Control File (.tcf) and the available commands listed in Appendix I.
The SWMM Control File can reference multiple SWMM input files (inp) potentially varying by scenario. These input files are merged giving preferences to later files allowing for the overriding (upgrading) of existing features. After the input files are merged, the final SWMM model will be placed in the TUFLOW output folder and will be executed from there. The report and output files generated by the SWMM engine will also be placed in the TUFLOW output folder.
13.6.2 Using TUFLOW with Flood Modeller
Flood Modeller is shipped with the latest release of TUFLOW available at the time of development. It is possible to set Flood Modeller up to use a different version of TUFLOW. Before changing the version of TUFLOW, firstly check the compatibility between Flood Modeller and TUFLOW versions in Figure 10.12. To utilise a newer, or different, version of TUFLOW with Flood Modeller, use the Flood Modeller interface to set the TUFLOW Engine File location to the path of the TUFLOW executable required in the Project Settings as shown in Figure 13.7.
Figure 13.7: Flood Modeller Settings
If running TUFLOW HPC utilising GPU hardware, it is important that the Flood Modeller folder contains copies of four TUFLOW files (aka kernels). The Flood Modeller folder is called the “bin” folder and is located (by default) in “C:\Program Files\Flood Modeller”. It is important that the kernel files in the Flood Modeller ‘bin’ folder match those of the TUFLOW version specified in the Flood Modeller project settings. For TUFLOW versions 2023-03-AA release or later, the names of the kernels have changed and are:
- hpcKernels_nSP.ptx
- hpcKernels_nDP.ptx
- qpcKernels_nSP.ptx
- qpcKernels_nDP.ptx
If you need to use a later release of TUFLOW or if you find that the link in a Flood Modeller-TUFLOW coupled model is failing with a ERROR 3999 message (and an error description of ‘ptx file version mismatch’ in the hpc.tlf), then browse to your TUFLOW engine folder, and copy the above four files and then paste them into your Flood Modeller bin folder (replacing the files there). If wanting to use the 2023-03-AA release or later, ensure the relevant kernels are in the Flood Modeller Bin folder and any other older TUFLOW .ptx files (ie, kernels_nSP.ptx, kernels_nDP.ptx) are removed.
It is also possible to run Flood Modeller-TUFLOW models using a batch file, which can be useful where different combinations of Flood Modeller and TUFLOW versions are required. This page provides instructions on how to set this up.
13.6.3 Using TUFLOW with 12D
12D is shipped and installed with the latest version of TUFLOW at the time of release. Although it is recommended to use the latest version of TUFLOW, it is possible to use other versions of TUFLOW by copying the TUFLOW engine files into the 12D installation folder, which is usually C:Files\12d\12dmodel\15.00_2d. It is possible to have multiple versions of 12D installed to allow multiple 12D-TUFLOW pairs to be installed.
There are two ways to run a TUFLOW model within 12D. The first is via the 2D Quick Analysis, which enables an efficient approach to using TUFLOW with a limited number of parameter options as shown in Figure 13.8.
Figure 13.8: 12D 2D Quick Setting
The second approach is to use the TCF editor allowing full TCF control, and String attribute editing which exposes all the TUFLOW functionality to a 12D user. The TCF editor is shown in Figure 13.9.
Figure 13.9: 12D TUFLOW Project Editor
13.6.4 Using TUFLOW with XP-SWMM
XP-SWMM is generally shipped with the latest release of TUFLOW available at the time of development. To utilise a new version of TUFLOW with XP-SWMM, all of the .dll and .ptx files described in the previous section need to be copied to the location where XP SWMM accesses them (usually in the same folder as the XP-SWMM .exe files). Alternatively, modifying the path and environment variable found within Advanced Settings allows the user to point to the location of the TUFLOW .dll and .ptx files. They do not access the TUFLOW.exe file, although there are no issues in copying this file as well. Note, it is always wise to keep copies of any old .dll/.ptx files in a separate folder.
13.6.5 Using TUFLOW with SMS
When running TUFLOW from SMS, SMS by default looks for a TUFLOW.exe in the installation folder. To change this to the location where you have placed the TUFLOW.exe and .dll/.ptx files, go to the Edit, Preferences, File Locations tab as shown in Figure 13.10.
Figure 13.10: SMS Settings
13.6.6 Optimising Startup and Run Times
Regardless of the method for initiating a TUFLOW simulation, a simulation start stats file (
From the 2018-03-AA release, a new output file is created named “
13.6.6.1 Improved pre-processing of 1D Model Inputs
For the 2020-01 release, reading and processing of 1D inputs has been significantly improved, particularly for large urban drainage models (>1,000 1D pipe network elements). For a tested model with 25,000 1D channels, the start-up was approximately 40 times faster with the 2020-01 release compared to the previous 2018-03 release changing the start-up time from nearly two hours to less than 3 minutes. No changes in model files is required to implement this improved pre-processing time.
13.6.6.2 Parallel Processing for SGS initialisation
With the default SGS Method C (the default if “
By default, all CPU threads will be used for final SGS elevation pre-processing unless the number of threads (-nt[thread count]) command line argument has been specified. For example, to run on 8 threads the command line argument “-nt8” would be used. There is no check for thread licensing used for pre-processing. If the number of threads specified in the command line argument exceeds the number of threads available, all threads are used.
At the end of the .tgc file, after all elevation datasets have been processed, an XF file is written if the XF Files command is set to on (default). The XF file is written to an “xf” folder, which sits in the same location as the .tgc. The XF file is then used for any subsequent simulations for optimised pre-processing performance. To avoid re-processing when changes are made to .tgc data other than elevation (e.g. active cells, materials, soils, etc.), the XF file is not written with the same filename as the .tgc. Instead, the .xf will be prefixed by “hpc” or “qdt” for HPC single grid and Quadtree simulations respectively and includes the nesting level and cell size. Any text set with the .tcf command XF Files Include in Filename is included.
When reading the pre-processed SGS XF file, a check is done on the final SGS elevations, if these are consistent then the XF file is used.
13.6.6.3 Optimising Multi-GPU Performance (HPC Only)
If a model is simulated across multiple GPU devices, one of the devices (usually the one with the most wet cells) will be controlling the speed of the simulation and the other devices will be underutilised. By default, TUFLOW HPC divides a model equally over multiple GPU devices. However, for real-world models, it is usual for the GPUs to have an inequitable amount of workload due to the number of active cells and number of wet cells, and this can change throughout the simulation as the model wets and dries.
From the TUFLOW 2020-01 release it is possible to distribute the workload unequally to the GPU devices. During a simulation the workload efficiency of each GPU is output to the console and to the .tlf file with a suggested distribution provided at the end of the simulation. A number of iterations may be required to fully optimise the distribution.
For example, a model simulated across four GPU devices reported at the end of the simulation in the .tlf file:
HPC Suggested workload balance HPC Device Split == 1.23, 0.74, 0.89, 1.40.
The command
Note: The benefit depends on the model, but if you have a significant variation in workload efficiencies between GPU devices this feature should provide a noticeable decrease in run times.
13.6.7 Auto Terminate (Simulation End) Options
TUFLOW Classic and HPC include an Auto-Terminate feature for stopping simulations after the flood peak has been experienced within the simulation. This can help project efficiencies by avoiding unnecessary model simulation time once the peak flood extent has been achieved.
The 2D cells that are monitored to trigger the auto-termination are controlled by specifying a value of 0 (exclude) or 1 (include) using the .tcf commands: Set Auto Terminate and Read GIS Auto Terminate.
For example, in the below, all cells are first set to be excluded for monitoring followed by the reading of a GIS layer to set cells individually.
At each Map Output Interval the monitored cells are compared against two criteria:
- The percentage of the wet cells that have become wet since the last
map output interval.
- The velocity-depth product at the current timestep compared to the tracked maximum.
For the percentage of cells that have become wet since the last interval, the maximum allowable value is controlled with the .tcf command:
If set to 0, then if any monitored cells have become wet since the last map output the simulation continues. If set to a value of 5, then up to 5% of monitored cells can become wet since the last map output while still triggering an auto-termination of the simulation.
For the velocity-depth tolerance, at each output interval the velocity-depth product is compared to the tracked maximum value. If the current dV product is within the specified tolerance Auto Terminate dV Value Tolerance the simulation is not terminated.
The total number of cells that are allowable within the specified range is controlled with Auto Terminate dV Cell Tolerance. If set to a value of 1, then up to 1% of monitored cells can be within the tolerance value without triggering an auto-termination of the simulation. The larger the Auto Terminate dV Value Tolerance the further the dV product needs to have dropped from the peak value.
The time that the auto-terminate feature commences can be controlled using the .tcf command Auto Terminate Start Time otherwise the Start Time is used.
Note, this option is only assessed at every Map Output Interval.
13.7 Reproducibility of Results
A key concern with hydraulic modelling is the replication of results across a range of hardware. The following sections provide some information to be aware of when looking at the replication of results.
13.7.1 TUFLOW Classic (CPU only)
The TUFLOW Classic engine is written in Fortran and compiled for Windows™ with Intel™’s Fortran compiler, leading to well optimised CPU code. The computational algorithm is implicit in nature and difficult to parallelise for multi-thread execution. As a result race conditions do not exist and a repeat run of a model with the same inputs, with the same executable on the same type of CPU, should yield bit-wise identical results (i.e. a subtraction of results should be exactly zero). If a user finds that repeat model runs (same executable, same CPU) do not yield identical results then please contact support@tuflow.com as this may indicate a memory access error or an un-initialised variable in the code.
A repeat model run with the same executable but on a different type of CPU (e.g. Intel Xeon vs i9, or AMD Epyc vs Ryzen, or Intel vs AMD) may produce very slight differences in results due to the CPUs having different instruction set extensions that may or may not be utilised. Such differences are typically less than 1 mm water surface elevation but, in some locations, the differences can become accentuated due to changes in overtopping or an operational structure threshold.
Even though the source code for TUFLOW Classic is currently seeing little development, as we update Intel Fortran compilers, different releases of TUFLOW can be expected to show minor differences even on the same CPU.
13.7.2 TUFLOW HPC (incl. Quadtree) on CPU
The TUFLOW HPC engine (incl Quadtree) is written in C++ with NVIDIA™ CUDA GPU kernels. The kernels have been carefully written so that they can be compiled for both GPU and CPU execution. When compiled for CPU execution, the Microsoft Visual Studio compiler is used and the code is built into the dynamic link libraries (DLLs) in the executable bundle. Similar to TUFLOW Classic, repeat model runs of the same model (same TUFLOW executable, same CPU type) should produce bit-wise identical results. Again, if differences are observed please contact support@tuflow.com as this may indicate an issue that needs to be addressed.
As the TUFLOW HPC and Quadtree solvers are fundamentally different to TUFLOW Classic, there will always be minor differences between solutions from the different schemes even when using the same turbulence model and without sub-grid sampling (SGS). As TUFLOW HPC and Quadtree both default to using the Wu turbulence model while Classic can only use the Smagorinksy model, the differences will be more significant when run with the default settings.
13.7.3 TUFLOW HPC (incl. Quadtree) on GPU
When the CUDA kernels are compiled for GPU execution, a GPU agnostic intermediate code file (ptx) is produced with the final compilation of that being done on a just-in-time basis by the GPU driver on the machine that the executable is running on. The results may now depend not just on the version of TUFLOW executable used and type of CPU (since the pre-processing of the model input is still performed on CPU), but also on the type of GPU and the version of the GPU driver installed. GPUs have thousands of cores that work in parallel. However, the kernels have been carefully written to avoid race conditions, and the adaptive timestep control avoids relying on variables summed with atomic additions of floating point data. Therefore, provided these factors (version of TUFLOW, CPU type, GPU type, GPU driver version) are kept constant, a repeat model run will yield bit-wise reproducible results.
Also note that the GPU cores use different hardware for the floating point operations and the GPU compiler may re-optimise the sequence of instructions for complex lines of code compared to the CPU compiler. Therefore, even when using the same version of TUFLOW HPC, very small differences in the solution can be seen when the model is run on GPU compared to running on CPU. These differences are typically less than 1 mm water surface elevation but again, in some locations, the differences can become accentuated due to changes in overtopping or an operational structure threshold.
13.7.4 TUFLOW HPC (incl. Quadtree) on multiple GPUs / CPU threads
TUFLOW HPC (incl. Quadtree) support running a model across multiple GPU devices in one computer. In this case the model is decomposed into subdomains, and model data are exchanged and sychronised at the domain boundaries as required. Care has been taken to ensure that results when run on multiple GPUs (of all the same GPU type) are bit-wise identical to when run on a single GPU. If differences are observed between results from a run on multiple devices vs a single device please contact support@tuflow.com.
Likewise when running on multiple CPU threads (default is four unless
specified), the model results should be bit-wise identical to when
running on a single thread (CPU Threads
