CompareGlobal

CompareGlobal Tool

If you are making changes to NairnMPM (or OSParticulas), an important development process is that the changes do not cause unintended changes to other calculation options. One way to check for problems is

1. Run a simulation with original code writing various interesting results to a global results file.
2. Run the same simulation with revised code saving same results to a new global results file.
3. Compare the two global results files.

This process does not guarantee identical results, but is an excellent check. It is like a parity check in other computer methods. If the averages differ, the results are certainly different. If the global averages are the same, the two codes are likely generating identical results.

One way to compare global results files (which are plain-text files) is to do a simple text compare. The problem with this approach is that negligible changes (e.g., only in 6^th significant digit) will show up as a text difference. Such small change might be common if the revisions altered the order of certain calculations of the the comparison files were run in parallel. It is difficult to interpret such minor text differences thoroughly without inspecting each text difference (and every number might differ slightly). The CompareGlobal tool avoids limitation of text comparisons by comparing the global files numerically. In brief, it will

1. Open the original global results file
2. Read the new global results file
3. Locate all columns of global results that the two files have in common and calculate some numeric metrics that evaluate their differences
4. Report all results in a table.

The comparison table will have the following columns

• Column Name - name of the global result being compared. The first result compared is always "Time". If this result does not exactly match, it means the global results in the two files were recorded and different time intervals. Even if the two codes are identical, the results certainly be different. You should always run original and new simulations using the same global archiving time.
• Max SM - maximum value of the symmetric mean (SM), which for one row of data is defined by:
             [math]\displaystyle{ {\rm SM}_i = {|X_i| + |Y_i|\over 2} }[/math]
  where [math]\displaystyle{ X_i }[/math] and [math]\displaystyle{ Y_i }[/math] are values in row [math]\displaystyle{ i }[/math] of the original and comparison files, respectively. If this maximum is very low (i.e., all results close to zero), comparisons of that result may not be meaningful.
• MAE - mean absolute error defined by:
             [math]\displaystyle{ {\rm MAE} = \frac{1}{N} \sum_{i=1}^N |X_i - Y_i| }[/math]
  where [math]\displaystyle{ N }[/math] is number of rows of data that were compared. This mean should be much less than maximum SM (ideally would be zero).
• Max AE - the maximum absolute error found, which is the maximum summand found when evaluating MAE.
• SMAPE (%) - the symmetric mean absolute proportional error defined by:
             [math]\displaystyle{ {\rm SMAPE\ (\%)} = \frac{100}{N} \sum_{i=1}^N \frac{2|X_i - Y_i|}{|X_i| + |Y_i|} }[/math]
  This normalized error is reported in percent difference. This mean should be very low (ideally would be zero). This metric may be unreliable for global results that are close to zero (i.e., if the "Max SM" is small).
• Max APE (%) - maximum absolute proportional error, which is the maximum summand found when evaluating SMAPE.

Compiling CompareGlobal

The first step is to compile the CompareGlobal command-line tool. On MacOS X or Windows, you can compile CompareGlobal using its target in the XCode or Visual Studio projects found in the nairn-mpm-fea/Common/Projects folder. On any other system (or alternatively in MacOS X), you can compile by changing directory to the nairn-mpm-fea/NairnMPM/tools directory and use the command

make CompareGlobal SYSTEM=(system)

where (system) defines the system being used. This option mainly picks path to the c++ compiler on your computer. For unsupported systems, you can manually set the path to that compiler using

make CompareGlobal CC=/usr/local/gcc48/bin/g++

Other options that can be passed to the make command are

• CFLAGS - options passed to compiler (default: -c -O3)
• LFLAGS - options during linking step (default: none)

Other make functions are:

make
make clean
make install

A make by itself will compile all tools (and not just CompareGlobal). The clean option will delete compiled objects and executables for all tools. The install option will copy all compiled tool executables to an install folder. The default install location is ~/bin but that can be changing by adding ioutput=(new folder) to the make command.

Using CompareGlobal MPM

You run the CompareGlobal command line tool on a command line with

CompareGlobal [-Hh] [-m pow] -o originalpath newpath ...

CompareGlobal compares the original global results file at originalpath to one (or more) global results files specified in command arguments newpath .... The command options are:

-o originalpath

Path to the original global results file. This option is required.

-m pow

Ignore values with absolute value of symmetric mean less than 1.e(-pow) (default is to not ignore anything)

-H or -h

Print version number and brief help on this tool and then exit.