CompareGlobal

CompareGlobal Tool

If you are making changes to NairnMPM (or OSParticulas), an important development process is that the changes to not change other calculations options. One way to check for problems is

1. Run a simulation with original code writing various interesting results to a global results file.
2. Run the same simulation with revised code saving same results to a new global results file.
3. Compare the too files.

This process does not guarantee identical results, but is an excellent check. It is like a parity check in other computer methods. If the averages differ, the results are certainly different. If the global averages are the same, the two codes are likely generating identical results.

One was to compare the files is a simple test compare. The problem with this approach is that revisions to the code that change order of calculations or comparing two results that were run in parallel, the two calculations might nominally be identical, but the text result might different only in insignificant digits. It is difficult to interpret such minor text differences thoroughly. The CompareGlobal tool avoids limitation of text comparisons by comparing the global files numerically. In brief, it will

1. Open the original global results file
2. Read the new global results file
3. Locate all columns of global results that the two files have in common and calculate some numeric metrics that evaluate their differences
4. Report all results in a table.

The comparison table will have the following columns

• Column Name - name of the global result being compared. The first column is always "Time". If this column does not exactly match, it means the global results in the two files were record and different time intervals. If the two codes are identical, the results will likely be different. You should always run original and new simulations using the same global archiving time.
• Max SM - maximum value of the symmetric mean (SM), which for in row of data is defined by:
             [math]\displaystyle{ {\rm SM}_i = {|X_i| + |Y_i|\over 2} }[/math]
  where [math]\displaystyle{ X_i }[/math] and [math]\displaystyle{ Y_i }[/math] are values in row [math]\displaystyle{ i }[/math] of the original and comparison files, respectively. If this maximum is very low (i.e., all results close to zero), comparisons of that result may not be meaningful.
• MAE - mean absolute error defined by:
             [math]\displaystyle{ {\rm MAE} = {1\over N} \sum_{i=1}^N |X_i - Y_i| }[/math]
  where [math]\displaystyle{ N }[/math] is number of rows of data that was compared. This mean should be much less than maximum SM (ideally would be zero).
• Max AE - the maximum absolute error found.
• SMAPE - the symmetric mean absolute proportional error define by:
             [math]\displaystyle{ {\rm SMAPE\ (%)} = {1\over N} \sum_{i=1}^N {2|X_i - Y_i|\over |X_i| + |Y_i|} }[/math]
  This normalized error is reported in percent different. This mean should be very low (ideally would be zero). This metric may be unreliable for global results that are close to zero (or if the "Max SM" is small).
• Max APE (%) - maximum absolute proportional error, which is the maximum summand found when evaluating SMAPE.

Compiling CompareGlobal

The first step is to compile the CompareGlobal command-line tool. On MacOS X or Windows, you can compile CompareGlobal using its target in the XCode or Visual Studio projects found in the nairn-mpm-fea/Common/Projects folder. On any other system (or alternatively in MacOS X), you can compile by changing directory to the nairn-mpm-fea/NairnMPM/tools directory and use the command

make CompareGlobal SYSTEM=(system)

where (system) defines the system being used. This option mainly picks path to the c++ compiler on your computer. For unsupported systems, you can manually set the path to that compiler using

make ExtractMPM CC=/usr/local/gcc48/bin/g++

Other options that can be passed to the make command are

• CFLAGS - options passed to compiler (default: -c -O3)
• LFLAGS - options during linking step (default: none)

Other make functions are:

make
make clean
make install

A make by itself will compile all tools (and not just CompareGlobal). The clean option will delete compiled objects and executables for all tools. The install option will copy all compiled tool executables to an install folder. The default install location is ~/bin but that can be changing by adding ioutput=(new folder) to the make command.

Using CompareGlobal MPM

You run the CompareGlobal command line tool on a command line with

CompareGlobal [-h?] -o originalpath newpath ...

CompareGlobal compares the original global results file at originalpath to one (or more) global results files specified in command arguments newpath .... The command options are:

-o originalpath

Path to the original global results file. This option is required.

-h or -?

Print brief help on this tool and exit.