Currently Loaded Modulefiles:
  1) xalt/2.3.8-nogpu   2) cuda/10.0.130
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
mpirun (Open MPI) 3.1.2

Report bugs to http://www.open-mpi.org/community/help/
HOSTNAME=dgx05
Tue Oct 23 12:52:10 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
| N/A   37C    P0    44W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   38C    P0    42W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
| N/A   41C    P0    45W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
| N/A   38C    P0    44W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |
| N/A   39C    P0    43W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
| N/A   40C    P0    43W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |
| N/A   41C    P0    44W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
| N/A   38C    P0    42W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
 ****** running HPCG - binary=xhpcg-3.1_gcc_485_cuda-10.0.130_ompi-3.1.0_sm_35_sm_50_sm_60_sm_70_sm_75_ver_10_9_18 on 2 NODES, 4 GPUs ***************************** 
Currently Loaded Modulefiles:
  1) cuda/10.0.130
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'dgx06', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
[dgx05:47263] 11 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix
[dgx05:47263] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

start of application (1 OMP threads)...
2018-10-23 12:52:17.279

Problem setup...
Setup time: 0.341873 sec 

GPU: 'Tesla V100-SXM2-16GB' 
Memory use: 8548 MB / 16130 MB
2x2x1 process grid
256x256x256 local domain

Reference SpMV+MG...

Reference CG...
Initial Residual: 9.233453e+03 Max_err: 1.000000e+00 tot_err: 8.192000e+03
REF  Iter = 1 Scaled Residual: 1.894800e-01 Max error: 1.000000e+00 tot_error: 9.798293e-01 
REF  Iter = 2 Scaled Residual: 1.045095e-01 Max error: 1.000000e+00 tot_error: 9.611759e-01 
REF  Iter = 3 Scaled Residual: 7.221370e-02 Max error: 1.000000e+00 tot_error: 9.428239e-01 
REF  Iter = 4 Scaled Residual: 5.516883e-02 Max error: 1.000000e+00 tot_error: 9.245937e-01 
REF  Iter = 5 Scaled Residual: 4.458568e-02 Max error: 1.000000e+00 tot_error: 9.064397e-01 
REF  Iter = 6 Scaled Residual: 3.737752e-02 Max error: 1.000000e+00 tot_error: 8.883560e-01 
REF  Iter = 7 Scaled Residual: 3.216713e-02 Max error: 1.000000e+00 tot_error: 8.703404e-01 
REF  Iter = 8 Scaled Residual: 2.824598e-02 Max error: 1.000000e+00 tot_error: 8.523981e-01 
REF  Iter = 9 Scaled Residual: 2.520266e-02 Max error: 1.000000e+00 tot_error: 8.345331e-01 
REF  Iter = 10 Scaled Residual: 2.277302e-02 Max error: 1.000000e+00 tot_error: 8.167494e-01 
REF  Iter = 11 Scaled Residual: 2.077845e-02 Max error: 1.000000e+00 tot_error: 7.990462e-01 
REF  Iter = 12 Scaled Residual: 1.909602e-02 Max error: 1.000000e+00 tot_error: 7.814205e-01 
REF  Iter = 13 Scaled Residual: 1.764213e-02 Max error: 1.000000e+00 tot_error: 7.638669e-01 
REF  Iter = 14 Scaled Residual: 1.636760e-02 Max error: 1.000000e+00 tot_error: 7.463798e-01 
REF  Iter = 15 Scaled Residual: 1.524558e-02 Max error: 1.000000e+00 tot_error: 7.289577e-01 
REF  Iter = 16 Scaled Residual: 1.425938e-02 Max error: 9.999999e-01 tot_error: 7.116014e-01 
REF  Iter = 17 Scaled Residual: 1.339670e-02 Max error: 9.999999e-01 tot_error: 6.943144e-01 
REF  Iter = 18 Scaled Residual: 1.264377e-02 Max error: 9.999999e-01 tot_error: 6.771014e-01 
REF  Iter = 19 Scaled Residual: 1.198255e-02 Max error: 9.999998e-01 tot_error: 6.599660e-01 
REF  Iter = 20 Scaled Residual: 1.139395e-02 Max error: 9.999996e-01 tot_error: 6.429077e-01 
REF  Iter = 21 Scaled Residual: 1.085899e-02 Max error: 9.999994e-01 tot_error: 6.259241e-01 
REF  Iter = 22 Scaled Residual: 1.036238e-02 Max error: 9.999990e-01 tot_error: 6.090083e-01 
REF  Iter = 23 Scaled Residual: 9.897296e-03 Max error: 9.999983e-01 tot_error: 5.921534e-01 
REF  Iter = 24 Scaled Residual: 9.464289e-03 Max error: 9.999971e-01 tot_error: 5.753541e-01 
REF  Iter = 25 Scaled Residual: 9.067186e-03 Max error: 9.999948e-01 tot_error: 5.586088e-01 
REF  Iter = 26 Scaled Residual: 8.711162e-03 Max error: 9.999906e-01 tot_error: 5.419175e-01 
REF  Iter = 27 Scaled Residual: 8.397579e-03 Max error: 9.999825e-01 tot_error: 5.252841e-01 
REF  Iter = 28 Scaled Residual: 8.122298e-03 Max error: 9.999669e-01 tot_error: 5.087091e-01 
REF  Iter = 29 Scaled Residual: 7.876354e-03 Max error: 9.999357e-01 tot_error: 4.921916e-01 
REF  Iter = 30 Scaled Residual: 7.647565e-03 Max error: 9.998723e-01 tot_error: 4.757236e-01 
REF  Iter = 31 Scaled Residual: 7.425040e-03 Max error: 9.997414e-01 tot_error: 4.592907e-01 
REF  Iter = 32 Scaled Residual: 7.207766e-03 Max error: 9.994679e-01 tot_error: 4.428638e-01 
REF  Iter = 33 Scaled Residual: 7.008807e-03 Max error: 9.988951e-01 tot_error: 4.263839e-01 
REF  Iter = 34 Scaled Residual: 6.865640e-03 Max error: 9.976993e-01 tot_error: 4.096947e-01 
REF  Iter = 35 Scaled Residual: 6.852925e-03 Max error: 9.952148e-01 tot_error: 3.924038e-01 
REF  Iter = 36 Scaled Residual: 7.081007e-03 Max error: 9.900459e-01 tot_error: 3.736333e-01 
REF  Iter = 37 Scaled Residual: 7.611184e-03 Max error: 9.792538e-01 tot_error: 3.518694e-01 
REF  Iter = 38 Scaled Residual: 8.178412e-03 Max error: 9.575477e-01 tot_error: 3.258551e-01 
REF  Iter = 39 Scaled Residual: 7.978768e-03 Max error: 9.203108e-01 tot_error: 2.979929e-01 
REF  Iter = 40 Scaled Residual: 6.697031e-03 Max error: 8.720683e-01 tot_error: 2.751011e-01 
REF  Iter = 41 Scaled Residual: 5.505983e-03 Max error: 8.206773e-01 tot_error: 2.594283e-01 
REF  Iter = 42 Scaled Residual: 5.331168e-03 Max error: 7.613824e-01 tot_error: 2.465493e-01 
REF  Iter = 43 Scaled Residual: 5.444615e-03 Max error: 7.131979e-01 tot_error: 2.327185e-01 
REF  Iter = 44 Scaled Residual: 4.826719e-03 Max error: 6.947780e-01 tot_error: 2.198636e-01 
REF  Iter = 45 Scaled Residual: 4.514340e-03 Max error: 6.593172e-01 tot_error: 2.093133e-01 
REF  Iter = 46 Scaled Residual: 4.576498e-03 Max error: 6.183717e-01 tot_error: 1.985335e-01 
REF  Iter = 47 Scaled Residual: 4.093291e-03 Max error: 5.994736e-01 tot_error: 1.885031e-01 
REF  Iter = 48 Scaled Residual: 4.046602e-03 Max error: 5.741396e-01 tot_error: 1.795383e-01 
REF  Iter = 49 Scaled Residual: 3.873832e-03 Max error: 5.502849e-01 tot_error: 1.701490e-01 
REF  Iter = 50 Scaled Residual: 3.538446e-03 Max error: 5.192932e-01 tot_error: 1.620299e-01 

Optimization...
Optimization time: 1.291420e-01 sec 

Validation...

Optimized CG Setup...
Initial Residual: 9.233453e+03 Max_err: 1.000000e+00 tot_err: 8.192000e+03
Iteration = 1 Scaled Residual: 2.202111e-01 Max error: 1.000000e+00 tot_error: 9.797574e-01 
Iteration = 2 Scaled Residual: 1.190703e-01 Max error: 1.000000e+00 tot_error: 9.607646e-01 
Iteration = 3 Scaled Residual: 8.101890e-02 Max error: 1.000000e+00 tot_error: 9.421919e-01 
Iteration = 4 Scaled Residual: 6.137596e-02 Max error: 1.000000e+00 tot_error: 9.238130e-01 
Iteration = 5 Scaled Residual: 4.944777e-02 Max error: 1.000000e+00 tot_error: 9.055212e-01 
Iteration = 6 Scaled Residual: 4.139154e-02 Max error: 1.000000e+00 tot_error: 8.873053e-01 
Iteration = 7 Scaled Residual: 3.559727e-02 Max error: 1.000000e+00 tot_error: 8.691492e-01 
Iteration = 8 Scaled Residual: 3.123255e-02 Max error: 1.000000e+00 tot_error: 8.510562e-01 
Iteration = 9 Scaled Residual: 2.781345e-02 Max error: 1.000000e+00 tot_error: 8.330266e-01 
Iteration = 10 Scaled Residual: 2.506648e-02 Max error: 1.000000e+00 tot_error: 8.150665e-01 
Iteration = 11 Scaled Residual: 2.281087e-02 Max error: 1.000000e+00 tot_error: 7.971730e-01 
Iteration = 12 Scaled Residual: 2.092110e-02 Max error: 1.000000e+00 tot_error: 7.793514e-01 
Iteration = 13 Scaled Residual: 1.931886e-02 Max error: 1.000000e+00 tot_error: 7.616005e-01 
Iteration = 14 Scaled Residual: 1.794147e-02 Max error: 1.000000e+00 tot_error: 7.439218e-01 
Iteration = 15 Scaled Residual: 1.674320e-02 Max error: 1.000000e+00 tot_error: 7.263155e-01 
Iteration = 16 Scaled Residual: 1.569296e-02 Max error: 1.000000e+00 tot_error: 7.087841e-01 
Iteration = 17 Scaled Residual: 1.476470e-02 Max error: 1.000000e+00 tot_error: 6.913244e-01 
Iteration = 18 Scaled Residual: 1.393921e-02 Max error: 9.999999e-01 tot_error: 6.739393e-01 
Iteration = 19 Scaled Residual: 1.320106e-02 Max error: 9.999998e-01 tot_error: 6.566267e-01 
Iteration = 20 Scaled Residual: 1.253703e-02 Max error: 9.999996e-01 tot_error: 6.393875e-01 
Iteration = 21 Scaled Residual: 1.193777e-02 Max error: 9.999991e-01 tot_error: 6.222198e-01 
Iteration = 22 Scaled Residual: 1.139461e-02 Max error: 9.999980e-01 tot_error: 6.051258e-01 
Iteration = 23 Scaled Residual: 1.090068e-02 Max error: 9.999957e-01 tot_error: 5.881001e-01 
Iteration = 24 Scaled Residual: 1.044936e-02 Max error: 9.999909e-01 tot_error: 5.711453e-01 
Iteration = 25 Scaled Residual: 1.003422e-02 Max error: 9.999812e-01 tot_error: 5.542559e-01 
Iteration = 26 Scaled Residual: 9.650262e-03 Max error: 9.999619e-01 tot_error: 5.374316e-01 
Iteration = 27 Scaled Residual: 9.292903e-03 Max error: 9.999239e-01 tot_error: 5.206655e-01 
Iteration = 28 Scaled Residual: 8.957010e-03 Max error: 9.998511e-01 tot_error: 5.039548e-01 
Iteration = 29 Scaled Residual: 8.640192e-03 Max error: 9.997136e-01 tot_error: 4.872853e-01 
Iteration = 30 Scaled Residual: 8.340557e-03 Max error: 9.994594e-01 tot_error: 4.706459e-01 
Iteration = 31 Scaled Residual: 8.060893e-03 Max error: 9.989975e-01 tot_error: 4.540023e-01 
Iteration = 32 Scaled Residual: 7.810202e-03 Max error: 9.981725e-01 tot_error: 4.372997e-01 
Iteration = 33 Scaled Residual: 7.606986e-03 Max error: 9.967219e-01 tot_error: 4.204175e-01 
Iteration = 34 Scaled Residual: 7.482241e-03 Max error: 9.942005e-01 tot_error: 4.031334e-01 
Iteration = 35 Scaled Residual: 7.481233e-03 Max error: 9.898461e-01 tot_error: 3.850423e-01 
Iteration = 36 Scaled Residual: 7.645395e-03 Max error: 9.823368e-01 tot_error: 3.655163e-01 
Iteration = 37 Scaled Residual: 7.969409e-03 Max error: 9.694391e-01 tot_error: 3.437695e-01 
Iteration = 38 Scaled Residual: 8.304470e-03 Max error: 9.478902e-01 tot_error: 3.194099e-01 
Iteration = 39 Scaled Residual: 8.288599e-03 Max error: 9.147664e-01 tot_error: 2.936815e-01 
Iteration = 40 Scaled Residual: 7.574667e-03 Max error: 8.711815e-01 tot_error: 2.701863e-01 
Iteration = 41 Scaled Residual: 6.374398e-03 Max error: 8.232633e-01 tot_error: 2.522020e-01 
Iteration = 42 Scaled Residual: 5.420762e-03 Max error: 7.749511e-01 tot_error: 2.392787e-01 
Iteration = 43 Scaled Residual: 5.218150e-03 Max error: 7.221808e-01 tot_error: 2.283125e-01 
Iteration = 44 Scaled Residual: 5.458080e-03 Max error: 6.785474e-01 tot_error: 2.164807e-01 
Iteration = 45 Scaled Residual: 5.162536e-03 Max error: 6.613720e-01 tot_error: 2.041660e-01 
Iteration = 46 Scaled Residual: 4.490808e-03 Max error: 6.303717e-01 tot_error: 1.939758e-01 
Iteration = 47 Scaled Residual: 4.423855e-03 Max error: 5.903372e-01 tot_error: 1.849524e-01 
Iteration = 48 Scaled Residual: 4.423771e-03 Max error: 5.740467e-01 tot_error: 1.752061e-01 
Iteration = 49 Scaled Residual: 3.913967e-03 Max error: 5.421331e-01 tot_error: 1.663600e-01 
Iteration = 50 Scaled Residual: 3.897876e-03 Max error: 5.294938e-01 tot_error: 1.583122e-01 
Iteration = 51 Scaled Residual: 3.741572e-03 Max error: 5.010993e-01 tot_error: 1.497186e-01 
Iteration = 52 Scaled Residual: 3.397221e-03 Max error: 4.789262e-01 tot_error: 1.422584e-01 

Starting Benchmarking Phase... 
Performing 29 CG sets	 expected time:   60.0 seconds	 expected Perf:     576.7 GF (144.2 GF_per)
2018-10-23 12:56:53.750
progress = 3.5% 	    2.1 /   60.0 sec elapsed 	   57.9 sec remain 	   577.019 GF 	 144.255 GF_per
progress = 7.1% 	    4.2 /   60.0 sec elapsed 	   55.8 sec remain 	   576.970 GF 	 144.242 GF_per
progress = 10.6% 	    6.4 /   60.0 sec elapsed 	   53.6 sec remain 	   576.956 GF 	 144.239 GF_per
progress = 14.2% 	    8.5 /   60.0 sec elapsed 	   51.5 sec remain 	   576.919 GF 	 144.230 GF_per
progress = 17.7% 	   10.6 /   60.0 sec elapsed 	   49.4 sec remain 	   576.955 GF 	 144.239 GF_per
progress = 21.2% 	   12.7 /   60.0 sec elapsed 	   47.3 sec remain 	   576.964 GF 	 144.241 GF_per
progress = 24.8% 	   14.9 /   60.0 sec elapsed 	   45.1 sec remain 	   576.952 GF 	 144.238 GF_per
progress = 28.3% 	   17.0 /   60.0 sec elapsed 	   43.0 sec remain 	   576.966 GF 	 144.241 GF_per
progress = 31.8% 	   19.1 /   60.0 sec elapsed 	   40.9 sec remain 	   576.980 GF 	 144.245 GF_per
progress = 35.4% 	   21.2 /   60.0 sec elapsed 	   38.8 sec remain 	   576.985 GF 	 144.246 GF_per
progress = 38.9% 	   23.4 /   60.0 sec elapsed 	   36.6 sec remain 	   576.988 GF 	 144.247 GF_per
progress = 42.5% 	   25.5 /   60.0 sec elapsed 	   34.5 sec remain 	   576.982 GF 	 144.246 GF_per
progress = 46.0% 	   27.6 /   60.0 sec elapsed 	   32.4 sec remain 	   576.974 GF 	 144.244 GF_per
progress = 49.5% 	   29.7 /   60.0 sec elapsed 	   30.3 sec remain 	   576.962 GF 	 144.241 GF_per
progress = 53.1% 	   31.8 /   60.0 sec elapsed 	   28.2 sec remain 	   576.958 GF 	 144.239 GF_per
progress = 56.6% 	   34.0 /   60.0 sec elapsed 	   26.0 sec remain 	   576.959 GF 	 144.240 GF_per
progress = 60.2% 	   36.1 /   60.0 sec elapsed 	   23.9 sec remain 	   576.954 GF 	 144.238 GF_per
progress = 63.7% 	   38.2 /   60.0 sec elapsed 	   21.8 sec remain 	   576.948 GF 	 144.237 GF_per
progress = 67.2% 	   40.3 /   60.0 sec elapsed 	   19.7 sec remain 	   576.948 GF 	 144.237 GF_per
progress = 70.8% 	   42.5 /   60.0 sec elapsed 	   17.5 sec remain 	   576.941 GF 	 144.235 GF_per
progress = 74.3% 	   44.6 /   60.0 sec elapsed 	   15.4 sec remain 	   576.938 GF 	 144.235 GF_per
progress = 77.9% 	   46.7 /   60.0 sec elapsed 	   13.3 sec remain 	   576.932 GF 	 144.233 GF_per
progress = 81.4% 	   48.8 /   60.0 sec elapsed 	   11.2 sec remain 	   576.932 GF 	 144.233 GF_per
progress = 84.9% 	   51.0 /   60.0 sec elapsed 	    9.0 sec remain 	   576.930 GF 	 144.233 GF_per
progress = 88.5% 	   53.1 /   60.0 sec elapsed 	    6.9 sec remain 	   576.936 GF 	 144.234 GF_per
progress = 92.0% 	   55.2 /   60.0 sec elapsed 	    4.8 sec remain 	   576.937 GF 	 144.234 GF_per
progress = 95.5% 	   57.3 /   60.0 sec elapsed 	    2.7 sec remain 	   576.937 GF 	 144.234 GF_per
progress = 99.1% 	   59.5 /   60.0 sec elapsed 	    0.5 sec remain 	   576.926 GF 	 144.231 GF_per

Completed Benchmarking Phase... elapsed time:   61.6 seconds 
2018-10-23 12:57:55.345

Number of CG sets:	29 
Iterations per set:	52 
scaled res mean:	3.397221e-03 
scaled res variance:	0.000000e+00 

Total Time: 6.157399e+01 sec 
Setup        Overhead: 1.58%
Optimization Overhead: 0.60%
Convergence  Overhead: 3.85%

2x2x1 process grid
256x256x256 local domain
SpMV  =  485.6 GF (3058.4 GB/s Effective)  121.4 GF_per ( 764.6 GB/s Effective)
SymGS =  665.5 GF (5136.6 GB/s Effective)  166.4 GF_per (1284.1 GB/s Effective)
total =  613.3 GF (4651.1 GB/s Effective)  153.3 GF_per (1162.8 GB/s Effective)
final =  576.9 GF (4375.1 GB/s Effective)  144.2 GF_per (1093.8 GB/s Effective)

end of application...
2018-10-23 12:57:55.464
 ****** running HPCG -  binary=xhpcg-3.1_gcc_485_cuda-10.0.130_ompi-3.1.0_sm_35_sm_50_sm_60_sm_70_sm_75_ver_10_9_18 on 2 NODES, 8 GPUs ***************************** 
Currently Loaded Modulefiles:
  1) cuda/10.0.130
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'dgx06', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
[dgx05:47687] 23 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix
[dgx05:47687] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

start of application (1 OMP threads)...
2018-10-23 12:58:04.026

Problem setup...
Setup time: 0.577075 sec 

GPU: 'Tesla V100-SXM2-16GB' 
Memory use: 8580 MB / 16130 MB
2x2x2 process grid
256x256x256 local domain

Reference SpMV+MG...

Reference CG...
Initial Residual: 1.130436e+04 Max_err: 1.000000e+00 tot_err: 1.158524e+04
REF  Iter = 1 Scaled Residual: 1.866786e-01 Max error: 1.000000e+00 tot_error: 9.849293e-01 
REF  Iter = 2 Scaled Residual: 1.025624e-01 Max error: 1.000000e+00 tot_error: 9.709075e-01 
REF  Iter = 3 Scaled Residual: 7.081692e-02 Max error: 1.000000e+00 tot_error: 9.571438e-01 
REF  Iter = 4 Scaled Residual: 5.410908e-02 Max error: 1.000000e+00 tot_error: 9.434838e-01 
REF  Iter = 5 Scaled Residual: 4.375764e-02 Max error: 1.000000e+00 tot_error: 9.298920e-01 
REF  Iter = 6 Scaled Residual: 3.671068e-02 Max error: 1.000000e+00 tot_error: 9.163611e-01 
REF  Iter = 7 Scaled Residual: 3.161214e-02 Max error: 1.000000e+00 tot_error: 9.028886e-01 
REF  Iter = 8 Scaled Residual: 2.776606e-02 Max error: 1.000000e+00 tot_error: 8.894770e-01 
REF  Iter = 9 Scaled Residual: 2.477195e-02 Max error: 1.000000e+00 tot_error: 8.761282e-01 
REF  Iter = 10 Scaled Residual: 2.237653e-02 Max error: 1.000000e+00 tot_error: 8.628447e-01 
REF  Iter = 11 Scaled Residual: 2.041054e-02 Max error: 1.000000e+00 tot_error: 8.496263e-01 
REF  Iter = 12 Scaled Residual: 1.875762e-02 Max error: 1.000000e+00 tot_error: 8.364720e-01 
REF  Iter = 13 Scaled Residual: 1.733753e-02 Max error: 1.000000e+00 tot_error: 8.233799e-01 
REF  Iter = 14 Scaled Residual: 1.609948e-02 Max error: 1.000000e+00 tot_error: 8.103479e-01 
REF  Iter = 15 Scaled Residual: 1.501260e-02 Max error: 1.000000e+00 tot_error: 7.973761e-01 
REF  Iter = 16 Scaled Residual: 1.405666e-02 Max error: 1.000000e+00 tot_error: 7.844660e-01 
REF  Iter = 17 Scaled Residual: 1.321713e-02 Max error: 1.000000e+00 tot_error: 7.716205e-01 
REF  Iter = 18 Scaled Residual: 1.248015e-02 Max error: 1.000000e+00 tot_error: 7.588430e-01 
REF  Iter = 19 Scaled Residual: 1.182996e-02 Max error: 1.000000e+00 tot_error: 7.461364e-01 
REF  Iter = 20 Scaled Residual: 1.125051e-02 Max error: 1.000000e+00 tot_error: 7.335016e-01 
REF  Iter = 21 Scaled Residual: 1.072619e-02 Max error: 1.000000e+00 tot_error: 7.209387e-01 
REF  Iter = 22 Scaled Residual: 1.024404e-02 Max error: 1.000000e+00 tot_error: 7.084458e-01 
REF  Iter = 23 Scaled Residual: 9.797113e-03 Max error: 1.000000e+00 tot_error: 6.960213e-01 
REF  Iter = 24 Scaled Residual: 9.383760e-03 Max error: 1.000000e+00 tot_error: 6.836645e-01 
REF  Iter = 25 Scaled Residual: 9.005280e-03 Max error: 1.000000e+00 tot_error: 6.713770e-01 
REF  Iter = 26 Scaled Residual: 8.664336e-03 Max error: 1.000000e+00 tot_error: 6.591612e-01 
REF  Iter = 27 Scaled Residual: 8.361679e-03 Max error: 1.000000e+00 tot_error: 6.470217e-01 
REF  Iter = 28 Scaled Residual: 8.094356e-03 Max error: 1.000000e+00 tot_error: 6.349618e-01 
REF  Iter = 29 Scaled Residual: 7.856248e-03 Max error: 1.000000e+00 tot_error: 6.229842e-01 
REF  Iter = 30 Scaled Residual: 7.638411e-03 Max error: 1.000000e+00 tot_error: 6.110895e-01 
REF  Iter = 31 Scaled Residual: 7.431725e-03 Max error: 1.000000e+00 tot_error: 5.992771e-01 
REF  Iter = 32 Scaled Residual: 7.230632e-03 Max error: 1.000000e+00 tot_error: 5.875451e-01 
REF  Iter = 33 Scaled Residual: 7.033108e-03 Max error: 1.000000e+00 tot_error: 5.758935e-01 
REF  Iter = 34 Scaled Residual: 6.839429e-03 Max error: 1.000000e+00 tot_error: 5.643230e-01 
REF  Iter = 35 Scaled Residual: 6.649585e-03 Max error: 1.000000e+00 tot_error: 5.528367e-01 
REF  Iter = 36 Scaled Residual: 6.460266e-03 Max error: 1.000000e+00 tot_error: 5.414384e-01 
REF  Iter = 37 Scaled Residual: 6.266134e-03 Max error: 1.000000e+00 tot_error: 5.301323e-01 
REF  Iter = 38 Scaled Residual: 6.064667e-03 Max error: 1.000000e+00 tot_error: 5.189197e-01 
REF  Iter = 39 Scaled Residual: 5.859845e-03 Max error: 1.000000e+00 tot_error: 5.077945e-01 
REF  Iter = 40 Scaled Residual: 5.660937e-03 Max error: 1.000000e+00 tot_error: 4.967385e-01 
REF  Iter = 41 Scaled Residual: 5.474462e-03 Max error: 1.000000e+00 tot_error: 4.857365e-01 
REF  Iter = 42 Scaled Residual: 5.303830e-03 Max error: 1.000000e+00 tot_error: 4.747937e-01 
REF  Iter = 43 Scaled Residual: 5.155532e-03 Max error: 1.000000e+00 tot_error: 4.639201e-01 
REF  Iter = 44 Scaled Residual: 5.029801e-03 Max error: 1.000000e+00 tot_error: 4.531153e-01 
REF  Iter = 45 Scaled Residual: 4.919530e-03 Max error: 1.000000e+00 tot_error: 4.423834e-01 
REF  Iter = 46 Scaled Residual: 4.819439e-03 Max error: 1.000000e+00 tot_error: 4.317252e-01 
REF  Iter = 47 Scaled Residual: 4.721621e-03 Max error: 1.000000e+00 tot_error: 4.211404e-01 
REF  Iter = 48 Scaled Residual: 4.624066e-03 Max error: 1.000000e+00 tot_error: 4.106276e-01 
REF  Iter = 49 Scaled Residual: 4.524096e-03 Max error: 1.000000e+00 tot_error: 4.001856e-01 
REF  Iter = 50 Scaled Residual: 4.425263e-03 Max error: 1.000000e+00 tot_error: 3.898165e-01 

Optimization...
Optimization time: 1.389884e-01 sec 

Validation...

Optimized CG Setup...
Initial Residual: 1.130436e+04 Max_err: 1.000000e+00 tot_err: 1.158524e+04
Iteration = 1 Scaled Residual: 2.207820e-01 Max error: 1.000000e+00 tot_error: 9.848263e-01 
Iteration = 2 Scaled Residual: 1.195819e-01 Max error: 1.000000e+00 tot_error: 9.706033e-01 
Iteration = 3 Scaled Residual: 8.139207e-02 Max error: 1.000000e+00 tot_error: 9.566972e-01 
Iteration = 4 Scaled Residual: 6.165808e-02 Max error: 1.000000e+00 tot_error: 9.429398e-01 
Iteration = 5 Scaled Residual: 4.967278e-02 Max error: 1.000000e+00 tot_error: 9.292521e-01 
Iteration = 6 Scaled Residual: 4.158389e-02 Max error: 1.000000e+00 tot_error: 9.156258e-01 
Iteration = 7 Scaled Residual: 3.576953e-02 Max error: 1.000000e+00 tot_error: 9.020490e-01 
Iteration = 8 Scaled Residual: 3.139175e-02 Max error: 1.000000e+00 tot_error: 8.885251e-01 
Iteration = 9 Scaled Residual: 2.796567e-02 Max error: 1.000000e+00 tot_error: 8.750543e-01 
Iteration = 10 Scaled Residual: 2.521453e-02 Max error: 1.000000e+00 tot_error: 8.616418e-01 
Iteration = 11 Scaled Residual: 2.295672e-02 Max error: 1.000000e+00 tot_error: 8.482859e-01 
Iteration = 12 Scaled Residual: 2.106689e-02 Max error: 1.000000e+00 tot_error: 8.349914e-01 
Iteration = 13 Scaled Residual: 1.946574e-02 Max error: 1.000000e+00 tot_error: 8.217580e-01 
Iteration = 14 Scaled Residual: 1.809035e-02 Max error: 1.000000e+00 tot_error: 8.085880e-01 
Iteration = 15 Scaled Residual: 1.689473e-02 Max error: 1.000000e+00 tot_error: 7.954822e-01 
Iteration = 16 Scaled Residual: 1.584746e-02 Max error: 1.000000e+00 tot_error: 7.824436e-01 
Iteration = 17 Scaled Residual: 1.492268e-02 Max error: 1.000000e+00 tot_error: 7.694711e-01 
Iteration = 18 Scaled Residual: 1.410121e-02 Max error: 1.000000e+00 tot_error: 7.565683e-01 
Iteration = 19 Scaled Residual: 1.336749e-02 Max error: 1.000000e+00 tot_error: 7.437350e-01 
Iteration = 20 Scaled Residual: 1.270845e-02 Max error: 1.000000e+00 tot_error: 7.309738e-01 
Iteration = 21 Scaled Residual: 1.211470e-02 Max error: 1.000000e+00 tot_error: 7.182850e-01 
Iteration = 22 Scaled Residual: 1.157777e-02 Max error: 1.000000e+00 tot_error: 7.056725e-01 
Iteration = 23 Scaled Residual: 1.109062e-02 Max error: 1.000000e+00 tot_error: 6.931348e-01 
Iteration = 24 Scaled Residual: 1.064638e-02 Max error: 1.000000e+00 tot_error: 6.806764e-01 
Iteration = 25 Scaled Residual: 1.023819e-02 Max error: 1.000000e+00 tot_error: 6.682961e-01 
Iteration = 26 Scaled Residual: 9.860575e-03 Max error: 1.000000e+00 tot_error: 6.559970e-01 
Iteration = 27 Scaled Residual: 9.508176e-03 Max error: 1.000000e+00 tot_error: 6.437776e-01 
Iteration = 28 Scaled Residual: 9.175130e-03 Max error: 1.000000e+00 tot_error: 6.316402e-01 
Iteration = 29 Scaled Residual: 8.857803e-03 Max error: 1.000000e+00 tot_error: 6.195809e-01 
Iteration = 30 Scaled Residual: 8.552436e-03 Max error: 1.000000e+00 tot_error: 6.076019e-01 
Iteration = 31 Scaled Residual: 8.257667e-03 Max error: 1.000000e+00 tot_error: 5.956986e-01 
Iteration = 32 Scaled Residual: 7.973539e-03 Max error: 1.000000e+00 tot_error: 5.838727e-01 
Iteration = 33 Scaled Residual: 7.700550e-03 Max error: 1.000000e+00 tot_error: 5.721214e-01 
Iteration = 34 Scaled Residual: 7.439293e-03 Max error: 1.000000e+00 tot_error: 5.604475e-01 
Iteration = 35 Scaled Residual: 7.191580e-03 Max error: 1.000000e+00 tot_error: 5.488508e-01 
Iteration = 36 Scaled Residual: 6.957137e-03 Max error: 1.000000e+00 tot_error: 5.373380e-01 
Iteration = 37 Scaled Residual: 6.736682e-03 Max error: 1.000000e+00 tot_error: 5.259116e-01 
Iteration = 38 Scaled Residual: 6.530211e-03 Max error: 1.000000e+00 tot_error: 5.145791e-01 
Iteration = 39 Scaled Residual: 6.338444e-03 Max error: 1.000000e+00 tot_error: 5.033395e-01 
Iteration = 40 Scaled Residual: 6.162254e-03 Max error: 1.000000e+00 tot_error: 4.921890e-01 
Iteration = 41 Scaled Residual: 6.002128e-03 Max error: 1.000000e+00 tot_error: 4.811143e-01 
Iteration = 42 Scaled Residual: 5.853732e-03 Max error: 1.000000e+00 tot_error: 4.701054e-01 
Iteration = 43 Scaled Residual: 5.711831e-03 Max error: 1.000000e+00 tot_error: 4.591599e-01 
Iteration = 44 Scaled Residual: 5.572338e-03 Max error: 1.000000e+00 tot_error: 4.482895e-01 
Iteration = 45 Scaled Residual: 5.436712e-03 Max error: 1.000000e+00 tot_error: 4.374971e-01 
Iteration = 46 Scaled Residual: 5.305107e-03 Max error: 1.000000e+00 tot_error: 4.267747e-01 
Iteration = 47 Scaled Residual: 5.173265e-03 Max error: 1.000000e+00 tot_error: 4.161159e-01 
Iteration = 48 Scaled Residual: 5.039648e-03 Max error: 1.000000e+00 tot_error: 4.055277e-01 
Iteration = 49 Scaled Residual: 4.909518e-03 Max error: 9.999999e-01 tot_error: 3.950059e-01 
Iteration = 50 Scaled Residual: 4.781372e-03 Max error: 9.999998e-01 tot_error: 3.845456e-01 
Iteration = 51 Scaled Residual: 4.656561e-03 Max error: 9.999997e-01 tot_error: 3.741512e-01 
Iteration = 52 Scaled Residual: 4.539125e-03 Max error: 9.999994e-01 tot_error: 3.638202e-01 
Iteration = 53 Scaled Residual: 4.427002e-03 Max error: 9.999989e-01 tot_error: 3.535533e-01 
Iteration = 54 Scaled Residual: 4.322927e-03 Max error: 9.999978e-01 tot_error: 3.433555e-01 

Starting Benchmarking Phase... 
Performing 27 CG sets	 expected time:   60.0 seconds	 expected Perf:    1086.7 GF (135.8 GF_per)
2018-10-23 13:02:53.803
progress = 3.7% 	    2.2 /   60.0 sec elapsed 	   57.8 sec remain 	  1087.355 GF 	 135.919 GF_per
progress = 7.4% 	    4.5 /   60.0 sec elapsed 	   55.5 sec remain 	  1087.455 GF 	 135.932 GF_per
progress = 11.2% 	    6.7 /   60.0 sec elapsed 	   53.3 sec remain 	  1087.503 GF 	 135.938 GF_per
progress = 14.9% 	    8.9 /   60.0 sec elapsed 	   51.1 sec remain 	  1087.449 GF 	 135.931 GF_per
progress = 18.6% 	   11.2 /   60.0 sec elapsed 	   48.8 sec remain 	  1087.438 GF 	 135.930 GF_per
progress = 22.3% 	   13.4 /   60.0 sec elapsed 	   46.6 sec remain 	  1087.471 GF 	 135.934 GF_per
progress = 26.1% 	   15.6 /   60.0 sec elapsed 	   44.4 sec remain 	  1087.478 GF 	 135.935 GF_per
progress = 29.8% 	   17.9 /   60.0 sec elapsed 	   42.1 sec remain 	  1087.493 GF 	 135.937 GF_per
progress = 33.5% 	   20.1 /   60.0 sec elapsed 	   39.9 sec remain 	  1087.510 GF 	 135.939 GF_per
progress = 37.2% 	   22.3 /   60.0 sec elapsed 	   37.7 sec remain 	  1087.537 GF 	 135.942 GF_per
progress = 41.0% 	   24.6 /   60.0 sec elapsed 	   35.4 sec remain 	  1087.552 GF 	 135.944 GF_per
progress = 44.7% 	   26.8 /   60.0 sec elapsed 	   33.2 sec remain 	  1087.480 GF 	 135.935 GF_per
progress = 48.4% 	   29.0 /   60.0 sec elapsed 	   31.0 sec remain 	  1087.495 GF 	 135.937 GF_per
progress = 52.1% 	   31.3 /   60.0 sec elapsed 	   28.7 sec remain 	  1087.511 GF 	 135.939 GF_per
progress = 55.9% 	   33.5 /   60.0 sec elapsed 	   26.5 sec remain 	  1087.506 GF 	 135.938 GF_per
progress = 59.6% 	   35.7 /   60.0 sec elapsed 	   24.3 sec remain 	  1087.497 GF 	 135.937 GF_per
progress = 63.3% 	   38.0 /   60.0 sec elapsed 	   22.0 sec remain 	  1087.453 GF 	 135.932 GF_per
progress = 67.0% 	   40.2 /   60.0 sec elapsed 	   19.8 sec remain 	  1087.443 GF 	 135.930 GF_per
progress = 70.8% 	   42.5 /   60.0 sec elapsed 	   17.5 sec remain 	  1087.437 GF 	 135.930 GF_per
progress = 74.5% 	   44.7 /   60.0 sec elapsed 	   15.3 sec remain 	  1087.401 GF 	 135.925 GF_per
progress = 78.2% 	   46.9 /   60.0 sec elapsed 	   13.1 sec remain 	  1087.418 GF 	 135.927 GF_per
progress = 81.9% 	   49.2 /   60.0 sec elapsed 	   10.8 sec remain 	  1087.411 GF 	 135.926 GF_per
progress = 85.7% 	   51.4 /   60.0 sec elapsed 	    8.6 sec remain 	  1087.395 GF 	 135.924 GF_per
progress = 89.4% 	   53.6 /   60.0 sec elapsed 	    6.4 sec remain 	  1087.374 GF 	 135.922 GF_per
progress = 93.1% 	   55.9 /   60.0 sec elapsed 	    4.1 sec remain 	  1087.384 GF 	 135.923 GF_per
progress = 96.8% 	   58.1 /   60.0 sec elapsed 	    1.9 sec remain 	  1087.379 GF 	 135.922 GF_per

Completed Benchmarking Phase... elapsed time:   60.3 seconds 
2018-10-23 13:03:54.153

Number of CG sets:	27 
Iterations per set:	54 
scaled res mean:	4.322927e-03 
scaled res variance:	0.000000e+00 

Total Time: 6.033050e+01 sec 
Setup        Overhead: 2.52%
Optimization Overhead: 0.62%
Convergence  Overhead: 7.41%

2x2x2 process grid
256x256x256 local domain
SpMV  =  969.1 GF (6102.7 GB/s Effective)  121.1 GF_per ( 762.8 GB/s Effective)
SymGS = 1312.7 GF (10131.8 GB/s Effective)  164.1 GF_per (1266.5 GB/s Effective)
total = 1212.0 GF (9191.0 GB/s Effective)  151.5 GF_per (1148.9 GB/s Effective)
final = 1087.4 GF (8246.0 GB/s Effective)  135.9 GF_per (1030.7 GB/s Effective)

end of application...
2018-10-23 13:03:54.374
 ****** running HPCG -  binary=xhpcg-3.1_gcc_485_cuda-10.0.130_ompi-3.1.0_sm_35_sm_50_sm_60_sm_70_sm_75_ver_10_9_18 on 2 NODES, 16 GPUs ***************************** 
Currently Loaded Modulefiles:
  1) cuda/10.0.130
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'dgx05', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
[dgx05:48233] 47 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix
[dgx05:48233] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

start of application (1 OMP threads)...
2018-10-23 13:04:06.144

Problem setup...
Setup time: 1.163728 sec 

GPU: 'Tesla V100-SXM2-16GB' 
Memory use: 8580 MB / 16130 MB
4x2x2 process grid
256x256x256 local domain

Reference SpMV+MG...

Reference CG...
Initial Residual: 1.458945e+04 Max_err: 1.000000e+00 tot_err: 1.638400e+04
REF  Iter = 1 Scaled Residual: 1.882590e-01 Max error: 1.000000e+00 tot_error: 9.874203e-01 
REF  Iter = 2 Scaled Residual: 1.036687e-01 Max error: 1.000000e+00 tot_error: 9.757378e-01 
REF  Iter = 3 Scaled Residual: 7.163583e-02 Max error: 1.000000e+00 tot_error: 9.642460e-01 
REF  Iter = 4 Scaled Residual: 5.475617e-02 Max error: 1.000000e+00 tot_error: 9.528226e-01 
REF  Iter = 5 Scaled Residual: 4.428867e-02 Max error: 1.000000e+00 tot_error: 9.414380e-01 
REF  Iter = 6 Scaled Residual: 3.716203e-02 Max error: 1.000000e+00 tot_error: 9.300870e-01 
REF  Iter = 7 Scaled Residual: 3.200870e-02 Max error: 1.000000e+00 tot_error: 9.187674e-01 
REF  Iter = 8 Scaled Residual: 2.812588e-02 Max error: 1.000000e+00 tot_error: 9.074817e-01 
REF  Iter = 9 Scaled Residual: 2.510742e-02 Max error: 1.000000e+00 tot_error: 8.962317e-01 
REF  Iter = 10 Scaled Residual: 2.269512e-02 Max error: 1.000000e+00 tot_error: 8.850194e-01 
REF  Iter = 11 Scaled Residual: 2.071567e-02 Max error: 1.000000e+00 tot_error: 8.738445e-01 
REF  Iter = 12 Scaled Residual: 1.904986e-02 Max error: 1.000000e+00 tot_error: 8.627055e-01 
REF  Iter = 13 Scaled Residual: 1.761595e-02 Max error: 1.000000e+00 tot_error: 8.516001e-01 
REF  Iter = 14 Scaled Residual: 1.636371e-02 Max error: 1.000000e+00 tot_error: 8.405258e-01 
REF  Iter = 15 Scaled Residual: 1.526384e-02 Max error: 1.000000e+00 tot_error: 8.294822e-01 
REF  Iter = 16 Scaled Residual: 1.429749e-02 Max error: 1.000000e+00 tot_error: 8.184703e-01 
REF  Iter = 17 Scaled Residual: 1.345095e-02 Max error: 1.000000e+00 tot_error: 8.074926e-01 
REF  Iter = 18 Scaled Residual: 1.271028e-02 Max error: 1.000000e+00 tot_error: 7.965521e-01 
REF  Iter = 19 Scaled Residual: 1.205882e-02 Max error: 1.000000e+00 tot_error: 7.856516e-01 
REF  Iter = 20 Scaled Residual: 1.147931e-02 Max error: 1.000000e+00 tot_error: 7.747915e-01 
REF  Iter = 21 Scaled Residual: 1.095482e-02 Max error: 1.000000e+00 tot_error: 7.639715e-01 
REF  Iter = 22 Scaled Residual: 1.047149e-02 Max error: 1.000000e+00 tot_error: 7.531894e-01 
REF  Iter = 23 Scaled Residual: 1.002255e-02 Max error: 1.000000e+00 tot_error: 7.424430e-01 
REF  Iter = 24 Scaled Residual: 9.607398e-03 Max error: 1.000000e+00 tot_error: 7.317314e-01 
REF  Iter = 25 Scaled Residual: 9.228480e-03 Max error: 1.000000e+00 tot_error: 7.210555e-01 
REF  Iter = 26 Scaled Residual: 8.889497e-03 Max error: 1.000000e+00 tot_error: 7.104178e-01 
REF  Iter = 27 Scaled Residual: 8.591413e-03 Max error: 1.000000e+00 tot_error: 6.998225e-01 
REF  Iter = 28 Scaled Residual: 8.330440e-03 Max error: 1.000000e+00 tot_error: 6.892727e-01 
REF  Iter = 29 Scaled Residual: 8.098671e-03 Max error: 1.000000e+00 tot_error: 6.787708e-01 
REF  Iter = 30 Scaled Residual: 7.884743e-03 Max error: 1.000000e+00 tot_error: 6.683170e-01 
REF  Iter = 31 Scaled Residual: 7.677429e-03 Max error: 1.000000e+00 tot_error: 6.579096e-01 
REF  Iter = 32 Scaled Residual: 7.470716e-03 Max error: 1.000000e+00 tot_error: 6.475459e-01 
REF  Iter = 33 Scaled Residual: 7.263598e-03 Max error: 1.000000e+00 tot_error: 6.372247e-01 
REF  Iter = 34 Scaled Residual: 7.058386e-03 Max error: 1.000000e+00 tot_error: 6.269458e-01 
REF  Iter = 35 Scaled Residual: 6.857189e-03 Max error: 1.000000e+00 tot_error: 6.167112e-01 
REF  Iter = 36 Scaled Residual: 6.658340e-03 Max error: 1.000000e+00 tot_error: 6.065235e-01 
REF  Iter = 37 Scaled Residual: 6.457707e-03 Max error: 1.000000e+00 tot_error: 5.963851e-01 
REF  Iter = 38 Scaled Residual: 6.253557e-03 Max error: 1.000000e+00 tot_error: 5.862962e-01 
REF  Iter = 39 Scaled Residual: 6.049729e-03 Max error: 1.000000e+00 tot_error: 5.762520e-01 
REF  Iter = 40 Scaled Residual: 5.855143e-03 Max error: 1.000000e+00 tot_error: 5.662394e-01 
REF  Iter = 41 Scaled Residual: 5.676508e-03 Max error: 1.000000e+00 tot_error: 5.562476e-01 
REF  Iter = 42 Scaled Residual: 5.516830e-03 Max error: 1.000000e+00 tot_error: 5.462806e-01 
REF  Iter = 43 Scaled Residual: 5.380606e-03 Max error: 1.000000e+00 tot_error: 5.363472e-01 
REF  Iter = 44 Scaled Residual: 5.266921e-03 Max error: 1.000000e+00 tot_error: 5.264489e-01 
REF  Iter = 45 Scaled Residual: 5.167420e-03 Max error: 1.000000e+00 tot_error: 5.165885e-01 
REF  Iter = 46 Scaled Residual: 5.074576e-03 Max error: 1.000000e+00 tot_error: 5.067661e-01 
REF  Iter = 47 Scaled Residual: 4.979902e-03 Max error: 1.000000e+00 tot_error: 4.969801e-01 
REF  Iter = 48 Scaled Residual: 4.881020e-03 Max error: 1.000000e+00 tot_error: 4.872279e-01 
REF  Iter = 49 Scaled Residual: 4.777131e-03 Max error: 1.000000e+00 tot_error: 4.775073e-01 
REF  Iter = 50 Scaled Residual: 4.673238e-03 Max error: 1.000000e+00 tot_error: 4.678192e-01 

Optimization...
Optimization time: 1.584322e-01 sec 

Validation...

Optimized CG Setup...
Initial Residual: 1.458945e+04 Max_err: 1.000000e+00 tot_err: 1.638400e+04
Iteration = 1 Scaled Residual: 2.206650e-01 Max error: 1.000000e+00 tot_error: 9.873501e-01 
Iteration = 2 Scaled Residual: 1.194731e-01 Max error: 1.000000e+00 tot_error: 9.754804e-01 
Iteration = 3 Scaled Residual: 8.133623e-02 Max error: 1.000000e+00 tot_error: 9.638621e-01 
Iteration = 4 Scaled Residual: 6.163521e-02 Max error: 1.000000e+00 tot_error: 9.523538e-01 
Iteration = 5 Scaled Residual: 4.967222e-02 Max error: 1.000000e+00 tot_error: 9.408881e-01 
Iteration = 6 Scaled Residual: 4.159753e-02 Max error: 1.000000e+00 tot_error: 9.294579e-01 
Iteration = 7 Scaled Residual: 3.579382e-02 Max error: 1.000000e+00 tot_error: 9.180525e-01 
Iteration = 8 Scaled Residual: 3.142501e-02 Max error: 1.000000e+00 tot_error: 9.066745e-01 
Iteration = 9 Scaled Residual: 2.800633e-02 Max error: 1.000000e+00 tot_error: 8.953235e-01 
Iteration = 10 Scaled Residual: 2.526169e-02 Max error: 1.000000e+00 tot_error: 8.840036e-01 
Iteration = 11 Scaled Residual: 2.300980e-02 Max error: 1.000000e+00 tot_error: 8.727130e-01 
Iteration = 12 Scaled Residual: 2.112549e-02 Max error: 1.000000e+00 tot_error: 8.614553e-01 
Iteration = 13 Scaled Residual: 1.952990e-02 Max error: 1.000000e+00 tot_error: 8.502300e-01 
Iteration = 14 Scaled Residual: 1.816012e-02 Max error: 1.000000e+00 tot_error: 8.390388e-01 
Iteration = 15 Scaled Residual: 1.696997e-02 Max error: 1.000000e+00 tot_error: 8.278818e-01 
Iteration = 16 Scaled Residual: 1.592831e-02 Max error: 1.000000e+00 tot_error: 8.167614e-01 
Iteration = 17 Scaled Residual: 1.500932e-02 Max error: 1.000000e+00 tot_error: 8.056764e-01 
Iteration = 18 Scaled Residual: 1.419393e-02 Max error: 1.000000e+00 tot_error: 7.946295e-01 
Iteration = 19 Scaled Residual: 1.346662e-02 Max error: 1.000000e+00 tot_error: 7.836202e-01 
Iteration = 20 Scaled Residual: 1.281420e-02 Max error: 1.000000e+00 tot_error: 7.726507e-01 
Iteration = 21 Scaled Residual: 1.222729e-02 Max error: 1.000000e+00 tot_error: 7.617208e-01 
Iteration = 22 Scaled Residual: 1.169737e-02 Max error: 1.000000e+00 tot_error: 7.508336e-01 
Iteration = 23 Scaled Residual: 1.121707e-02 Max error: 1.000000e+00 tot_error: 7.399877e-01 
Iteration = 24 Scaled Residual: 1.077921e-02 Max error: 1.000000e+00 tot_error: 7.291866e-01 
Iteration = 25 Scaled Residual: 1.037635e-02 Max error: 1.000000e+00 tot_error: 7.184287e-01 
Iteration = 26 Scaled Residual: 1.000269e-02 Max error: 1.000000e+00 tot_error: 7.077164e-01 
Iteration = 27 Scaled Residual: 9.652610e-03 Max error: 1.000000e+00 tot_error: 6.970477e-01 
Iteration = 28 Scaled Residual: 9.320347e-03 Max error: 1.000000e+00 tot_error: 6.864242e-01 
Iteration = 29 Scaled Residual: 9.002589e-03 Max error: 1.000000e+00 tot_error: 6.758415e-01 
Iteration = 30 Scaled Residual: 8.696476e-03 Max error: 1.000000e+00 tot_error: 6.653012e-01 
Iteration = 31 Scaled Residual: 8.401402e-03 Max error: 1.000000e+00 tot_error: 6.547990e-01 
Iteration = 32 Scaled Residual: 8.118345e-03 Max error: 1.000000e+00 tot_error: 6.443360e-01 
Iteration = 33 Scaled Residual: 7.848076e-03 Max error: 1.000000e+00 tot_error: 6.339097e-01 
Iteration = 34 Scaled Residual: 7.591385e-03 Max error: 1.000000e+00 tot_error: 6.235223e-01 
Iteration = 35 Scaled Residual: 7.349776e-03 Max error: 1.000000e+00 tot_error: 6.131736e-01 
Iteration = 36 Scaled Residual: 7.122789e-03 Max error: 1.000000e+00 tot_error: 6.028688e-01 
Iteration = 37 Scaled Residual: 6.910518e-03 Max error: 1.000000e+00 tot_error: 5.926100e-01 
Iteration = 38 Scaled Residual: 6.712586e-03 Max error: 1.000000e+00 tot_error: 5.824029e-01 
Iteration = 39 Scaled Residual: 6.528862e-03 Max error: 1.000000e+00 tot_error: 5.722471e-01 
Iteration = 40 Scaled Residual: 6.359736e-03 Max error: 1.000000e+00 tot_error: 5.621404e-01 
Iteration = 41 Scaled Residual: 6.205140e-03 Max error: 1.000000e+00 tot_error: 5.520727e-01 
Iteration = 42 Scaled Residual: 6.061087e-03 Max error: 1.000000e+00 tot_error: 5.420363e-01 
Iteration = 43 Scaled Residual: 5.922689e-03 Max error: 1.000000e+00 tot_error: 5.320284e-01 
Iteration = 44 Scaled Residual: 5.786226e-03 Max error: 1.000000e+00 tot_error: 5.220566e-01 
Iteration = 45 Scaled Residual: 5.652517e-03 Max error: 1.000000e+00 tot_error: 5.121227e-01 
Iteration = 46 Scaled Residual: 5.522107e-03 Max error: 1.000000e+00 tot_error: 5.022203e-01 
Iteration = 47 Scaled Residual: 5.391834e-03 Max error: 1.000000e+00 tot_error: 4.923441e-01 
Iteration = 48 Scaled Residual: 5.260840e-03 Max error: 1.000000e+00 tot_error: 4.824984e-01 
Iteration = 49 Scaled Residual: 5.133806e-03 Max error: 9.999999e-01 tot_error: 4.726800e-01 
Iteration = 50 Scaled Residual: 5.010070e-03 Max error: 9.999999e-01 tot_error: 4.628853e-01 
Iteration = 51 Scaled Residual: 4.890753e-03 Max error: 9.999998e-01 tot_error: 4.531175e-01 
Iteration = 52 Scaled Residual: 4.779287e-03 Max error: 9.999996e-01 tot_error: 4.433746e-01 
Iteration = 53 Scaled Residual: 4.673833e-03 Max error: 9.999992e-01 tot_error: 4.336572e-01 
Iteration = 54 Scaled Residual: 4.576485e-03 Max error: 9.999985e-01 tot_error: 4.239690e-01 

Starting Benchmarking Phase... 
Performing 27 CG sets	 expected time:   60.0 seconds	 expected Perf:    2073.9 GF (129.6 GF_per)
2018-10-23 13:09:26.562
progress = 3.8% 	    2.3 /   60.0 sec elapsed 	   57.7 sec remain 	  2074.450 GF 	 129.653 GF_per
progress = 7.6% 	    4.6 /   60.0 sec elapsed 	   55.4 sec remain 	  2074.550 GF 	 129.659 GF_per
progress = 11.4% 	    6.9 /   60.0 sec elapsed 	   53.1 sec remain 	  2074.558 GF 	 129.660 GF_per
progress = 15.2% 	    9.1 /   60.0 sec elapsed 	   50.9 sec remain 	  2074.189 GF 	 129.637 GF_per
progress = 19.1% 	   11.4 /   60.0 sec elapsed 	   48.6 sec remain 	  2074.134 GF 	 129.633 GF_per
progress = 22.9% 	   13.7 /   60.0 sec elapsed 	   46.3 sec remain 	  2073.957 GF 	 129.622 GF_per
progress = 26.7% 	   16.0 /   60.0 sec elapsed 	   44.0 sec remain 	  2073.609 GF 	 129.601 GF_per
progress = 30.5% 	   18.3 /   60.0 sec elapsed 	   41.7 sec remain 	  2073.652 GF 	 129.603 GF_per
progress = 34.3% 	   20.6 /   60.0 sec elapsed 	   39.4 sec remain 	  2073.707 GF 	 129.607 GF_per
progress = 38.1% 	   22.9 /   60.0 sec elapsed 	   37.1 sec remain 	  2073.800 GF 	 129.612 GF_per
progress = 41.9% 	   25.2 /   60.0 sec elapsed 	   34.8 sec remain 	  2073.871 GF 	 129.617 GF_per
progress = 45.8% 	   27.5 /   60.0 sec elapsed 	   32.5 sec remain 	  2073.862 GF 	 129.616 GF_per
progress = 49.6% 	   29.7 /   60.0 sec elapsed 	   30.3 sec remain 	  2073.873 GF 	 129.617 GF_per
progress = 53.4% 	   32.0 /   60.0 sec elapsed 	   28.0 sec remain 	  2073.909 GF 	 129.619 GF_per
progress = 57.2% 	   34.3 /   60.0 sec elapsed 	   25.7 sec remain 	  2073.948 GF 	 129.622 GF_per
progress = 61.0% 	   36.6 /   60.0 sec elapsed 	   23.4 sec remain 	  2073.891 GF 	 129.618 GF_per
progress = 64.8% 	   38.9 /   60.0 sec elapsed 	   21.1 sec remain 	  2073.808 GF 	 129.613 GF_per
progress = 68.6% 	   41.2 /   60.0 sec elapsed 	   18.8 sec remain 	  2073.678 GF 	 129.605 GF_per
progress = 72.5% 	   43.5 /   60.0 sec elapsed 	   16.5 sec remain 	  2073.644 GF 	 129.603 GF_per
progress = 76.3% 	   45.8 /   60.0 sec elapsed 	   14.2 sec remain 	  2073.619 GF 	 129.601 GF_per
progress = 80.1% 	   48.0 /   60.0 sec elapsed 	   12.0 sec remain 	  2073.564 GF 	 129.598 GF_per
progress = 83.9% 	   50.3 /   60.0 sec elapsed 	    9.7 sec remain 	  2073.524 GF 	 129.595 GF_per
progress = 87.7% 	   52.6 /   60.0 sec elapsed 	    7.4 sec remain 	  2073.564 GF 	 129.598 GF_per
progress = 91.5% 	   54.9 /   60.0 sec elapsed 	    5.1 sec remain 	  2073.531 GF 	 129.596 GF_per
progress = 95.3% 	   57.2 /   60.0 sec elapsed 	    2.8 sec remain 	  2073.526 GF 	 129.595 GF_per
progress = 99.2% 	   59.5 /   60.0 sec elapsed 	    0.5 sec remain 	  2073.523 GF 	 129.595 GF_per

Completed Benchmarking Phase... elapsed time:   61.8 seconds 
2018-10-23 13:10:28.384

Number of CG sets:	27 
Iterations per set:	54 
scaled res mean:	4.576485e-03 
scaled res variance:	0.000000e+00 

Total Time: 6.177929e+01 sec 
Setup        Overhead: 4.84%
Optimization Overhead: 0.69%
Convergence  Overhead: 7.41%

4x2x2 process grid
256x256x256 local domain
SpMV  = 1939.2 GF (12211.5 GB/s Effective)  121.2 GF_per ( 763.2 GB/s Effective)
SymGS = 2589.4 GF (19984.8 GB/s Effective)  161.8 GF_per (1249.0 GB/s Effective)
total = 2368.8 GF (17963.4 GB/s Effective)  148.1 GF_per (1122.7 GB/s Effective)
final = 2073.5 GF (15724.2 GB/s Effective)  129.6 GF_per ( 982.8 GB/s Effective)

end of application...
2018-10-23 13:10:28.832
