1) Are you sure the -vecscatter_reproduce is working, run with -
options_left and see if
says the option was not used.
I have harwired it into the 2.3.3-p8, following your suggestion.
2) did you do the -ksp_rtol 1.e-12 at the same time as the -
vecscatter_reproduce? They
must be done together.
No, I tested the two separately. I will do as you suggest.
3) what happens on 1 process? Does it behave exactly the same for
two identical runs?
The current testcase is too large to be run on a single processor.
Tests performed with smaller datasets (both 2 and 3D) have shown
that on 1 proc
subsequent runs produce identical output.
It should also be mentioned that, on a different grid (somewhat less
stretched)
the same testcase produces far more repeatible non-linear
convergence histories.
By "far more repeatible" I mean that the output of subsequent runs
are NOT
identical, but the non-linear convergence histories are almost
superimposed at the "plotting" level, small differences arising only
when the residuals are close to machine eps.
On this grid, the linear solver (either BJ+ILU(k) or ASM+ILU(k))
also behaves far better.
4) there is too much going on here to figure out why you get this
behavior. Can you please
FIX the continuation parameter
Here I am not sure about the nomenclature: by "continuation
parameter" do you
mean the strategy by which the pseudo-time derivative term is
progressively reduced so as to revert to a true Newton algorithm?
My pseudo-time derivative term looks like
1/(CFL) * (Volume/dt)
Volume/dt is locally computed based on an explicit stability
criterion and CFL is ramped from a starting value (CFL_0, tipically
of order 1 or 10)
using the ratio between the current and initial residuals of one
of the conservation eqns (mass, tipically).