[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Non repeatability issue




Aldo,

Actually it is far easier than I thought. I have added the argument -vecscatter_reproduce
that will cause the receives to always be processed in the same order (though order or
operations in the MPI reductions may still result in slightly different convergence histories.)
to petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html


If you are using petsc-2.3.3 and do no wish to change to using petsc-dev you can edit
src/vec/vec/utils/vpscat.h and replace the line
ierr = MPI_Waitany(nrecvs,rwaits,&imdex,&xrstatus);CHKERRQ(ierr);
with the two lines
imdex = count - 1;
ierr = MPI_Wait(rwaits+imdex,&xrstatus);CHKERRQ(ierr);
then run "make libfast" in that directory.


  Hope this helps clear things up,


Barry

On Mar 12, 2008, at 11:02 AM, Aldo Bonfiglioli wrote:

Barry,
thanks for your prompt reply.

1) Have you made runs where you require, say -ksp_rtol 1.e-12 to eliminate the effects of
not solving the linear systems accurately?

I am trying this right now, although it may require very many linear
iterations, particularly in the last Newton steps, when the CFL number
becomes infinite.
The systems arising from the fully coupled RANS eqns appear to be fairly ill-conditioned:
even in the 2D cases we generally need to increase the level of fill beyond 0
to obtain reasonable convergence of the linear solver.


In the 3D case (internal flow through an elbow) the previous plots refer to, there are
about 200,000 nodes times 5 dof per node and
I have been using BJ+ILU(1) with GMRES(60).


2) Have you run the exact example that you ran with geometric decomposition also with
the parmetis decomposition? Is that what you sent? (This is too eliminate any fundamental
differences with the two problems.)

Precisely.

3) In your plots you plot L_2 norm of the mass residual while Newton is running on all
equations. This means the Newton's criteria for progress is based on || u,v,m .....||
as it chugs along. What do plots of || u,v, m....|| (that is what Newton calls the residual
when you use -snes_monitor) look like, are they also unstable? Are they decreasing?

All residuals appear to be fairly un-stable. This is shown in the enclosed plot.
The strategy to ramp the CFL number looks at the ratio btw the current non-linear
residual _of one of the conservation eqns_ and the initial residual.
The Newton strategy adopted in the code is NOT implemented through PETSc's TS
(just for "historical" reasons).


Sometimes people scale some equations stronger than others if those are the residuals
they are most interested in pushing down. What happens if you scale the mass residual
equations by some factor (say 100 or 1000) in your FormFunction?

Never tried. We had once tested on a different problem row scaling without seeing noticeable differences. But I guess this may be different from what you mention.


Regards, Aldo

--
Dr. Aldo Bonfiglioli
Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA)
Universita' della Basilicata
V.le dell'Ateneo lucano, 10 85100 Potenza ITALY
tel:+39.0971.205203 fax:+39.0971.205160

<convhst.pdf>