[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problems porting code to an IBM power5+ machine

On 3/20/07, Knut Erik Teigen <knutert@xxxxxxxxxxxx> wrote: 
On Mon, 2007-03-19 at 08:39 -0500, Matthew Knepley wrote:
>   This smells like a memory overwrite or use of an uninitialized
> variable. The initial norm is 1e28.
>
> 1) Try Jacobi instead of ICC to see if it is localized to the PC

With jacobi I get:
[0] PCSetUpSetting up new PC
  0 KSP Residual norm 1.466015468749e+13
  1 KSP Residual norm 2.720083022075e+23
[0] KSPDefaultConvergedLinear solver is diverging. Initial right hand
size norm 1.46602e+13, current residual norm 2.72008e+23 at iteration 1

And without a preconditioner at all:
 0 KSP Residual norm 7.719804763794e+00
  1 KSP Residual norm 1.137387752533e+01
[0] KSPSolve_CGdiverging due to indefinite or negative definite matrix

The matrix definitely isn't indefinite or negative definite, as is clear
from the output in my previous post.

> 2) Run with valgrind or something similar to check for a memory overwrite
>
> 3) Maybe insert CHKMEMQ statements into the code

I've run with CHKMEMQ statements, and with -malloc_debug, but didn't get
any complaints.
Valgrind unfortunately isn't installed on the cluster.


Then I am afraid it will be a slow painful process. I am almost
positive that you
are overwriting memory somewhere. It is different on the two machines because
the initial layout is different. The huge norm can be traced back to a huge Vec
element, which can be localied. You will just have to go through the code
methodically with the debugger to find it. Or install valgrind.

  Matt

-Knut Erik-
>
>   Thanks,
>
>     Matt
>
> On 3/19/07, Knut Erik Teigen <knutert@xxxxxxxxxxxx> wrote:
> > Hello
> >
> > I have got a working code on my local machine( Pentium 4), but when I
> > try to run the code on a power5+ machine, the equation solver won't
> > converge. It seems like it calculates the wrong right hand side norm.
> > Below is the result with run-time options
> > "-ksp_type cg -pc_type icc -ksp_monitor -ksp_view -info
> > First with the code running on the power5+ machine, then on my local
> > machine. I've also printed the right hand side, jacobian matrix and
> > solution for a small 3x3 grid.
> >
> > Can anyone help me figure out what's wrong?
> >
> > Regards,
> > Knut Erik Teigen
> >
> > Code running on Power5+:
> > rhs:
> > 980
> > 980
> > 980
> > -0
> > -0
> > -0
> > -980
> > -980
> > -980
> > jacobian:
> > row 0: (0, 20)  (1, -10)  (3, -10)
> > row 1: (0, -10)  (1, 30)  (2, -10)  (4, -10)
> > row 2: (1, -10)  (2, 20)  (5, -10)
> > row 3: (0, -10)  (3, 30)  (4, -10)  (6, -10)
> > row 4: (1, -10)  (3, -10)  (4, 40)  (5, -10)  (7, -10)
> > row 5: (2, -10)  (4, -10)  (5, 30)  (8, -10)
> > row 6: (3, -10)  (6, 20)  (7, -10)
> > row 7: (4, -10)  (6, -10)  (7, 30)  (8, -10)
> > row 8: (5, -10)  (7, -10)  (8, 20)
> > [0] PCSetUpSetting up new PC
> > [0] PetscCommDuplicateDuplicating a communicator 1 4 max tags =
> > 1073741823
> > [0] PetscCommDuplicateUsing internal PETSc communicator 1 4
> > [0] PetscCommDuplicateUsing internal PETSc communicator 1 4
> >   0 KSP Residual norm 7.410163701832e+28
> >   1 KSP Residual norm 6.464393707520e+11
> > [0] KSPDefaultConvergedLinear solver has converged. Residual norm
> > 6.46439e+11 is less than relative tolerance 1e-07 times initial right
> > hand side norm 7.41016e+28 at iteration 1
> > KSP Object:
> >   type: cg
> >   maximum iterations=10000, initial guess is zero
> >   tolerances:  relative=1e-07, absolute=1e-50, divergence=10000
> >   left preconditioning
> > PC Object:
> >   type: icc
> >     ICC: 0 levels of fill
> >     ICC: factor fill ratio allocated 1
> >     ICC: factor fill ratio needed 0.636364
> >          Factored matrix follows
> >         Matrix Object:
> >           type=seqsbaij, rows=9, cols=9
> >           total: nonzeros=21, allocated nonzeros=21
> >               block size is 1
> >   linear system matrix = precond matrix:
> >   Matrix Object:
> >     type=seqaij, rows=9, cols=9
> >     total: nonzeros=33, allocated nonzeros=45
> >       not using I-node routines
> > solution:
> > 6.37205e-09
> > 7.13167e-09
> > 7.49911e-09
> > -2.48277e-09
> > -4.56885e-10
> > 0
> > 0
> > 0
> > 0
> >
> > Code running on local machine:
> > rhs:
> > 980
> > 980
> > 980
> > -0
> > -0
> > -0
> > -980
> > -980
> > -980
> > jacobian:
> > row 0: (0, 20)  (1, -10)  (3, -10)
> > row 1: (0, -10)  (1, 30)  (2, -10)  (4, -10)
> > row 2: (1, -10)  (2, 20)  (5, -10)
> > row 3: (0, -10)  (3, 30)  (4, -10)  (6, -10)
> > row 4: (1, -10)  (3, -10)  (4, 40)  (5, -10)  (7, -10)
> > row 5: (2, -10)  (4, -10)  (5, 30)  (8, -10)
> > row 6: (3, -10)  (6, 20)  (7, -10)
> > row 7: (4, -10)  (6, -10)  (7, 30)  (8, -10)
> > row 8: (5, -10)  (7, -10)  (8, 20)
> > [0] PCSetUp(): Setting up new PC
> > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689
> > -2080374783 max tags = 2147483647
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374783
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374783
> >   0 KSP Residual norm 2.505507810276e+02
> >   1 KSP Residual norm 3.596555656581e+01
> >   2 KSP Residual norm 2.632672485513e+00
> >   3 KSP Residual norm 1.888285055287e-01
> >   4 KSP Residual norm 7.029433008806e-03
> >   5 KSP Residual norm 3.635267067420e-14
> > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm
> > 3.63527e-14 is less than relative tolerance 1e-07 times initial right
> > hand side norm 250.551 at iteration 5
> > KSP Object:
> >   type: cg
> >   maximum iterations=10000, initial guess is zero
> >   tolerances:  relative=1e-07, absolute=1e-50, divergence=10000
> >   left preconditioning
> > PC Object:
> >   type: icc
> >     ICC: 0 levels of fill
> >     ICC: factor fill ratio allocated 1
> >     ICC: factor fill ratio needed 0.636364
> >          Factored matrix follows
> >         Matrix Object:
> >           type=seqsbaij, rows=9, cols=9
> >           total: nonzeros=21, allocated nonzeros=21
> >               block size is 1
> >   linear system matrix = precond matrix:
> >   Matrix Object:
> >     type=seqaij, rows=9, cols=9
> >     total: nonzeros=33, allocated nonzeros=45
> >       not using I-node routines
> > solution:
> > 92.3023
> > 92.3023
> > 92.3023
> > -5.69767
> > -5.69767
> > -5.69767
> > -103.698
> > -103.698
> > -103.698
> >
> >
> >
> >
>
>




--
One trouble is that despite this system, anyone who reads journals widely
and critically is forced to realize that there are scarcely any bars to eventual
publication. There seems to be no study too fragmented, no hypothesis too
trivial, no literature citation too biased or too egotistical, no design too
warped, no methodology too bungled, no presentation of results too
inaccurate, too obscure, and too contradictory, no analysis too self-serving,
no argument too circular, no conclusions too trifling or too unjustified, and
no grammar and syntax too offensive for a paper to end up in print. --
Drummond Rennie