Re: DAcreate2d process layout order

[Barry Smith <bsmith@xxxxxxxxxxx>]

On Thu, 18 May 2006, Sean Dettrick wrote:

> Hi Barry,
> the order is determined by MPI_Cart_create.

   Do you mean that MPI_Cart_create() orders across the 2nd (y-axis)
fastest and then the first (x-axis)? Hmmm, maybe we should change the
DA? Changing it once and for all (not supporting both) is probably
not a big deal and shouldn't break much (I hope).
Bill, what do you think?

   Barry

>  I've tried to swap x and y in 
> the DA to make the problem go away but haven't succeeded yet.  Not enough 
> brain cells I suspect ... still thinking about it.
> Thanks for the response,
> Sean
>
> Barry Smith wrote:
>
> >   Sean,
> >
> >    I don't completely understand what goes wrong. Is it because YOUR
> > application orders the processors related to geometry in the following way?
> >
> >     ^ y direction
> >     |
> >        2   5  8
> >        1   4  7
> >        0   3  6
> >
> >                     -> x direction
> >
> > Or is this something inherent in MPI_Cart_create?
> >
> > PETSc does it so
> >
> >     ^ y direction
> >     |
> >        6   7  8
> >        3   4  5
> >        0   1  2
> >
> >                     -> x direction
> >
> >   If you want to "force" the PETSc DA to match the first case you could 
> > just
> > "lie" to PETSc and treat the x direction as the y direction and the x as 
> > the y.
> > Thus you would flip the i and j indices. You could do the same trick in
> > 3d.
> >
> > There is no easy way to change the DA to do the ordering as the first
> > case above or support both approaches (possible? yes, but ugly
> > duplicate code to handle the two cases).
> >
> > I've cc:ed to Bill since he would know more about the details of 
> > MPI_cart_create().
> >
> >    Barry
> >
> > On Mon, 15 May 2006, Sean Dettrick wrote:
> >
> > > Hi,
> > >
> > > I'm trying to use DACreate2d and KSP in my existing MPI application.  I 
> > > already have a Cartesian communicator established, and I set 
> > > PETSC_COMM_WORLD equal to it and then call PetscInitialize.
> > >
> > > This works fine on a prime number of CPUs, because there is only one 
> > > possible ordered MPI layout in one dimension.  But with a non-prime number 
> > > there are two possible ordered layouts and it just happens that my 2D CPU 
> > > layout (determined by MPI_Cart_create) is the transpose of the PETSc 2D 
> > > CPU layout. Is there a way to organize the DA layout more explicitly than 
> > > with DACreate2d?   Or to tell PETSc to transpose its CPU order?  I also 
> > > wonder about the 3D case.
> > >
> > > thanks
> > > Sean