[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: DAcreate2d process layout order

Hi Barry,
the order is determined by MPI_Cart_create. I've tried to swap x and y in the DA to make the problem go away but haven't succeeded yet. Not enough brain cells I suspect ... still thinking about it.
Thanks for the response,
Sean

Barry Smith wrote:



  Sean,

I don't completely understand what goes wrong. Is it because YOUR
application orders the processors related to geometry in the following way?

    ^ y direction
    |
       2   5  8
       1   4  7
       0   3  6

                    -> x direction

Or is this something inherent in MPI_Cart_create?

PETSc does it so

    ^ y direction
    |
       6   7  8
       3   4  5
       0   1  2

                    -> x direction

If you want to "force" the PETSc DA to match the first case you could just
"lie" to PETSc and treat the x direction as the y direction and the x as the y.
Thus you would flip the i and j indices. You could do the same trick in
3d.

There is no easy way to change the DA to do the ordering as the first
case above or support both approaches (possible? yes, but ugly
duplicate code to handle the two cases).

I've cc:ed to Bill since he would know more about the details of MPI_cart_create().

   Barry

On Mon, 15 May 2006, Sean Dettrick wrote:

Hi,

I'm trying to use DACreate2d and KSP in my existing MPI application. I already have a Cartesian communicator established, and I set PETSC_COMM_WORLD equal to it and then call PetscInitialize.

This works fine on a prime number of CPUs, because there is only one possible ordered MPI layout in one dimension. But with a non-prime number there are two possible ordered layouts and it just happens that my 2D CPU layout (determined by MPI_Cart_create) is the transpose of the PETSc 2D CPU layout. Is there a way to organize the DA layout more explicitly than with DACreate2d? Or to tell PETSc to transpose its CPU order? I also wonder about the 3D case.

thanks
Sean