[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: DAcreate2d process layout order

Barry,

just to follow up, I created a cartesian communicator workaround, without changing the order of x and y in the DA calls.

Instead I permuted my periodicity orders, created the 2D cartesian communicator, and then created with MPI_Cart_sub an array of 1D cartesian communicators, one for each dimension, with the dimension orders permuted.

Now I can use, say, Cartcomm1D[0] to send in the 0th (x) direction of the DA, with the periodicity matching DA_XPERIODIC.

I still can't use the 2D communicator - it still has confused dimension order and periodicity. But the 1D communicators are a good compromise.

Thanks for your help,
Sean

Barry Smith wrote:


Ok, so it is laid-out like a matrix, not like x, y coordinates.

Will put the change to DA on the list of things to do.


Barry

On Thu, 18 May 2006, Sean Dettrick wrote:

Barry Smith wrote:


Bill, does the MPI standard dictate this decomposition or
could different implementations do it the opposite way?
Then we'd have to make the DA logic a bit more complicated.


I don't have a copy of the standard, but to quote page 255 of "MPI, the complete reference" by Snir et al:
"Row-major numbering is always used for the processes in a Cartesian structure".
Their diagram in figure 6.1 matches my code output for coords couplets (i,j):

0 1 2 3
(0,0) (0,1) (0,2) (0,3)

4 5 6 7
(1,0) (1,1) (1,2) (1,3)

8 9 10 11
(2,0) (2,1) (2,2) (2,3)


By the way I agree with you, I *should* be able to swap the x and y myself. Just haven't had much luck yet in that regard.

Sean



On Thu, 18 May 2006, Sean Dettrick wrote:

Barry Smith wrote:

On Thu, 18 May 2006, Sean Dettrick wrote:

Hi Barry,
the order is determined by MPI_Cart_create.





Do you mean that MPI_Cart_create() orders across the 2nd (y-axis)
fastest and then the first (x-axis)? Hmmm, maybe we should change the
DA? Changing it once and for all (not supporting both) is probably
not a big deal and shouldn't break much (I hope).




Hi Barry,

it depends, what do you call x and what do you call y?
MPI_Cart_coords returns a vector, coords - I tend to say x is coords[0], y is coords[1] and z is coords[2]. For what it's worth, there's a short code appended to this email, which produces:

rank = 0 has Cartesian coords = { 0, 0 }
rank = 1 has Cartesian coords = { 0, 1 }
rank = 2 has Cartesian coords = { 1, 0 }
rank = 3 has Cartesian coords = { 1, 1 }
rank = 0 has DA range x=[0,50) and y=[0,50)
rank = 1 has DA range x=[50,100) and y=[0,50)
rank = 2 has DA range x=[0,50) and y=[50,100)
rank = 3 has DA range x=[50,100) and y=[50,100)

I don't completely understand what goes wrong. Is it because YOUR
application orders the processors related to geometry in the following way?

^ y direction
|
2 5 8
1 4 7
0 3 6

-> x direction

Or is this something inherent in MPI_Cart_create?




For my interpretation of x and y, MPI_Cart_create produces the above layout. But if I said x=coords[1] and y=coords[0], then it would match the one below.


PETSc does it so

^ y direction
|
6 7 8
3 4 5
0 1 2

-> x direction




Code and makefile attached ... hopefully within the message size limit.
Just make cartcommtest.

Sean