[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Re:


On Jan 12, 2007, at 1:51 PM, Matthew Knepley wrote:

Why are you creating that ISGeneral with PETSC_COMM_WORLD. It does not
seem to be global? Shouldn't it be PETSC_COMM_SELF? Also, what are you
trying to do?

Thanks, that's a very good call, it works much better with PETSC_COMM_SELF, because then I don't have any MPI calls.

What I was trying to do is basically determine the nonzero elements of a matrix row by row and then use this information to get exactly those entries from a different matrix. The actual code was too long and I just included everything that was needed for a crash.

Thanks for your help,
Arvid


 Matt

On 1/12/07, Arvid Bessen <bessen@xxxxxxxxxxxxxxx> wrote: 

Dear all,

I have a problem with MPI in PETSc that is probably simple and you
will hopefully know the answer to. In my code I treat the local rows
of a MPIAIJ matrix one by one. If the number of local rows is equal
on all processors, everything works well. However, if the number of
rows is different, the program crashes. I have attached a simple
example program to demonstrate what I want to do.
Suppose process A has one more local row than process B. In the
debugger I found out that the crash occurs in "MPI_Allreduce".
Process A will call MPI_Allreduce one more time than process B,
because it processes an additional row. While process A is still
waiting for the result of the last MPI_Allreduce, process B will
proceed and call MPI_Allreduce in a different context. This is, when
the program crashes.

What is the best way to make sure that all MPI calls are matched on
the different processes?

Thanks,
Arvid

static char help[] = "Example program with different row sizes\n\n";

#include "petscmat.h"

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int Argc,char **Args)
{
   const PetscInt  size = 3;
   PetscErrorCode  ierr;
   Mat             cmat, dmat;
   PetscInt        cmat_loc_start, cmat_loc_end, cmat_loc_size;
   PetscInt        *rowsize_arr = 0;
   const PetscInt  **row_arr = 0;
   IS              *is_arr = 0;
   PetscInt        a;
   Mat            *submat_arr = 0;

   PetscInitialize(&Argc,&Args,(char *)0,help);

ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE,
PETSC_DECIDE, size, size,
1, PETSC_NULL, 2, PETSC_NULL, &cmat);CHKERRQ(ierr);
ierr = MatAssemblyBegin(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
ierr = MatAssemblyEnd(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE,
PETSC_DECIDE, size, size,
1, PETSC_NULL, 2, PETSC_NULL, &dmat);CHKERRQ(ierr);
ierr = MatAssemblyBegin(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
ierr = MatAssemblyEnd(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);

   ierr = MatGetOwnershipRange(cmat, &cmat_loc_start, &cmat_loc_end);
CHKERRQ(ierr);
   cmat_loc_size = cmat_loc_end-cmat_loc_start;

ierr = PetscMalloc(sizeof(PetscInt *)*cmat_loc_size, &row_arr);
CHKERRQ(ierr);
ierr = PetscMalloc(sizeof(PetscInt)*cmat_loc_size, &rowsize_arr);
CHKERRQ(ierr);
ierr = PetscMalloc(sizeof(IS)*cmat_loc_size, &is_arr); CHKERRQ (ierr);

for( a=0; a<cmat_loc_size; a++ ) {
ierr = MatGetRow(cmat, a+cmat_loc_start, &(rowsize_arr[a]), &
(row_arr[a]), 0); CHKERRQ(ierr);
ierr = ISCreateGeneral(PETSC_COMM_WORLD, rowsize_arr[a], row_arr
[a], &(is_arr[a])); CHKERRQ(ierr);
ierr = MatRestoreRow(cmat, a+cmat_loc_start, &(rowsize_arr [a]), &
(row_arr[a]), 0); CHKERRQ(ierr);
}

ierr = PetscMalloc(sizeof(Mat)*cmat_loc_size, &submat_arr); CHKERRQ
(ierr);
ierr = MatGetSubMatrices(dmat, cmat_loc_size, is_arr, is_arr,
MAT_INITIAL_MATRIX, &submat_arr); CHKERRQ(ierr);

   ierr = PetscFinalize();CHKERRQ(ierr);
   return 0;
}





--
One trouble is that despite this system, anyone who reads journals widely
and critically is forced to realize that there are scarcely any bars to eventual
publication. There seems to be no study too fragmented, no hypothesis too
trivial, no literature citation too biased or too egotistical, no design too
warped, no methodology too bungled, no presentation of results too
inaccurate, too obscure, and too contradictory, no analysis too self-serving,
no argument too circular, no conclusions too trifling or too unjustified, and
no grammar and syntax too offensive for a paper to end up in print. --
Drummond Rennie