Question on matrix preallocation

[Tobias Neckel <neckel@xxxxxxxxx>]

Hello,

when using petsc (version 2.3.2 on a linux 32bit Intel architecture) to 
set up a serial sparse linear system of equations, I recently noticed 
the well-known allocation performance problem: The matrix setup needs 
more memory than preallocated with a fixed number of column entries for 
all rows.
Thus, I switched to the strategy described in the Users Manual (first 
counting the number of matrix entries for each row individually and then 
 using the nnz parameter in MatCreateSeqAIJ()). But this did not change 
the dynamic allocation behaviour at all.

Therefore, I tried to bring everything down to a (very) small test 
example. I set up a nnz-1D-array of type int and length 4 which holds 
the number of expected non-zero column entries for each row of a matrix 
(in particular 2 columns in row 0). Using this nnz-array, I create a 4x4 
matrix. Afterwards, I set the entry (0,0) of the matrix to a non-zero 
value.
The source code part for this simple test can be found in the attached 
file testMatPreallocation.cpp.

When I run this test (with the additional -info runtime option), the one 
and only matrix entry setting results in an additional memory allocation
(see attached file commandLineOutput.txt)!

This is quite surprising, as I would have expected enough preallocated 
memory for the matrix, which is also visible from the output. Am I 
misusing or missing something necessary to make the preallocation work?

Thanks in advance for any hints,
best regards
Tobias Neckel

--
Dipl.-Tech. Math. Tobias Neckel

Institut für Informatik V, TU München
Boltzmannstr. 3, 85748 Garching

Tel.:   089/289-18602
Email:  neckel@xxxxxxxxx
URL:    http://www5.in.tum.de/persons/neckel.html

 14:33:53 debug    petsc::PETScLibTest::testMatPreallocation()             start PETSc mat preallocation test
[0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4 X 4; storage space: 10 unneeded,0 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
[0] Mat_CheckInode(): Found 1 nodes of 4. Limit used: 5. Using Inode routines
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4 X 4; storage space: 14 unneeded,1 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 1
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] Mat_CheckInode(): Found 2 nodes of 4. Limit used: 5. Using Inode routines
 14:33:53 debug    petsc::PETScLibTest::testMatPreallocation()             stop PETSc mat preallocation test

#undef __FUNCT__
#define __FUNCT__ "petsc::PETScLibTest::testMatPreallocation"
void petsc::PETScLibTest::testMatPreallocation(){
  
  _log.debug("testMatPreallocation()","start PETSc mat preallocation test");
  
  Mat matrix;
  const int rows = 4;
  const int cols = 4;
  
  int nnzCounter[rows];
  for (int i=0; i<rows; i++) {
    nnzCounter[i] = 0;
  }
  nnzCounter[0] = 2;
  nnzCounter[1] = 4;
  nnzCounter[2] = 3;
  nnzCounter[3] = 1;
  
  _ierr = MatCreateSeqAIJ(PETSC_COMM_WORLD, rows, cols, 
                            PETSC_NULL, nnzCounter, &matrix);
  PETSc_CHKERRQ(_ierr); 
  
  MatInfo info;
  _ierr = MatGetInfo(matrix,MAT_GLOBAL_SUM,&info);
  PETSc_CHKERRQ(_ierr);
  int expectedNonzerosUsed = 0;
  int expectedNonzerosAllocated = 10;
  int expectedNonzerosUnneeded = expectedNonzerosAllocated - expectedNonzerosUsed;
  validateEquals( (PetscInt)info.nz_used,      expectedNonzerosUsed,      "testMatPreallocation()");
  validateEquals( (PetscInt)info.nz_allocated, expectedNonzerosAllocated, "testMatPreallocation()");
  validateEquals( (PetscInt)info.nz_unneeded,  expectedNonzerosUnneeded,  "testMatPreallocation()");
  
  _ierr = MatAssemblyBegin(matrix,MAT_FINAL_ASSEMBLY);
  PETSc_CHKERRQ(_ierr);
  _ierr = MatAssemblyEnd(matrix,MAT_FINAL_ASSEMBLY);  
  PETSc_CHKERRQ(_ierr);
  
  
  double dummyValue = 1.5;
  int row = 0;
  int col = 0;
  _ierr = MatSetValue(matrix, row, col, dummyValue, ADD_VALUES); 
  PETSc_CHKERRQ(_ierr);
  
  _ierr = MatAssemblyBegin(matrix,MAT_FINAL_ASSEMBLY);
  PETSc_CHKERRQ(_ierr);
  _ierr = MatAssemblyEnd(matrix,MAT_FINAL_ASSEMBLY);  
  PETSc_CHKERRQ(_ierr);
  
  _log.debug("testMatPreallocation()","stop PETSc mat preallocation test");
}