[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Cray and PETSc

Hello all,

First of all, I am NOT a PETSc developer, but as a member of the DoD HPC Modernization Program I DO install PETSc on a wide variety of hardware platforms (including XT3, XD1, and even X1 [that was rough; had to cross-compile on one machine and transfer files]), and have had numerous email exchanges working out all of the little quirks involved in installing PETSc on these machines, so the PETSc folks have been kind enough to let me watch this development email list.

I hope I'm not over-stepping my bounds by asking this, but I have two comments:

1)  Making PETSc available on the Cray machines sounds like a GREAT idea

2) Several of our users also like to have packages such as HYPRE and MUMPS, etc... installed along with PETSc. Would such external packages also be installed by Cray, or would users requiring a combination of, say, PETSc and MUMPS still have to make their own installation?

Thanks,

	-Tom Cortese
	PET Computational Environnment Onsite

On Thu, 23 Aug 2007, Adrian Tate wrote:


Hi Barry

Thanks for your reply.

We will expect to provide the new full releases of PETSc as soon as we possibly can. Obviously there is something of a lag because Cray need to test, package and release the library, but we'll try to do so with minimal delay. Obviously any heads-up you can give us in advance will help reduce the lag. With respect to patch versions, it is going to be very difficult for Cray to release every new version, and by looking at the logs it seems that many of the patches would not be relevant to a Cray specific build. We will endeavor to monitor all patch releases and will go with an unscheduled new release of PETSc when we see a relevant patch. One thing that we can do to minimize confusion is to informally provide the most currently patched version to any customer who is experiencing a problem, although we might not go with a full release of the version in question.

Regarding Cray modifications, we (Keita Teranishi or myself) will check in periodically with you to show you our improvements and to see which, if any need to be channeled back into the PETSc build.

I hope this support structure works OK from your perspective and I look forward to working with the PETSc team.

Thanks!
Adrian

-----Original Message-----
From: Barry Smith [mailto:bsmith@xxxxxxxxxxx]
Sent: Monday, August 20, 2007 11:12 AM
To: Adrian Tate
Cc: petsc-developers@xxxxxxxxxxx; petsc-dev@xxxxxxxxxxx
Subject: Re: Cray and PETSc


  Adrian,

   Thank you for in the inquiry.


On Thu, 16 Aug 2007, Adrian Tate wrote:

Hello Barry

I'm not sure we met in person - I am the lead of the libraries group
at Cray. We decided some time ago that our iterative solver strategy
would be to leverage PETSc, and to hopefully provide some Cray
specific tunings to improve performance of the KSP solvers (largely
through our custom sparse BLAS and some parallel tuning for our
interconnect). I believe that John Lewis has been in communication
with you with respect to the tuning of Sparse BLAS and their
integration with the PETSc build.

We are however, considering packaging and supplying PETSc along with
the scientific library that we provide (libSci). This allows for
better integration of our internal kernels and also it means that we
are no longer requiring users (including benchmakers and internal
users) to build their own PETSc. I see from your online page that
doing so is acceptable as long as we use a copyright switch during
configure. By applying this switch, do we make our PETSc library
unsupported from your perspective?


  The GNU copyright code is tiny; it would not effect the usability
or support of PETSc

We do not expect to be able to
provide anywhere near the degree of support that your team provide,
and I was hoping to supply a pre-built library whose users could still
seek assistance through your normal support channels - is this
realistic?

  Yes.

  The two isses that concern us with pre-packaged versions of PETSc are
1) keeping up to date on our releases. We generally make two releases
a year and much prefer that users use the most recently release.
If they are using an older release it means we are less able to help
them.
2) keeping up to date on our patches. We may make several bug patches
to each release. Users with a pre-packaged version have trouble keeping
up with the source code patches we provide.





Also, I would be interested to know your degree of interest in the
Cray-specific modifications that we make - would you prefer those to
be channeled back into the PETSc library?


 If they involved directly changing PETSc code, we much prefer to get
it channeled back into the master copy of the source code. Makes it
much easier to debug user code. If it is auxiliary code, like a faster
ddot() etc then it is more appropriate to not try to include it.

Any other comments you have
on the way that Cray can contribute to the PETSc project, I would be
very glad to hear.


 PETSc has a variety of "tuning factors" that could theoretically be
set optimally for a particular machine, these range from simple things
like compiler options, to a choice between C and Fortran versions of
the same code (what we call them Fortran kernels), to different loop
unrollings (in the inline.h file), even something like PetscMemzero()
which has five possible forms. Now currently we do not tune these
or even have a test harness for selecting a good tuning. One thing
you could/would like to do, is determine good choices for these
options on your machines. Just as a simple example, on some Linux systems
the basic libraries are just compiled with the GNU compiler, hence the
system memset() is not particularly effective. A version compiled of PETSc
compiled using a "Fortran memset" may be much faster, or Intel provides
its own _intel_fast_memset() which is better. I've seen a few percent
increase in performance of entire nonlinear PDE solver applications
just from using the non-default memset(). [This was specifically on
an Itanium system, but is likely on other configurations also].


  Barry





Regards,


Adrian Tate

---

Technical Lead

Math Software
Cray Inc.

(206) 349 5868