Subject: hpff-task: CCI's

From: Rob Schreiber <[email-masked]>
Date: Mon, 9 Oct 1995 14:51:25 -0700

---------------------------------------------------------------------------
[email-masked] is a mailing list for discussion of control-parallel
features in HPF.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------

Here is my understanding of the four CCI items on our agenda.
Please correct:

Group C, CCI status.

Number 18.   Source:  Zongaro   date 5-4-95
Subject:  Defined assignment in Forall
Status:

Resolved.  CHK will provide clarification in HPF 2 document.   It will be
achange to the "sequentializatio" of the forall to account for defined
assignments.

Number 19:  Source:   Rosing    date 6-27-95
Subject:  independent Do; implementation questions

Status:  Resolved.   No unanticpated or controversial questions about the
semantics of HPF were raised, and we do not have much to say about the
implementaion issues he raised.

Number 29:   Source:  Schreiber    date 8-3-95
Subject:   extrinsic local routines called in forall/independent

Status:   Under discussion.   Summary of the discussion in September:

1.  Only and HPF type routine can be PURE.  Thus, only and HPF_LOCAL
routine could be local, pure, and hence called in a forall.

2.  There is no semantic problem with this, or with invocation of any
extrinsic routine in an independent loop.

3.  An example

	real x(N, 100000)
	!hpf$ distribute x(*, block)

	!hpf$ independent
	do i = 1, N
           call extrinsic_fn(x(i,:))
	enddo

The independent loop is a mechanism for spawning N threads per processor, each
independent of the others;  the load per thread may be variable.  It would
possibly be useful here to NOT synchronize after each call to the extrinsic
routine!   Is there any semantic reason to force the synchronization?

4.  There are some very important issues in the implementation, with possible
language impacts.

Let us assume an MPI based implementation of HPF calling an extrinsic
local routine that uses MPI for communication.  Because of the unity of
purpose and of hotel between HPF and MPI, it is arguably necessary for
HPFF to make this work cleanly and efficiently.

Issue MPI-1:   Does HPF handle MPI_Init and MPI_Finalize, automatically.

Issue MPI-2:   Are any of the MPI routines PURE?

Issue MPI-3:   Thread safety.   In a naive implementation, HPF does a barrier
before and after the call to the extrinsic;  but there is no guarantee that
there are no outstanding, nonreceived messages in the messaging system.  Thus, to
be safe, any extrinsic routine should use its own communicator.   To prevent
interference between separate calls to the routine, a new communicator should be
created for every call.

An obvious way to do this is to call

      MPI_Comm_Dup(MPI_COMM_WORLD, New_comm)

at the beginning of every such extrinsic.    However, an extrinsic
that consumes all its messages would be justified in doing this once, on its
first invocation, and saving the communicator for reuse on later invocations.

But consider what happens if the extrinsic is called in an independent DO loop
as in the example above, and there is no barrier used.  Now we really need a
separate communicator per thread.
On the other hand, a call to MPI_Comm_Dup is a collective call, which
synchronizes the processes.

Perhaps this should be done by the calling HPF routine, so that the
MPI_COMM_WORLD communicator is different on every call.

Issue MPI-4:  If called from the range of an ON_HOME directive, what
set of processors does MPI_COMM_WORLD correspond to?   If it corresponds to the
subset executing the ON block, then how can the called routine access nonresident
data?   Should there be a way to access a communicator that corresponds to these
executing processors, while MPI_COMM_WORLD always corresponds to all of the
processors?

Issue MPI-5:  If called from separate ON_HOME blocks in the scope of
a TASK directive, with disjoint processors groups, so that the two
ON blocks may be executed concurrently, what communicators correspond to the
two processor groups?   (If, in issue 4 above, the answer is that MPI_COMM_WORLD
corresponds to the executing subset of the processors, then the answer
here is MPI_COMM_WORLD.)

Number ????     Source:   Meadows     date:   10-7-95
Subject:   Remapping at subprogram interface in INDEPENDENT

Status:   To be discussed.

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------

---

From: "James Cownie <[email-masked]>" <[email-masked]>
Date: Tue, 10 Oct 1995 09:59:06 +0100

A few comments on Rob's MPI related issues

> Issue MPI-1:   Does HPF handle MPI_Init and MPI_Finalize, automatically.
I would say that the HPF run-time should have called MPI_Init before
any user code has run, therefore user extrinsic functions which need
MPI can just use it.

This is actually not a big deal, since the user routine can always do
use MPI_Initialized() to guard her call to MPI_Init. (Though if this
is done, then the HPF run-time needs to do the same, since MPI_Init
should only be called once.) That's why it's simpler to say that
MPI_Init has already been called before any user HPF code has run.
	
> Issue MPI-2:   Are any of the MPI routines PURE?
Probably. For instance one could cast reductions as functions which
return the result, and only read the arguments (though why you'd want
to use an MPI reduction extrinsically rather than an HPF one is beyond
me).

> Issue MPI-3: Thread safety.  In a naive implementation, HPF does a
> barrier before and after the call to the extrinsic; but there is no
> guarantee that there are no outstanding, nonreceived messages in the
> messaging system.  Thus, to be safe, any extrinsic routine should use
> its own communicator.  To prevent interference between separate calls
> to the routine, a new communicator should be created for every call.
> 
> at the beginning of every such extrinsic.  However, an extrinsic that
> consumes all its messages would be justified in doing this once, on
> its first invocation, and saving the communicator for reuse on later
> invocations.
> 
> But consider what happens if the extrinsic is called in an independent
> DO loop as in the example above, and there is no barrier used.  Now we
> really need a separate communicator per thread.
> 

> corresponds to the executing subset of the processors, then the answer

I would suggest that in all of these cases the HPF run-time should
provide a "current communicator" which includes the set of processes
running the current construct. In some cases this will be
MPI_COMM_WORLD (or a Comm_Dup of COMM_WORLD), in others (ON_HOME, task
parallelism, processor subsets) it will represent a subset of the
available processes. In MPI MPI_COMM_WORLD is always available as the
set of all processes (until MPI-2 introduces a dynamic process model,
though that shouldn't worry HPF implementations).

Therefore I think
1) MPI_COMM_WORLD is *always* the set of all processes. (This is the
   current MPI view).
2) If you need subsets then you should create new communicators, and
   provide a way for the user code to access them. 

MPI_COMM_WORLD should mean the same thing in a routine called from HPF
extrinsic as it did in a "raw" MPI program. The HPF extrinsic MPI
environment should contain additions to the raw MPI environment (new
communicators, maybe pre-defined datatypes giving array distributions,
etc), but should not change the meaning of things in the raw MPI
world. In other words you may need to learn more to work in the HPF
extrinsic environment, but you shouldn't have to unlearn things you
already knew about MPI.

-- Jim 

James Cownie 
BBN UK Ltd
Phone : +44 117 9071438
E-Mail: [email-masked]