Subject: hpff-task: CCI for Mtg.

From: Rob Schreiber <[email-masked]>
Date: Thu, 4 Jan 1996 11:13:05 -0800

---------------------------------------------------------------------------
[email-masked] is a mailing list for discussion of control-parallel
features in HPF.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------

Group C CCI LIST AS OF JAN 4, 1996
Rob Schreiber

Please, please please read the high priority items -- we will try to 
slay them at the meeting.

CONTENTS

Item    About                       From    Date      Status
 36     Remapping in INDEPENDENT    Larry   10/7/95   Open   HIGH PRIORITY
 37     NEW status for locals       Larry   10/7/95   Open   HIGH PRIORITY
 38     returned array indices      Henry   ??        Open   HIGH PRIORITY
 AA     New for indep loop index    Henry   ??        Open   MEDIUM
 BB     PURE list concat function   Rob/Joel ??       Open   MEDIUM

 18     Defined assign in forall    Henry   10/23/95  Resolved - almost
 29     Extrinsic called in indep   Rob      8/03/95  Resolved

HIGH PRIORITY ITEMS:

=====================================

CCI #36
Independent and remapping
Group C
Larry Meadows
Submitted: 10/7/95
status: new

Question:
In Section 4.4 of the standard, one of the conditions on INDEPENDENT is
that realignment and redistribution cannot occur, since they may change
the processor storing a particular array element.

I would argue that the same reasoning applies to remapping of arguments
in subroutines called inside of INDEPENDENT DO loops. For example:

!hpf$ distribute a(block)
!hpf$ independent
        do i = 1,n
            call sub(a)
        enddo

        subroutine sub(a)
        real a(:)
!hpf$ distribute a(cyclic)
        ...
        return
        end

>From an implementation point of view, remapping of arguments is a
collective operation, just as is realignment or redistribution, so
is difficult to implement inside INDEPENDENT DO loops.

Couple of other points:

- Would be nice to have some examples in 4.4.1 where subroutines were
  called, legally and illegally.

(see cci #37 for other point)
Thanks.
lfm
=====================================
=====================================

CCI #37
Status of locals in procedures called inside Independent DO
Group C
Larry Meadows
Submitted 10/7/95
Status: new
Question:
- I seem to recall that we decided that local variables of subroutines
  called inside INDEPENDENT DO loops were automatically NEW (and I
  assume that this doesn't apply to SAVE or COMMON variables). Is this
  documented anywhere?
Thanks.
lfm

=====================================
=========================

CCI #38 -- Base of coordinates for functions returning array indices

Hello,

     A number of the procedures in the HPF_LOCAL_LIBRARY refer to their
arguments as containing or receiving "coordinates".  It's not clear whether the
values returned or required are supposed to be based on the corresponding lower
bounds of the array (or processor as the case may be) or whether they should be
one-based.

     For example,

           PROGRAM P
             INTERFACE
               EXTRINSIC(HPF_LOCAL) SUBROUTINE SUB(D)
                 INTEGER D(4:)
     !HPF$       PROCESSORS PROC(NUMBER_OF_PROCESSORS())
     !HPF$       DISTRIBUTE D(BLOCK) ONTO PROC
               END SUBROUTINE SUB
             END INTERFACE
             INTEGER A(5:10)

             CALL SUB(A)
           END PROGRAM P

           EXTRINSIC(HPF_LOCAL) SUBROUTINE SUB(D)
             USE HPF_LOCAL_LIBRARY
             INTEGER D(4:), L(1)

             CALL GLOBAL_TO_LOCAL(D, G_INDEX=(/5/), L_INDEX=L)
           END SUBROUTINE SUB

Assuming this program is run on one processor, what should be the value of L
after the call to GLOBAL_TO_LOCAL?  If G_INDEX and L_INDEX are indices that
are based on the actual lower bounds of A and D, respectively, then
G_INDEX=(/5/) would refer to the first element of A, and the corresponding
element of D would be D(4), and so L_INDEX should be (/4/).  If G_INDEX and
L_INDEX are based on one, then G_INDEX=(/5/) would refer to the fifth element
of A (i.e, A(9)), and the corresponding element of D would be D(8), which is
the fifth element of D, and so L_INDEX should be (/5/).

     This shows up in ABSTRACT_TO_PHYSICAL and PHYSICAL_TO_ABSTRACT (the INDEX
argument), LOCAL_TO_GLOBAL and GLOBAL_TO_LOCAL (the G_INDEX and L_INDEX
arguments) and (I think) the results of LOCAL_LINDEX and LOCAL_UINDEX.

     One thing to consider is that the results of GRADE_UP and GRADE_DOWN seem
to be values that can be used to index the array that's been sorted, but the
results of MAXLOC and MINLOC are one-based, so there's a precedent for each.
(Were the results of GRADE_UP and GRADE_DOWN intentionally designed to be
different from those of MAXLOC and MINLOC?)

     It should also be noted that the INDEX argument of ABSTRACT_TO_PHYSICAL
and PHYSICAL_TO_ABSTRACT gives indices in a processor arrangement, but there's
no direct way of finding out the bounds of such a processor arrangement -
HPF_DISTRIBUTION and GLOBAL_DISTRIBUTION will only give you the *shape*.

Thanks,

Henry
------------------

I am sorry to say we did not discuss CCI 38.  It's the one on
index bounds in hpf local library functions like global_to_local, etc?

My inclination, given that we have already got a difference of opinion
between grade_up/down and max/minloc is to choose the interpretation that seems
right, and make that the default (by a vote of 2 to 1).  Are these the only
intrinsic/library functions that return indices?

I think it makes sense to return usable indices, so that in your example,
I would vote for returning "4".

Remind me to discuss this at the next meeting, in January, please

Rob

--------------------
	Hi Rob,

	> I am sorry to say we did not discuss CCI 38.  It's the one on
	> index bounds in hpf local library functions like global_to_local, etc?
	>
	> My inclination, given that we have already got a difference of opinion
	> between grade_up/down and max/minloc is to choose the interpretation that seems
	> right, and make that the default (by a vote of 2 to 1).  Are these the only
	> intrinsic/library functions that return indices?
	>
	> I think it makes sense to return usable indices, so that in your example,
	> I would vote for returning "4".

	    One thing to consider, though, is that GRADE_UP and GRADE_DOWN, as defined
	now, can't be written in standard Fortran 90.  The reason is that assumed-shape
	arrays only assume shape - they can't assume lower bounds.  So if the interface
	for the specific function that performs GRADE_UP for default integers of rank
	one looked like this:

	      FUNCTION I_GRADE_UP(ARRAY, DIM)
	        INTEGER, INTENT(IN) :: ARRAY(:)
	        INTEGER, INTENT(IN), OPTIONAL :: DIM
	        INTEGER :: I_GRADE_UP(SIZE(SHAPE(ARRAY)), PRODUCT(SHAPE(ARRAY)))
	        !
	        !  Code omitted.
	        !
	      END FUNCTION I_GRADE_UP

	the value of LBOUND(ARRAY, 1) would be 1, regardless of what the lower bound of
	the actual argument was.

	     Though it's more usable, it can also present implementation difficulties.
	That's why I wonder whether the current definitions of GRADE_UP and GRADE_DOWN
	are in error.

Hmmm.   They've been implemented, so its possible -- to bad you have to use
extralinguistic means.

Let's discuss this next time -- its on my agenda;

Rob

===================================

MEDIUM PRIORITY ITEMS 

===================================
CCI #AA  --  NEW clause for the loop index of an independent loop?

     One of my coworkers has the following question:

     May the index variable of a DO-loop appear in the NEW clause of an
INDEPENDENT directive on that DO?  For example,

         !HPF$ INDEPENDENT, NEW(I)
               DO I = 1, 10
               END DO

If so, what does it mean?  Does it simply mean that the programmer is
guaranteeing that the value of I is not used after the DO-loop?

Thanks,

Henry
To: [email-masked]
Subject: hpff-interpret: remapping of arguments inside independent
Sender: [email-masked]
Status: RO

===================================
CCI #BB  --  Is this a PURE procedure?

Hi,

Joel and I have a question about functions that produce linked structures
as a result, like a list concatenation function.   Can they be made PURE?

In such a function you really have to copy the data out of the input
lists when computing concat(A,B).

	type node_t
	  real value
	  type (node_t), pointer :: next
	end type

        function concat(a,b)
        type (node_t), pointer :: a, b. concat, head, tail
	integer len

	nullify(concat) 
	if (list_length(a) + list_length(b) == 0) return
	allocate(concat)
        tail => concat
        len = 0

	head => a
	do while (associated(head))
	    if (len > 0) then
	      allocate(tail%next)
	      tail => tail%next
            endif
	    tail%value = head%value
            nullify(tail%next)
	    head => head%next
	    len = len + 1
	end do
	
	head => b
        do while (associated(head))
	    if (len > 0) then
              allocate(tail%next)
              tail => tail%next
            endif
            tail%value = head%value
            nullify(tail%next)
            head => head%next
	    len = len + 1
        end do
	return
	end function concat

Now, ignoring the issue of whether I wrote it correctly, is concat
PURE?  It may be -- except maybe due to  a restriction on the use of
allocate in pure procedures.   Chuck, Henry, do you remember what
action we took on that question, in response to a CCI item from Henry
about allocate causing a status variable to be set by allocate?   I
hope we disallowed only the status, not the allocate!   If we did, I
view the inability to create pure functions that build a linked
structure and return a pointer to it as a serious problem.

Rob

To: [email-masked]
Subject: Re: pure list-oriented fns
Cc: [email-masked]
Status: RO

	From [email-masked] Thu Nov  9 16:31:57 1995
	Subject: Re: pure list-oriented fns
	To: [email-masked] (Rob Schreiber)
	Cc: [email-masked]
	X-Mailer: ELM [version 2.4 PL24alpha3]
	Content-Type: text
	Content-Length: 822

	> Here is a concat function -- pure.
	>
	>         pure function concat(a,b)
	>         type(node), pointer :: a, b, concat
	>
	>         lenresult = listlen(a) + listlen(b)
	>         if (lenresult == 0) then
	>             nullify(concat)
	>             return
	>         else
	>           allocate(concat)
	>             if (listlen(a) == 0) then
	>                 concat%val = b%val
	>                 concat%next = concat(0, b%next)
	>             else
	>                 concat%val = a%val
	>                 concat%next => concat(a%next, b)
	>           endif
	>       endif
	>       end

	     Except that a subobject of a dummy argument of a pure function can't be
	associated with a dummy argument with the POINTER attribute.  So the calls to
	listlen and concat are invalid.  (Did you mention last week that that was still
	a problem?)

Curses; the designers of PURE function fiendishly contrived to thwart me!

I mentioned last week that I thought recursion was a solution; you have
now set me straight.  Lets put this on the agenda, too.  The F90 boys
should be interested in this one.

Rob

=================================

LOW PRIORITY ITEMS

=================================

CCI # 18
Defined assignment in FORALL
Group C
Submitted: 05/04/95
Updated: 10/23/95
Submitted by: Henry Zongaro
Status: in progress

Maybe Resolved.  CHK will provide clarification in HPF 2 document.    It will be a change to the "sequentialization" of the forall to account for defined  assignments.   BUT Henry suggests alternative based on new X3J3 action.

Original question:
     I was wondering whether there's not a problem with allowing defined assignment to appear within a FORALL.  Consider the following example.

     module mod
       integer :: a(3) = (/1,2,3/)
     contains
       pure subroutine def_assign(lhs, rhs)
         integer, intent(inout) :: lhs
         character, intent(in) :: rhs

         lhs = a(ichar(rhs)+1)
       end subroutine def_assign
     end module mod

     program p
       use mod
       interface assignment(=)
         module procedure def_assign
       end interface

     forall (i = 1:2) a(i) = char(i)  ! Sneaky way of passing 
                                      ! "i" to def_assign
    end program p

     The rules of forall specify that the right-hand side and the
indices of the left-hand side are evaluated, in any order, prior to
assignment, which also takes place in any order.  In the above example,
we have

          a(1) = char(1)
          a(2) = char(2)

as the two defined assignments which take place.  Inside of def_assign,
there's a host-associated reference to a, so what ends up happening is
the following:

          a(1) = a(2)
          a(2) = a(3)

     The order in which these assignments occurs affects the result.
The value of a after the forall statement is executed could be (/2,3,3/)
or (/3,3,3/).

     Basically, the problem is that in defined assignment, completely
evaluating the right-hand side for all active combinations does not
necessarily let the compiler precompute everything which might also
appear on the left-hand side.  Thanks, Henry

DISCUSSION AT MEETING:

y, Henry, Jerry to circulate proposed wording for this definition.

No action from the July meeting was recorded ... was this resolved or not?
-----
Rob's recollection:
Resolved.  CHK will provide clarification in HPF 2 document.   It will be a change to the "sequentialization" of the forall to account for defined assignments.

-----
new note from Henry

    X3J3 is taking a different approach on CCI 18.  They're trying to
prohibit references in the procedure that defines the assignment to the
variable that appears on the left-hand side of the defined assignment.
I believe WG5 will decide on this in their November meeting.

     We might want to pick up whatever they decide.  One advantage of
the restriction is that no extra compiler mechanisms are required for
this somewhat obscure case.

=====================================
CCI #29
Calling hpf_local from independent loop
Group C
Updated: 10/23/95
Rob Schreiber
Submitted: 8/3/95
Status: in progress
QUESTION:
I have two CCI questions.
Question I.  Can an extrinsic(hpf_local) be invoked in an independent loop?
In a Forall?
     Ex:

Forall (1 = 1:10) a(i) = f(i,a(i))

Note that part of the calling sequence, as specified in Ver 1.1, appendix A, is 
"The processors are synchronized.  In other words, all actions that logically precede the call are completed."

It seems clear that when this was written it was tacitly assumed that the call did not occur in an independent loop or forall.

Part 2:  May ony other kind of extrinsic be called in a forall or independent loop?
================
Discussion begins here:     Summary provided by Rob S. 

Status:   Under discussion.   Summary of the discussion in September:

1.  Only and HPF type routine can be PURE.  Thus, only and HPF_LOCAL
routine could be local, pure, and hence called in a forall.

2.  There is no semantic problem with this, or with invocation of any
extrinsic routine in an independent loop.

3.  An example
        real x(N, 100000)
        !hpf$ distribute x(*, block)

        !hpf$ independent
        do i = 1, N
           call extrinsic_fn(x(i,:))
        enddo
The independent loop is a mechanism for spawning N threads per processor, each independent of the others;  the load per thread may be variable.  It would
possibly be useful here to NOT synchronize after each call to the extrinsic
routine!   Is there any semantic reason to force the synchronization?

4.  There are some very important issues in the implementation, with possible
language impacts.

Let us assume an MPI based implementation of HPF calling an extrinsic
local routine that uses MPI for communication.  Because of the unity of
purpose and of hotel between HPF and MPI, it is arguably necessary for
HPFF to make this work cleanly and efficiently.

Issue MPI-1:   Does HPF handle MPI_Init and MPI_Finalize, automatically.

Issue MPI-2:   Are any of the MPI routines PURE?

Issue MPI-3:   Thread safety.   In a naive implementation, HPF does a barrier
before and after the call to the extrinsic;  but there is no guarantee that
there are no outstanding, nonreceived messages in the messaging system.  Thus, to be safe, any extrinsic routine should use its own communicator.   To prevent
interference between separate calls to the routine, a new communicator should 
be created for every call.   An obvious way to do this is to call
      MPI_Comm_Dup(MPI_COMM_WORLD, New_comm)
at the beginning of every such extrinsic.    However, an extrinsic
that consumes all its messages would be justified in doing this once, on its
first invocation, and saving the communicator for reuse on later invocations.
   But consider what happens if the extrinsic is called in an independent DO loop
as in the example above, and there is no barrier used.  Now we really need a
separate communicator per thread. On the other hand, a call to MPI_Comm_Dup is a collective call, which synchronizes the processes.
     Perhaps this should be done by the calling HPF routine, so that the
MPI_COMM_WORLD communicator is different on every call.

Issue MPI-4:  If called from the range of an ON_HOME directive, what
set of processors does MPI_COMM_WORLD correspond to?   If it corresponds to the subset executing the ON block, then how can the called routine access 
nonresident data?   Should there be a way to access a communicator that 
corresponds to these executing processors, while MPI_COMM_WORLD 
always corresponds to all of the processors?

Issue MPI-5:  If called from separate ON_HOME blocks in the scope of
a TASK directive, with disjoint processors groups, so that the two
ON blocks may be executed concurrently, what communicators correspond to the two processor groups?   (If, in issue 4 above, the answer is that MPI_COMM_WORLD corresponds to the executing subset of the processors, then the answer here is MPI_COMM_WORLD.)

=========== comments from Jim Cownie =================

> Issue MPI-1:  Does HPF handle MPI_Init and MPI_Finalize,
automatically.  I would say that the HPF run-time should have called
MPI_Init before any user code has run, therefore user extrinsic
functions which need MPI can just use it.

This is actually not a big deal, since the user routine can always do
use MPI_Initialized() to guard her call to MPI_Init.  (Though if this is
done, then the HPF run-time needs to do the same, since MPI_Init should
only be called once.)  That's why it's simpler to say that MPI_Init has
already been called before any user HPF code has run.

> Issue MPI-2:   Are any of the MPI routines PURE?
Probably. For instance one could cast reductions as functions which
return the result, and only read the arguments (though why you'd want
to use an MPI reduction extrinsically rather than an HPF one is beyond
me).
 ... after all 5 MPI issues ...

     I would suggest that in all of these cases the HPF run-time should
provide a "current communicator" which includes the set of processes
running the current construct.  In some cases this will be
MPI_COMM_WORLD (or a Comm_Dup of COMM_WORLD), in others (ON_HOME, task
parallelism, processor subsets) it will represent a subset of the
available processes.  In MPI MPI_COMM_WORLD is always available as the
set of all processes (until MPI-2 introduces a dynamic process model,
though that shouldn't worry HPF implementations).

     Therefore I think
1) MPI_COMM_WORLD is *always* the set of all processes. (This is the
   current MPI view).
2) If you need subsets then you should create new communicators, and
   provide a way for the user code to access them.

MPI_COMM_WORLD should mean the same thing in a routine called from HPF
extrinsic as it did in a "raw" MPI program.  The HPF extrinsic MPI
environment should contain additions to the raw MPI environment (new
communicators, maybe pre-defined datatypes giving array distributions,
etc), but should not change the meaning of things in the raw MPI world.
In other words you may need to learn more to work in the HPF extrinsic
environment, but you shouldn't have to unlearn things you already knew
about MPI.

-- Jim

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------