Subject: hpff-core: active cci

From: Mary E Zosel <[email-masked]>
Date: Fri, 5 Jan 1996 13:17:29 -0800

---------------------------------------------------------------------------
[email-masked] is a mailing list for announcements related to High
Performance Fortran.  Instructions for adding or deleting yourself
from this list appear at the bottom of this message.
---------------------------------------------------------------------------
TO HPFF-CORE
Here is a text copy of the active CCI's for next week's meeting. They are sorted
by group, then by item #. I think there are 9 total.  Some are essential already
answered and just need verification that the answer is complete.
Almost all are for group C.  
    -mary-

_____________________________________________________
_____________________________________________________

CCI #18	Defined assignment in FORALL	Group C	Updated: 10/23/95

Henry Zongaro	05/04/95		

Current reply:
	Maybe Resolved.  CHK will provide clarification in HPF 2 document.   It will be a change to the "sequentializatio" of the forall to account for defined assignments.   BUT Henry suggests alternative based on new X3J3 action.
----------
Original question:

Hello,

     I was wondering whether there's not a problem with allowing defined
assignment to appear within a FORALL.  Consider the following example.

         module mod
           integer :: a(3) = (/1,2,3/)
         contains
           pure subroutine def_assign(lhs, rhs)
             integer, intent(inout) :: lhs
             character, intent(in) :: rhs

             lhs = a(ichar(rhs)+1)
           end subroutine def_assign
         end module mod

         program p
           use mod
           interface assignment(=)
             module procedure def_assign
           end interface

           forall (i = 1:2) a(i) = char(i)    ! A sneaky way of passing "i"
                                              ! to def_assign
         end program p

     The rules of forall specify that the right-hand side and the indices of
the left-hand side are evaluated, in any order, prior to assignment, which also
takes place in any order.  In the above example, we have

          a(1) = char(1)
          a(2) = char(2)

as the two defined assignments which take place.  Inside of def_assign, there's
a host-associated reference to a, so what ends up happening is the following:

          a(1) = a(2)
          a(2) = a(3)

     The order in which these assignments occurs affects the result.  The value
of a after the forall statement is executed could be (/2,3,3/) or (/3,3,3/).

     Basically, the problem is that in defined assignment, completely
evaluating the right-hand side for all active combinations does not necessarily
let the compiler precompute everything which might also appear on the left-hand
side.

Thanks,

Henry
------------
	DISCUSSION AT MEETING:

y, Henry, Jerry to circulate proposed wording for this definition.

----
No action from the July meeting was recorded ... was this resolved or not?
---------
Rob's recollection:
Resolved.  CHK will provide clarification in HPF 2 document.   It will be a change to the "sequentializatio" of the forall to account for defined assignments.
------------
Note from Henry

    X3J3 is taking a different approach on CCI 18.  They're trying to prohibit
 references in the procedure that defines the assignment to the variable that
 appears on the left-hand side of the defined assignment.  I believe WG5 will
 decide on this in their November meeting.

     We might want to pick up whatever they decide.  One advantage of the
estriction is that no extra compiler mechanisms are required for this somewhat
obscure case.
_____________________________________________________
_____________________________________________________

CCI #36	Independent and remapping	Group C
Larry Meadows	10/7/95	status: new	

In Section 4.4 of the standard, one of the conditions on INDEPENDENT is
that realignment and redistribution cannot occur, since they may change
the processor storing a particular array element.

I would argue that the same reasoning applies to remapping of arguments
in subroutines called inside of INDEPENDENT DO loops. For example:

!hpf$ distribute a(block)
!hpf$ indepedent
        do i = 1,n
            call sub(a)
        enddo

        subroutine sub(a)
        real a(:)
!hpf$ distribute a(cyclic)
        ...
        return
        end

>From an implementation point of view, remapping of arguments is a
collective operation, just as is realignment or redistribution, so
is difficult to implement inside INDEPENDENT DO loops.

Couple of other points:

- Would be nice to have some examples in 4.4.1 where subroutines were
  called, legally and illegally.

(see cci #37 for other point)

Thanks.

lfm

_____________________________________________________
_____________________________________________________

CCI #37	status of locals in procedures called inside Independent DO
Group C	
Larry Meadows	10/7/95	status: new	

----------original question---------
(continued item from #36 - but separate question)

Couple of other points:

- I seem to recall that we decided that local variables of subroutines
  called inside INDEPENDENT DO loops were automatically NEW (and I
  assume that this doesn't apply to SAVE or COMMON variables). Is this
  documented anywhere?

Thanks.

lfm

_____________________________________________________
_____________________________________________________

CCI #38	Library Coordinates	Group C	
Henry Zongaro	10/30/95	status: new	

---------- original question -------------
     A number of the procedures in the HPF_LOCAL_LIBRARY refer to their
arguments as containing or receiving "coordinates".  It's not clear whether the
values returned or required are supposed to be based on the corresponding lower
bounds of the array (or processor as the case may be) or whether they should be
one-based.
     For example,
           PROGRAM P
             INTERFACE
               EXTRINSIC(HPF_LOCAL) SUBROUTINE SUB(D)
                 INTEGER D(4:)
     !HPF$       PROCESSORS PROC(NUMBER_OF_PROCESSORS())
     !HPF$       DISTRIBUTE D(BLOCK) ONTO PROC
               END SUBROUTINE SUB
             END INTERFACE
             INTEGER A(5:10)

             CALL SUB(A)
           END PROGRAM P

           EXTRINSIC(HPF_LOCAL) SUBROUTINE SUB(D)
             USE HPF_LOCAL_LIBRARY
             INTEGER D(4:), L(1)

             CALL GLOBAL_TO_LOCAL(D, G_INDEX=(/5/), L_INDEX=L)
           END SUBROUTINE SUB
Assuming this program is run on one processor, what should be the value of L
after the call to GLOBAL_TO_LOCAL?  If G_INDEX and L_INDEX are indices that
are based on the actual lower bounds of A and D, respectively, then
G_INDEX=(/5/) would refer to the first element of A, and the corresponding
element of D would be D(4), and so L_INDEX should be (/4/).  If G_INDEX and
L_INDEX are based on one, then G_INDEX=(/5/) would refer to the fifth element
of A (i.e, A(9)), and the corresponding element of D would be D(8), which is
the fifth element of D, and so L_INDEX should be (/5/).

     This shows up in ABSTRACT_TO_PHYSICAL and PHYSICAL_TO_ABSTRACT (the INDEX
argument), LOCAL_TO_GLOBAL and GLOBAL_TO_LOCAL (the G_INDEX and L_INDEX
arguments) and (I think) the results of LOCAL_LINDEX and LOCAL_UINDEX.

     One thing to consider is that the results of GRADE_UP and GRADE_DOWN seem
to be values that can be used to index the array that's been sorted, but the
results of MAXLOC and MINLOC are one-based, so there's a precedent for each.
(Were the results of GRADE_UP and GRADE_DOWN intentionally designed to be
different from those of MAXLOC and MINLOC?)

     It should also be noted that the INDEX argument of ABSTRACT_TO_PHYSICAL
and PHYSICAL_TO_ABSTRACT gives indices in a processor arrangement, but there's
no direct way of finding out the bounds of such a processor arrangement -
HPF_DISTRIBUTION and GLOBAL_DISTRIBUTION will only give you the *shape*.
Thanks,
Henry

_____________________________________________________
_____________________________________________________

CCI #40	loop index and new	Group C	
Henry Zongaro	11/30/95	status: new	

--------original question -------
Hello,

     One of my coworkers has the following question:

     May the index variable of a DO-loop appear in the NEW clause of an
INDEPENDENT directive on that DO?  For example,

         !HPF$ INDEPENDENT, NEW(I)
               DO I = 1, 10
               END DO

If so, what does it mean?  Does it simply mean that the programmer is
guaranteeing that the value of I is not used after the DO-loop?

Thanks,

Henry

_____________________________________________________
_____________________________________________________

CCI #41	interaction of distribute and forall	Group C	
Norton	12/21/95	status: new

--------- original question ---------
Given a data layout:

      real, dimension(25) :: a,b,c
!HPF$ DISTRIBUTE a (*)
!HPF$ DISTRIBUTE b (*)
!HPF$ DISTRIBUTE c (*)

Is the following legal syntax and could an implementor choose to
compute this in parallel even though the arrays have been declared
"serial"?

!HPF$ INDEPENDENT
      do i=1,25
         a(i) = b(i) + c(i)
      enddo

[The motivation for the inquiry - it would be nice if the INDEPENDENT
directive would be honored over the DISTRIBUTE directive to
parallelize execution on an SMP type system if the cost of data motion
on the bus is cheap, but the same code would honor the DISTRIBUTE
directive over the INDEPENDENT directive to run code serially on a
distributed memory system where the cost of data motion over the
network may be expensive.  Realizing that that is a bit much to ask of
implementors, it would be nice to know if there are any thoughts on
a defined behavior for this combination of directives.]

thanks
-dave

-------- DISCUSSION ---------
Chuck has already sent an answer --- group should verify -  if there is any answer, or any action implied.
============= chuck's reply ===========

The quick answer:

Yes, the syntax is legal.
Yes, this is a correct use of INDEPENDENT.
Yes, a compiler could implement INDEPENDENT as you suggest.
No, it is not written in stone that the compiler must implement INDEPENDENT
this way.

A bit of philosophy:

INDEPENDENT makes a statement about side effects of loop iterations (i.e.
it states that there aren't any side effects on other iterations).  There
is language in the specification to the effect that "The compiler can use
this information in any way that it sees fit."  With that in mind, I hope
it's clear that INDEPENDENT and DISTRIBUTE are basically orthogonal
constructs; you can mix and match them as needed.

The implementation you suggest is (IMHO) an excellent compiler heuristic
for handling INDEPENDENT.  I hope that SMP vendors are listening to you.
However, HPF bends over backwards to keep from requiring any particular
implementation.  (Similarly, you won't find many implementation hints in
the F90 standard, or most other language definitions.)  In part, this is
because HPF is supposed to run on many platforms which may need different
implementations.  In part, this is because we don't want HPF to be locked
into an old implementation technique if new research comes along.  And, of
course, it's hard enough to get agreement on a language without debating
"should this be implemented the APR way or the PGI way?"

So, HPF doesn't define a "behavior" for INDEPENDENT, in the sense that it
is legal for a given compiler to ignore the directive (in some cases, or in
all cases).  Or the compiler can use the INDEPENDENT information in ways
you haven't thought of (for example, INDEPENDENT implies you can turn off
cache coherency protocols for data assigned in the loop, if you turn
coherence back on at the right time).  I argue this is what you want for
portable programs, where each platform can treat INDEPENDENT the way that
it needs to.

If you really want/need to define a program's behavior, you may want to
look at the ON HOME directive being discussed in HPF 2.  (I promise, the
revised proposal is coming!)

                                                Chuck

_____________________________________________________
_____________________________________________________

CCI #42	table look-up	Group C	
Norton	1/4/96	status: new	
----------original questions ----------
I'm having a bit of trouble understanding how to do table lookups. Using some information from The Book, the only thing I have been able to come up with is to go to local space to do the actual lookup (under the requirement of only one table/processor):

      program foo
      real, dimension(137,NUMBER_OF_PROCESSORS) :: table
      real, dimension(1024,1024) :: AA, BB

!HPF$ DISTRIBUTE (*,BLOCK(1)) :: table
!HPF$ DISTRIBUTE (BLOCK,BLOCK) :: AA, BB

      interface
         EXTRINSIC(HPF_LOCAL) subroutine lookup (AA,BB,table)
            real, dimension(:,:), intent(INOUT) :: AA
            real, dimension(:,:), intent(IN) :: BB
            real, dimension(:,:), intent(IN)  :: table
!HPF$       DISTRIBUTE (:,:) :: AA,BB,table
         end subroutine lookup
      end interface

      data table/........./
      call lookup (AA,BB,table)
      stop
      end
      EXTRINSIC(HPF_LOCAL) subroutine lookup (AA,BB,table)

            real, dimension(:,:), intent(INOUT) :: AA
            real, dimension(:,:), intent(IN) :: BB
            real, dimension(:,:), intent(IN)  :: table
!HPF$ DISTRIBUTE (:,:) :: AA,BB,table

      integer i,j,k
!HPF$ INDEPENDENT
      do i=lbound(AA,1),ubound(AA,1)            ! 1,ubound(AA,1)
                                                ! should be enough
        do j=lbound(AA,2), ubound(AA,2)
          do k=lbound(table,2),ubound(table,2)  ! hopefully this == k=1,1
                                                ! so it's just here
                                                ! for fun
             AA(i,j) = AA(i,j) + table(BB(i,j),k)
          enddo
        enddo
      enddo

      return
      end
If this (or something close to this - I am temporarily without an HPF compiler) works, under HPF, great.  However, I would really like to find out if there is a way to do the table lookups - one
table/processor - that will also compile under F90 on a single CPU system.  The above example has an HPF library function (NUMBER_OF_PROCESSORS) and the HPF EXTRINSIC syntax.

What might be a nice solution to code this for both HPF and F90 (if it were legal syntax) would be something like:

      program foo
      integer i,j
      real, dimension(137) :: table

!  I don't think this next statement is legal syntax
!  but it would be nice if it were - treated as a comment under
!  F90, but used under HPF.  Yes - I could use something like the
!  -dlines flag to a compiler or cpp, but that depends on the
!  compiler, rather then the language
!HPF$ real, dimension(137,NUMBER_OF_PROCESSORS) :: table_align

      real, dimension(1024,1024) :: AA, BB

!HPF$ DISTRIBUTE (*,BLOCK(1)) :: table_align

!HPF$ ALIGN table(i) WITH table_align(i,*)

!HPF$ DISTRIBUTE (BLOCK,BLOCK) :: AA, BB

! sorry - I can't currently check if this would as intended
!HPF$ INDEPENDENT
      do i=1,1024
        do j=1,1024
          AA(i,j) = AA(i,j) + table(BB(i,j))
        enddo
      enddo
      ......
The short question then is - have I missed some other ways to do table lookups? 

-------- DISCUSSION -----------

ed note --- this is included in cci for completeness --- answers sent are probably sufficient, but subgroup can verify completness, and that there are no additional issues.  
----------chk replies-------
Reading between a few lines, I think this is what you are trying to do:
1. Do many lookup operations in parallel.
2. Use the same (conceptual) table for all the lookups.
   (i.e. the table values are the same on all the lookups)
3. Store a copy of the conceptual table on all processors.
4. Have only one version of the code.

So, I would suggest using ALIGN to replicate the table, and using a
straight INDEPENDENT loop.  Since the replication happens in ALIGN, the
same code should work under both HPF and F90.  I don't have an HPF compiler
handy either, but the following should do it:

     real, dimension(1024,1024) :: AA, BB
      real, dimension(137) :: table
!HPF$ DISTRIBUTE (BLOCK,BLOCK) :: AA, BB
!HPF$ TEMPLATE dummy(NUMBER_OF_PROCESORS())
!HPF$ ALIGN table(i) WITH dummy(*)
!HPF$ DISTRIBUTE dummy(BLOCK)

!HPF$ INDEPENDENT
      do i=1,1024
        do j=1,1024
          AA(i,j) = AA(i,j) + table( truncate(BB(i,j)) )
        enddo
      enddo

I think your second syntax would also work, but this seems a bit simpler.
A good compiler would even do the right thing (i.e. one copy per processor)
if you said

     real, dimension(1024,1024) :: AA, BB
      real, dimension(137) :: table
!HPF$ DISTRIBUTE (BLOCK,BLOCK) :: AA, BB
!HPF$ ALIGN table(i) WITH AA(*,*)

but it's possible that some compilers would create one copy per element of
AA, definitely not the right thing to do.
- about the independent loop:
The loop is definitely dependence-free, so it's correct to say INDEPENDENT.
Most compilers ought to notice this on their own, but you never know.
Whether your current compiler actually executes the INDEPENDENT loop in
parallel is a different question - ask you vendor.
--
   This is really two questions:
1. Are there other ways to do the lookups in parallel?
2. Are there other ways to copy the lookup tables?

In answer to 1., you could get the same effect with INDEPENDENT DO (see
above) and FORALL (with or without an INDEPENDENT).  This might or might
not make a difference in how your compiler handles the program.  The
EXTRINSIC routine you started with works too (assuming all the details are
right, I didn't check), but it seems like overkill- the lookups are
data-parallel, so you'd expect that HPF can handle them well without
resorting to extrinsics.

In answer to 2., ALIGN is the basic way to get replication on all
processors.  Adding a NUMBER_OF_PROCESSORS dimension as you did works, but
risks confusing the compiler (it has to realize which dimension is local).

                                                Chuck
======
adam marshall also replies ... 
Surely you could use a couple of cpp  "#defines" for the F90 version; you
could #define NUMBER_OF_PROCESSORS to be "1" and EXTRINSIC(HPF_LOCAL) to be
a blank space (or no space at all).

Adam Marshall

_____________________________________________________
_____________________________________________________

CCI #29	calling hpf_local from independent loop	Group C & E	10/23/95
Rob Schreiber	8/3/95	status: in progress
-------- original question------------
I have two CCI questions.
Question I.  Can an extrinsic(hpf_local) be invoked in an independent loop?
In a Forall?
     Ex:

Forall (1 = 1:10) a(i) = f(i,a(i))

Note that part of the calling sequence, as specified in Ver 1.1, appendix A, is 
"The processors are synchronized.  In other words, all actions that logically precede the call are completed."

It seems clear that when this was written it was tacitly assumed that the call did not occur in an independent loop or forall.

Part 2:  May ony other kind of extrinsic be called in a forall or independent loop?
================
Discussion begins here:     Summary provided by Rob S. 

Status:   Under discussion.   Summary of the discussion in September:

1.  Only and HPF type routine can be PURE.  Thus, only and HPF_LOCAL
routine could be local, pure, and hence called in a forall.

2.  There is no semantic problem with this, or with invocation of any
extrinsic routine in an independent loop.

3.  An example
        real x(N, 100000)
        !hpf$ distribute x(*, block)

        !hpf$ independent
        do i = 1, N
           call extrinsic_fn(x(i,:))
        enddo
The independent loop is a mechanism for spawning N threads per processor, each independent of the others;  the load per thread may be variable.  It would
possibly be useful here to NOT synchronize after each call to the extrinsic
routine!   Is there any semantic reason to force the synchronization?

4.  There are some very important issues in the implementation, with possible
language impacts.

Let us assume an MPI based implementation of HPF calling an extrinsic
local routine that uses MPI for communication.  Because of the unity of
purpose and of hotel between HPF and MPI, it is arguably necessary for
HPFF to make this work cleanly and efficiently.

Issue MPI-1:   Does HPF handle MPI_Init and MPI_Finalize, automatically.

Issue MPI-2:   Are any of the MPI routines PURE?

Issue MPI-3:   Thread safety.   In a naive implementation, HPF does a barrier
before and after the call to the extrinsic;  but there is no guarantee that
there are no outstanding, nonreceived messages in the messaging system.  Thus, to be safe, any extrinsic routine should use its own communicator.   To prevent
interference between separate calls to the routine, a new communicator should 
becreated for every call.   An obvious way to do this is to call
      MPI_Comm_Dup(MPI_COMM_WORLD, New_comm)
at the beginning of every such extrinsic.    However, an extrinsic
that consumes all its messages would be justified in doing this once, on its
first invocation, and saving the communicator for reuse on later invocations.
   But consider what happens if the extrinsic is called in an independent DO loop
as in the example above, and there is no barrier used.  Now we really need a		(discussion continued)
separate communicator per thread. On the other hand, a call to MPI_Comm_Dup is a collective call, which synchronizes the processes.
     Perhaps this should be done by the calling HPF routine, so that the
MPI_COMM_WORLD communicator is different on every call.

Issue MPI-4:  If called from the range of an ON_HOME directive, what
set of processors does MPI_COMM_WORLD correspond to?   If it corresponds to the subset executing the ON block, then how can the called routine access 
nonresident data?   Should there be a way to access a communicator that 
corresponds to these executing processors, while MPI_COMM_WORLD 
always corresponds to all of the processors?

Issue MPI-5:  If called from separate ON_HOME blocks in the scope of
a TASK directive, with disjoint processors groups, so that the two
ON blocks may be executed concurrently, what communicators correspond to the two processor groups?   (If, in issue 4 above, the answer is that MPI_COMM_WORLD corresponds to the executing subset of the processors, then the answer here is MPI_COMM_WORLD.)

=========== comments from Jim Cownie =================

> Issue MPI-1:   Does HPF handle MPI_Init and MPI_Finalize, automatically.
I would say that the HPF run-time should have called MPI_Init before
any user code has run, therefore user extrinsic functions which need
MPI can just use it.

This is actually not a big deal, since the user routine can always do
use MPI_Initialized() to guard her call to MPI_Init. (Though if this
is done, then the HPF run-time needs to do the same, since MPI_Init
should only be called once.) That's why it's simpler to say that
MPI_Init has already been called before any user HPF code has run.

> Issue MPI-2:   Are any of the MPI routines PURE?
Probably. For instance one could cast reductions as functions which
return the result, and only read the arguments (though why you'd want
to use an MPI reduction extrinsically rather than an HPF one is beyond
me).
 ... after all 5 MPI issues ...
     I would suggest that in all of these cases the HPF run-time should
provide a "current communicator" which includes the set of processes
running the current construct. In some cases this will be
MPI_COMM_WORLD (or a Comm_Dup of COMM_WORLD), in others (ON_HOME, task parallelism, processor subsets) it will represent a subset of the available processes. In MPI MPI_COMM_WORLD is always available as the set of all processes (until MPI-2 introduces a dynamic process model,
though that shouldn't worry HPF implementations).
     Therefore I think
1) MPI_COMM_WORLD is *always* the set of all processes. (This is the
   current MPI view).
2) If you need subsets then you should create new communicators, and
   provide a way for the user code to access them.

MPI_COMM_WORLD should mean the same thing in a routine called from HPF extrinsic as it did in a "raw" MPI program. The HPF extrinsic MPI
environment should contain additions to the raw MPI environment (new
communicators, maybe pre-defined datatypes giving array distributions,
etc), but should not change the meaning of things in the raw MPI
world. In other words you may need to learn more to work in the HPF
extrinsic environment, but you shouldn't have to unlearn things you
already knew about MPI.
-- Jim
=======
ed.  think this was answered, but I didn't get it recorded here ...
_____________________________________________________
_____________________________________________________

CCI #39	Mapping Pointer Restrictions	Group D & E	
Larry Meadows	11/21/95	status: new	
---------- questions--------------
I've noticed an inconsistency in the 1.1 document wrt F90 pointers. You'll
recall CCI 6.3, from last year; this resulted in additional constraints
to the 1.1 document, that alignees and distributees could not have the
POINTER target (p. 27, line 5, and p. 32, line 5). However, later in the
same chapter, there is some text (p. 38, line 33):

A variable with the POINTER or ALLOCATABLE attribute may appear as an alignee
in an ALIGN directive or as a distributee in a DISTRIBUTE directive.

This is clearly a contradiction.

This is also somewhat related to CCI #11 (what is the SEQUENCE attribute
when associated with a pointer).

I'm assuming the following, in HPF 1.1:

1) Mapping statements can't apply to pointers. Note, therefore, that allocated
   pointers cannot be explicitly mapped.

2) Pointers can point to any object, regardless of their mapping attributes.

For HPF 2.0, there was some restriction on pointers that I've forgotten.
It really seems that a restriction similar to the following would be in
the spirit of Kernel HPF:

        Mapping statements can be applied to pointers. These statements
        assert that any object with which the pointer is associated will
        have the described. Pointers with mapping statements may point
        only to whole arrays, not to subarrays.

        Pointers without mapping statements may point only to objects without
        explicit mappings, and may point to subarrays of those objects.

lfm
-------
ed note:   this has both a cci item for group D and kernel input for group E ... but is primarily a group D item.

End of active CCI items for Jan 96 HPFF meeting.
---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------