Subject: hpff-core: Active CCI for March Meeting

From: Mary E Zosel <[email-masked]>
Date: Mon, 11 Mar 1996 10:42:26 -0600 (CST)

---------------------------------------------------------------------------
[email-masked] is a mailing list for announcements related to High
Performance Fortran.  Instructions for adding or deleting yourself
from this list appear at the bottom of this message.
---------------------------------------------------------------------------

Active HPF CCI for March 96 Meeting
Sorted by Group
Thanks to all who provided first answers to these.

CCI # 52        use of forall index     Group C 3/8/96  
Dechering       [email-masked]  2/16/96 status: new

Questions:
I tried to figure out whether I may use array bounds in a forall
assignment that are defined by the index variable. For example, is the
forall in the following program fragment valid

        integer a (10, 2)
        integer b (11)

        forall (i:10) a (i, 1:2) = b (i:i+1)

I haven't found any constraints that forbid this construct, but I'm not
sure this garantuees that it is allowed.

Can you tell me whether this use of the forall statement is valid or not;
and/or can you tell me where I can find the answer to my question?

Thanks in advance for your help,

Paul Dechering
        ---------------------------------------------------------
       | telephone: +3115 2782419                                |
       | telefax  : +3115 2786740                                |
       | E-mail   : [email-masked]             |
       | WWW page : http://www.cp.tn.tudelft.nl/P.F.G.Dechering/ |
        ---------------------------------------------------------
------------
Carol answers ....

This is absolutely OK (assuming the typo is fixed). Typically the
forall indices are used on the right hand side of the assignment
as well as the left.

           forall (i=1:10)  a (i, 1:2) = b (i:i+1)
                    **
=================================================

GROUP D CCI

=================================================

CCI #   47      distribute/pointer contradiction        Group D 3/8/96
Singleton       [email-masked]      2/2/96  status: new
        
Question:
Is there a contradiction within the HPF 1.1 Language Specification
document with regards to distributed pointer arrays?

In section 3.3, page 27, line 5 (repeated in appendix C, page 186,
line 48), it is stated that a distributee in a DISTRIBUTE directive
may NOT have the POINTER attribute. However section 3.6, page 38, line
24, makes the contrary statement. The rest of section 3.6 clearly
suggests that pointer arrays are supposed to be distributable.

The negative statements where not in HPF Version 1.0.

Can you clarify please?

Thanks,
David Singleton

-----------
follow-up question after Henry's first reply ...

Thanks for your prompt reply Henry.

It is reasonable that a pointer array has the distribution of its
associated target. However this does not cover the case when the
pointer array is allocated instead of pointer assigned to an existing
array. If pointers are removed from section 3.6 then the only
conclusion is that pointer arrays cannot be allocated as distributed
arrays. Is this correct?

If true, this seems highly and unnecessarily restrictive. If
allocatable and automatic arrays can be distributed, why not
dynamically allocated pointer arrays? The big advantage of allocated
pointer arrays is that they can be placed in the global namespace via
COMMON or modules. The Fortran90 and HPF constraints on nonsequential
COMMON blocks should be sufficient to include distributed pointer
arrays.

Not allowing pointer arrays as distributees or alignees also
eliminates the possibility of REDISTRIBUTING or REALIGNING them after
allocation as well.

Even without having pointer arrays, CMFortran successfully encompassed
dynamic allocation of distributed arrays albeit by actually giving the
user access to array descriptors. Fortran90 pointer arrays should only
simplify the user interface to such dynamic allocation.

I would be very interested to understand the HPFF stance on allocated
distributed pointer arrays - it doesn't appear to be addressed in HPF2
documents. If you can point me at any other relevant online
documents, I'd be wery appreciative.

Thanks for your help and time,

-------------------

First answer from Henry
Hi David,
> Is there a contradiction within the HPF 1.1 Language Specification
...
> suggests that pointer arrays are supposed to be distributable.
     Yes, there is a contradiction in the HPF 1.1 document.  The response to an
interpretation question posed was that pointers cannot appear as distributees
in a DISTRIBUTE directive.  Instead, a pointer acquires the mapping of its
associated target, if any.
     The instance you cite in section 3.6 was missed in the HPF 1.1 document,
but will be fixed in HPF 1.2.
Thanks,  Henry
-----
followup reponse from Henry after second question
   Hi David,
   > It is reasonable that a pointer array has the distribution of its
  ...
   > conclusion is that pointer arrays cannot be allocated as distributed
   > arrays. Is this correct?

   That's correct; a pointer appearing in an ALLOCATE statement would get some compiler default distribution (probably replication).

   > If true, this seems highly and unnecessarily restrictive. If
   .....
   > eliminates the possibility of REDISTRIBUTING or REALIGNING them after
   > allocation as well.
   The original interpretation request that prompted this restriction asked
   whether the following was a valid program - could a pointer be given one
   mapping and made to point at something with a different mapping.
              program p
                integer, pointer :: ptr(:)
                integer, target :: targ(100)
       !hpf$    processors proc(number_of_processors())
       !hpf$    distribute targ(cyclic) onto proc
       !hpf$    distribute ptr(block) onto proc

                ptr => targ
              end program p
HPFF decided that it would be best to avoid such problems by prohibiting the
pointer from having an explicit mapping, and having it acquire the mapping of
its target.  This (perhaps inadvertently - I'm not sure) had the effect of
preventing the user from explicitly mapping pointers when they become
associated through the ALLOCATE statement.
   > Even without having pointer arrays, CMFortran successfully encompassed
  .........
   > documents, I'd be wery appreciative.
 This issue was raised by Larry Meadows for HPF 2.0, but I'm not sure off-hand
   how it was dealt with at the last HPFF meeting.  Does anyone recall whether
   this aspect of the problem came up for discussion?
 Thanks,  Henry
---- and additional comment from Carol  ---- 
I only recall that this came up at the vendors subgroup, where we proposed
that DISTRIBUTE and ALIGN be reallowed for POINTER and ALLOCATABLE 
variables for precisely this reason. The idea, as I understood it, was to let POINTER arrays 
be mapped, but only (in the absence of DYNAMIC) allow them to point to similarly 
mapped variables. (I don't think we talked about the question of pointing at array sections 
of similarly mapped variables.) I'm not sure what the last official HPFF vote on pointers 
was, although I believe it implied that nondynamic pointer arrays should only point to
things with the same mapping, which seems to me most compatible with their
having a mapping themselves, not with their having no mapping.

====================================================

CCI #   49      distribution with array border elements Group D 3/8/96          
Rosing  [email-masked]        2/7/96  status: new     

Question:
I'm trying to figure out how to distribute two types of arrays to
minimize communication, the first has border elements and the second
doesn't. The first array is declared B(-2:nx+2, -2:ny+2, -2:nz+2) and
the second is declared A(nx,ny,nz). Array A should be distributed by
blocks in one or more dimensions and array B should be distributed
such that B(i,j,k) is on the same node as A(i,j,k) for 0<i<=nx,
0<j<=ny, and 0<k<=nz. The border elements should be on the nodes where
the neighboring non-border elements are.

The align directive doesn't appear to work because the lengths of the
dimensions are not the same (page 34, line 42, version 1.1 of the
manual). I can't afford to put extra elements around A for several
reasons; A is really a 5D array and the extra storage would be large
(there are also lots of A's), for reasons of load balancing I
don't want the border elements included in how the block sizes are
computed and placed, and I don't want to rewrite all of the array
operations on the A arrays to not operate on any new borders.

This seems like a typical problem and I'm fairly sure it was discussed
at one time.

matt

([email-masked])
-----
initial reply from Mary
Matt ... I'm going to forward this to the hpff-interpret mailing
list.  In a quick reading of your message - it looks to me like you
need the "generalized block" feature that has been accepted as
a feature in the latest round of HPFF meetings  (but is likely  to be in
the set of features that has an uncertain future wrt implementation).
Someone else can comment if I've misread the question or if it needs
generalized block plus something additional.
   -mary-

---
follow-up from Chuck

In short, Mary's answer was correct.  To do precisely what you want, you
need the generalized block mapping.  Current thinking is that this will be
part of the "approved extended features" of HPF 2.0.  That means the syntax
and semantics will be defined, but vendors are likely to concentrate on the
core (aka kernel) features of the language first.

BTW, You can use ALIGN as follows:
        !HPF$ ALIGN A(i,j,k) WITH B(i,j,k)
Or
        !HPF$ ALIGN A(:,:,:) WITH B(1:nx,1:ny,1:nz)
The rule is basically that the array sections in ALIGN must conform, but
the align-target (B) doens't have to specify the whole array.  However,
this ALIGNment doesn't solve your problem with not counting boundary
elements in the block sizes.

                                                Chuck

=============================

CCI #   50      implicit remapping      Group D 3/8/96  
Zongaro [email-masked]    2/8/96  status: new     
Question:
Hello,
     The much-beloved section 3.10 of the HPF Language Spec. states (in part)
on page 54 that

      The rules on the interaction of the REALIGN and REDISTRIBUTE directives
      with a subprogram argument interface are:
      1. . . . .
      2. If an array or any section thereof is accessible by two or more paths,
         it is not HPF-conforming to remap it through any of those paths. . . .
         In general, in any situation where assignment to a variable would be
         nonconforming by reason of aliasing, remapping of that variable by an
         explicit REALIGN or REDISTRIBUTE directive is also forbidden.

     These rules deal explicitly with REALIGN and REDISTRIBUTE, but not with
some other situations in which aliasing might occur.  For example, is the
following a conforming program?  Here 'a' is remapped on becoming associated
with 'x'.  Can an element of 'a' then be referenced in sub?  If I had a call
to HPF_DISTRIBUTION passing 'a', what would be the AXIS_TYPE?
      program p
        integer :: a(100) = 1
!hpf$   distribute a(block)
        call sub(a)
      contains
        subroutine sub(x)
          integer x(100)
!hpf$     distribute x*(cyclic)

          i = a(1)
        end subroutine sub
      end program p

How about the following?  'a' is remapped when it becomes associated with 'x' -
can 'a' then be passed to sub2, which would require remapping as well?  Given
that REALIGN and REDISTRIBUTE are prohibited, my guess is that this would be prohibited as well.

      program p
        integer :: a(100) = 1
!hpf$   distribute a(block)
        call sub(a)
      contains
        subroutine sub(x)
          integer x(100)
!hpf$     distribute x*(cyclic)

          call sub2(a)
        end subroutine sub

        subroutine sub2(y)
          integer y(100)
!hpf$     distribute y*(*)
        end subroutine sub2
      end program p

How about this one?  Here, the same actual argument is associated with two
different dummy arguments, each of which requires a different mapping from the
actual.

      program p
        integer :: a(10, 10) = 1
!hpf$   distribute a(block, block)
        call sub(a, a)
      contains
        subroutine sub(x, y)
          integer x(10, 10), y(10, 10)
!hpf$     distribute x*(cyclic, block)
!hpf$     distribute y*(block, cyclic)

          print *, x, y
        end subroutine sub
      end program p

If the mapping of 'x' had been the same as that of 'a', would the third example
then be conforming?
Thanks,  Henry

================================

CCI # 39        Mapping Pointer Restrictions    Group D 1/9/96
Larry Meadows   [email-masked]  11/21/95        status: not so new
(didn't record answer to this one)

Question:       
I've noticed an inconsistency in the 1.1 document wrt F90 pointers. You'll
recall CCI 6.3, from last year; this resulted in additional constraints
to the 1.1 document, that alignees and distributees could not have the
POINTER target (p. 27, line 5, and p. 32, line 5). However, later in the
same chapter, there is some text (p. 38, line 33):

A variable with the POINTER or ALLOCATABLE attribute may appear as an alignee
in an ALIGN directive or as a distributee in a DISTRIBUTE directive.

This is clearly a contradiction.

This is also somewhat related to CCI #11 (what is the SEQUENCE attribute
when associated with a pointer).

I'm assuming the following, in HPF 1.1:

1) Mapping statements can't apply to pointers. Note, therefore, that allocated
   pointers cannot be explicitly mapped.

2) Pointers can point to any object, regardless of their mapping attributes.

For HPF 2.0, there was some restriction on pointers that I've forgotten.
It really seems that a restriction similar to the following would be in
the spirit of Kernel HPF:

        Mapping statements can be applied to pointers. These statements
        assert that any object with which the pointer is associated will
        have the described. Pointers with mapping statements may point
        only to whole arrays, not to subarrays.

        Pointers without mapping statements may point only to objects without
        explicit mappings, and may point to subarrays of those objects.

lfm
-------

What was the answer to this one???      
        
=========================

GROUP E CCI

=========================

CCI #   45      extrinsic clarifications        Group E 3/8/96  
Hennecke        [email-masked]    1/23/96 status: new     
Hello,
some questions and comments on EXTRINSIC are added below. I assembled them when additions to more suitable places and (b) significantly shortened to only those aspects that do not depend on possible restrictions for
<extrinsic-kind-keyword>s other than HPF. All such material should be collected in the annexes (A and B for the HPF_* keywords) to separate the general EXTRINSIC functionality from the specific implementations of the two interfaces defined within the HPF language.
Regards,
Michael Hennecke      http://www.uni-karlsruhe.de/~Michael.Hennecke/
 ----------------------------------------------------------------------
  University of Karlsruhe         RFC822: [email-masked]
  Computing Center (G20.21 R210)           BITNET: [email-masked]
  Zirkel 2  *  P.O. Box 69 80                 Phone: +49 721  608-4862
  D-76128  Karlsruhe                               Fax: +49 721  32550
 ====================================================
HPF v1.1, section 6
143:14+  change "subprograms" to "procedures"
         SISAL and C extrinsics are not subprograms.

143:14+  Is a FORTRAN_LOCAL subprogram an HPF subprogram?

143:16.5 "It defines the means for handling distributed and replicated
          data at the interface."

         I do not see these definitions in section 6, which only imposes general restrictions on the callee in 6.3. IMHO, this is not in the scope of this section which only defines the caller's (declaration) part. Everything else is processor-dependent, except for the HPF_LOCAL and HPF_SERIAL specifications which are also not in section 6 but in annex A and B.

144:6.8  "should be declared EXTRINSIC ..."
         Is this a requirement or a recommendation?
         (Since HPF supports EXTERNAL, it probably cannot be a requirement.)
         If it is a recommendation, perhaps add more explanatory material
         like "Because HPF has to communicate some information about the
         distribution/alignment of procedure arguments, a user's assumptions
         about implementation details are more likely to fail as in sequential
         languages."

144:11   "6.2  Definition and Invocation ..."
         Change "Definition" to "Declaration". Except for the requirement
         that EXTRINSIC(HPF) is redundant and may appear in the definition
         of an HPF subprogram, this section is about declaring interfaces
         to EXTRINSIC procedures, not defining them.

144:26   Change "\bnf{extrinsic-kind}" to "\bnf{extrinsic-kind-keyword}"
         (alternatively, delete \bnf markup.)

144:29   Add a reference to annex B for HPF_SERIAL specifications:
         "The keyword HPF_SERIAL is intended for use in calling uniprocessor
          routines described in Annex B."

144:41.6 change "subprogram" to "procedure" (see 143:14+)

144:42.7 change "subprograms" to "procedures" (see 143:14+)

145:1+   The examples given implicitly require the C_LOCAL and SISAL
         languages to be able to pass arrays by descriptor because
         assumed shape arrays are passed to them.
         On 146+:43+, COBOL_LOCAL also recieves assued-shape arrays.
         This should be avoided. However: all argument passing becomes
         tricky if the EXTRINSIC is not Fortran...
        doc status not known    question continued:

145:9+   The OPERATOR(+) interface is not standard-conforming:
         add INTENT(IN) for X and Y.

145:32.8 "The intent is ... "
         Context lost... This should be a general requirement on EXTRINSIC,
         here it appears after the discussion of a generic interface example.

145:33.7 change "has been" to "had been"

146:2    "... mut have an explicit interface"
         Question 1: shoudn't this be "EXTRINSIC interface"? A mere interface
                     block does not help, except for the default case.
         Question 2: Shouldn't this requirement be less strict? To my
                     understanding, the only case where such an interface is
                     always required is when the procedure arguments are
                     distributed HPF arrays (or the Fortran standard requires
                     one).

146:11   Move this stuff up to then beginning of 6.2.

146:28   "main-program"
         Change from \tt to \bnf markup

146:41   "... as the host scoping unit" (and the following example)
         There is no such thing: host association is restricted to internal
         and module subprograms, and derived type definitions (14.6.1.3).
         An interface block always has its own scope, it cannot access anything
         from outside by host association. Introducing such mechanisms
         in HPF seems to be very confusing.

147:38   change " ahs " to " has "

=====continued in #46  =======================

CCI # 46        extrinsic clarifications        Group E 3/8/96  
Hennecke        [email-masked]    1/23/96 status: new      
Question:
NOTE THIS IS A CONTINUATION OF #45 with the Annex part split into a
 separate item because of length.

HPF v1.1, annex A

162:8    Change "accessible to an extrinsic procedure (arrays passed as
         arguments)" to "passed as actual arguments to an extrinsic procedure",
         because there are more arrays accessible than array arguments: COMMON
         and MODULE objects.

164:4.5  "distributed HPF array"
         Does this include replicated arrays? If not, list them in the same
         lines as scalars.

165:3+   change \tt to \bnf markup three times

165:6    change "the the" to "the"

166:8.5  change "type" to "type and type parameters"

178:30   change "type" to "type and type parameters"

==============================

CCI#    48      Fortran 90 questions    Group E 3/8/96

Feng    [email-masked]    2/7/96  status: new     Dear Profassor,
        I am a graduate student of Peking Univ., P.R.China. Now, we are writing
a complier of HPF. I have some trouble, can you help me?
        I am processing COMMON statement and EQUIVALENCE statement, I do not
know how to process the following examples:
        program comm
                real a                          (1)
                common /b/c,d                   (2)
                f=1.0                           (3)

        contains
        function xx(ff)
                common /e/a,f                   (4)
                common /b/g,z                   (5)
                real j,k
                equivalence (c,j)               (6)
                equivalence (a,k)               (7)
        end
        end
on this example, I have some question :
        1. Can I treate the common block 'b' (in statement 5) as a continuation
of the common block 'b' in statement 2?
        2. Are the two 'a' (in statement 1 and statement 4) the same one? What
about 'f' (in statement 3 and statement 4)?
        3. Can I treate the variable 'c' in statement 6 as the same symbol of
'c' in statement 2? What about 'a' in statement 7?

        We will start our winter vocation at this week end. So can you replay
me before 9th?
        Thanks very much.

                                        Sinceresly yours
                                        Feng Xiaobing

-----------------
                Henry replies ...

     I hope this reaches you in time:
     1) No, the common block in (5) cannot be treated as a continuation.
        Statement 5 redefines the common block 'b', so 'c' in the main program
        shares storage with 'g' in the internal function, and 'd' in the main
        program shares storage with 'z' in the internal function.
     2) The 'a' variables defined in (1) and (4) are different variables.
        The 'f' variables defined by (3) and (4) are different variables.
     3) The 'c' variables defined by (2) and (6) are different variables.

If you have a copy of the Fortran 90 standard, see Section 12.1.2.2.1.  There
you'll find a list of situations in which host names (like those declared
by (1), (2) and (3)) are not accessible in an internal subprogram or module
subprogram - basically, whenever something with the same name is declared in
the internal or module subprogram.

     I hope this helps.

Thanks,

Henry

=============================

CCI #   51      array sections and HPF_LOCAL    Group E 3/8/96          
Zongaro [email-masked]    2/13/96 status: new

        Hello,
     I thought I had sent this question a few weeks ago, but I couldn't find
any record of having done that - my apologies if this is a duplicate.

     Consider the following program.

         program prog
           interface
             extrinsic(hpf_local) subroutine sub(dummy)
               integer :: dummy(:, :)
   !hpf$       inherit dummy
             end subroutine sub
           end interface
           integer actual(4, 4)
   !hpf$   processors p(2, 2)
   !hpf$   distribute actual(block, block) onto p

           call sub(actual(1:2, 1:2))
         end program prog

         extrinsic(hpf_local) subroutine sub(dummy)
           use hpf_local_library
           integer :: dummy(:, :)

           print *, 'proc = ', my_processor(),                                &
        &           'lbound = ', lbound(dummy),                               &
        &           'ubound = ', ubound(dummy)
         end subroutine sub

Assuming that elements of p correspond to physical processors as follows
             p(1,1)     0
             p(2,1)     1
             p(1,2)     2
             p(2,2)     3
which of the following correctly describes the output from the four instances
of the HPF_LOCAL subroutine?

a)     proc = 0 lbound = 1 1 ubound = 2 2
       proc = 1 lbound = 1 1 ubound = 0 2
       proc = 2 lbound = 1 1 ubound = 2 0
       proc = 3 lbound = 1 1 ubound = 0 0

b)     proc = 0 lbound = 1 1 ubound = 2 2
       proc = 1 lbound = 1 1 ubound = 0 0
       proc = 2 lbound = 1 1 ubound = 0 0
       proc = 3 lbound = 1 1 ubound = 0 0

In other words, should sub behave as if dummy is divided up like this

            -------------------------------------
           | dummy(1:2, 1:2)  |  dummy(1:2, 1:0) |
a)          -------------------------------------
           | dummy(1:0, 1:2)  |  dummy(1:0, 1:0) |
            -------------------------------------
or this?
            -------------------------------------
           | dummy(1:2, 1:2)  |  dummy(1:0, 1:0) |
b)          -------------------------------------
           | dummy(1:0, 1:0)  |  dummy(1:0, 1:0) |
            -------------------------------------
Thanks, Henry
--------------------
Rob replies ...
Henry,
Good question about lbound and ubound for dummies
of extrinsic local routines!    I favor iterpretation (a)

            -------------------------------------
           | dummy(1:2, 1:2)  |  dummy(1:2, 1:0) |
a)          -------------------------------------
           | dummy(1:0, 1:2)  |  dummy(1:0, 1:0) |
            -------------------------------------
All HPF distributions onto multidimensional processors arrangements are
cartesian products of one-dimensional distributions; this interpretation
is consistent with that rule.
Rob
----- 
comment from David Watson 
[email-masked]
---
Can I add support for interpretation a) i.e.
            -------------------------------------
           | dummy(1:2, 1:2)  |  dummy(1:2, 1:0) |
            -------------------------------------
           | dummy(1:0, 1:2)  |  dummy(1:0, 1:0) |
            -------------------------------------
If we consider a function or subroutine delivering a result sized according
to an incoming argument this interpretation is necessary for correct behaviour.
The following (exceedingly artificial) code illustrates the necessity

         program prog
           interface
             extrinsic(hpf_local) function funres(dummy)
               integer :: dummy(:, :)
               integer :: funres(size(dummy,1))
   !hpf$       inherit dummy
   !hpf$       align with dummy(:,*) :: funres
             end subroutine funres
           end interface
           integer actual(4, 4)
           integer junk(4)
   !hpf$   processors p(2, 2)
   !hpf$   distribute actual(block, block) onto p
   !hpf$   align junk(:) with actual(:,*)

           junk(1:2) = funres(actual(1:2,1:2))
         end program prog

         extrinsic(hpf_local) function funres(dummy)
               integer :: dummy(:, :)
               integer :: funres(size(dummy,1))
   !hpf$       inherit dummy
   !hpf$       align with dummy(:,*) :: funres
               funres = 0
               dummy  = 0
         end subroutine funres

under interpretation b) processor 2 (assuming the numbering in the original
example) would return a size zero array into a size 2 array.
Cheers,
   Dave

==============================
CCI#  53       extrinsic(f77_local)   Group E 3/8/96  

Whaley   [email-masked]     3/4/96  status: new

        Hi,   I am a research associate with Jack Dongarra.  I have been looking at providing an HPF interface to ScaLAPACK, our linear algebra Fortran77 message passing library.  In the course of this work, I've run into several problems involved in calling an extrinsic routine.  Most of these problems seem to be in the present generation of compilers.  However, I've run into one that seems to be a function of the HPF specification, which I believe could be fixed by a small addition to the extrinsic section.  I include below a small latex note on the
issue.  I would be happy to provide further information or help if I can.
     I have also been reading the HPFF minutes for information on how HPF is going to look in the future.  I was rather alarmed to note that kernel HPF appears
to be heading towards no support for cyclic(n).  I assume the committee is
assuming that algorithmic blocking will ameliorate the performance hit for
such a decision?  I know that our own library does not yet support algorithmic
blocking for many routines; where it is absent, a cyclic(1) distribution yields
minimal performance . . .  I am not abreast of all of the literature on the
subject, but it is not clear to me that all algorithms can efficiently use
algorithmic blocking . . 

  Perhaps I don't understand what kernel HPF really is. Is it HPF2.0's
subset HPF?   Or is it features the compiler should support efficiently,
with other features
supported but not with the same degree of optimality?
Thanks for your help,    Clint
(ed. note sorry - for unformatted input here)
\documentstyle[11pt]{article}
\topmargin .5in
\pagestyle{myheadings}
\markboth{\underline{\bf DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT}}
\textwidth=6in
\textheight=8.5in
\hoffset = -.6in
\voffset = -.6in
\title{Passing multidimensional arrays from HPF to extrinsic routines}
\author{
R. Clint Whaley
\thanks{
Dept. of Computer Sciences, Univ. of TN,
Knoxville, TN 37996, {\tt [email-masked]}
}  }
\begin{document}
\maketitle
\begin{abstract}
Under the present extrinsic interface rules, it is not possible to portably
pass multidimensional arrays from HPF to a language such as Fortran77 or C.
In this paper we discuss the problem and suggest some solutions.
\end{abstract}
\vspace*{1in}
\centerline{ \bf \Large \underline{DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT} }
\newpage
We believe that an important area of interfacing HPF routines and local
Fortran77 has been overlooked.  The problem involves the translation of
a HPF/F90 assumed {\em shape} array to a F77 assumed {\em size} array.
   This problem may be overcome in F90 by specifying an explicit interface.
This approach is illustrated in figure~\ref{fig-f902f77}.
Obviously, if the compiler has not already stored the indicated (sub)array
in an array whose first dimension is equal to the {\tt SIZE} of the array,
it will have to allocate space and copy the array.
begin{figure}
\begin{verbatim}
SUBROUTINE F90_TO_F77(A)
REAL, INTENT(INOUT) :: A(:,:)

INTEGER :: M, N, LDA

INTERFACE
   SUBROUTINE F77ROUT(M, N, A, LDA)
   INTEGER, INTENT(IN) :: M, N, LDA
   REAL, INTENT(INOUT) :: A(LDA,*)
   END SUBROUTINE F77ROUT
END INTERFACE

M = SIZE(A, 1)
N = SIZE(A, 2)
LDA = M
CALL F77ROUT(M, N, A, LDA)
RETURN
END
\end{verbatim}
\
\caption{A F90 routine passing a 2D array to a F77 routine\label{fig-f902f77}}
\end{figure}

Note that in distributed memory terms, leading dimension (\verb+LDA+ in the above example) is an inherently {\em local} quantity: it indicates the memory stride between elements in a row.  Thus the F90 approach cannot be directly utilized i n HPF, since the interface described by HPF is {\em global}.

This oversight means that only 1D arrays may be standardly passed from HPF to
external languages such as C or Fortran77 (this is obviously true because it is
impossible to determine how the compiler has locally laid out the matrix, and
thus it will be impossible to do indexing on such arrays).  This means that
codes wishing to pass arrays with greater than 1 dimension will have to be
compiler specific (actually, compiler version specific).

Compilers supporting extrinsic(f90\_local) should not have this problem. There,
the user may pass the HPF matrix to a f90\_local routine which takes the matrix
as assumed shape.  The f90\_local routine may then use the code in
figure~\ref{fig-f902f77} to pass the local array to the Fortran77 code.

Compilers which support only f77\_local will not have this option.  This makes
f77\_local less useful as an interface.  Obviously, specific extrinsics are
not part of the HPF standard, so it is not surprising that this issue has not
been addressed by HPF.  However, just as the local library can be part of
the standard for those compilers supporting hpf\_local, a solution to this
problem can be adopted as part of the standard for those compilers choosing
to support f77\_local.

We propose a two-fold solution to this problem.  The first takes the form of
an added distribution directive: \\
{\tt !HPF\$ ASSUMMED\_SIZE :: A(LD1, LD2, $\dots$, *)} \\
This command, which is legal only in an interface to an extrinsic routine
accepting assumed size arrays, asserts that the dimensions of the local
matrix match the dimensions specified by the parameters {LD1, LD2, \dots}.

This is the minimal extension required to make multidimensional arrays usable
across languages.  The user can fill in minimal values for {\tt LDx} by the
appropriate call to {\tt SIZE} from an {\tt HPF\_LOCAL} routine (assuming the
compiler supports {\tt HPF\_LOCAL}).  This fix has a weakness in
common with that used by F90, however.  To whit, if the compiler is currently
storing the array with a non-minimal dimension, a data copy will be dictated by
the interface which, at least theoretically, could have been avoided.

In today's compilers, data copies occur on most subarray accesses, so this poin
t
is not as significant as it might be.  However, as compilers become more effici
ent,
this should no longer be the case, and the implied data copy in forcing the use
r
to ``guess'' the dimension presently being used may become significant.  We
therefore believe that the HPF standard should encourage vendors supporting
extrinsic interfaces to include the routine:\\
{\tt INTEGER FUNCTION LOCAL\_DIM(ARRAY, DIM)}\\
\verb+   +Returns the local number of elements of ARRAY along the axis DIM.

With this routine, the user can query the dimensions of the array the compiler
will use if array ARRAY is passed to an extrinsic routine.  This should allow
the user to find the correct dimensions to avoid dictating a data copy, if he
so chooses.
\end{document}

================================

CCI # 54      local inquiry about globals     Group E 3/8/96
Offner  [email-masked]  3/7/96  status: new

        We have run into some problems with the hpf local inquiry intrinsics
(e.g., GLOBAL_TO_LOCAL, GLOBAL_UBOUND, etc.).  These intrinsics are
described as returning information about the
        "actual HPF global array argument associated with an
        HPF_LOCAL dummy array argument".
The problems we see are these:
1.  The actual argument might be an array expression.  This wouldn't
be a problem for things like array bounds information -- I think
Fortran already has a consistent way of defining such things in such
cases (e.g. for assumed-shape arrays).  But what about the MAPPING of
the actual?  Expressions have no mappings, but these intrinsics can be
used to extract global mapping information.
2.  The actual argument might be remapped by the compiler in the
course of processing the subroutine call.  (This would commonly happen
if the dummy argument had a different mapping.)  The user really is
then interested in the "remapped actual".
3.  This could also happen if the compiler (for whatever reasons of its
own) decides to use a different mapping than that specified by the
programmer.  Compilers today are not going to do that, but in the
future one could imagine an optimizing compiler with that kind of
capability.  The user certainly wants information about the real
object, not about whatever appears syntactically specified.
4.  Finally, there is an additional problem:  There are mappings that
can be specified in more than one way.  A compiler will typically not
store the source of the mapping directives -- it will pick some
internal representation for the mapping.  When it reports information
from this internal representation, it may look superficially different
to the programmer, even though it amounts to the same thing.
     Furthermore, the representation may depend on things like the number
of processors being compiled for.  To take an extreme case (one could
make up a more realistic case; but this exhibits the problem well
enough), suppose the compiler is told to compile an HPF program for
execution on 1 processor.  It is reasonable for the compiler to decide
internally that all mappings are serial, even if they were specified
as BLOCK in the source program.  Certainly there would be nothing
wrong with reporting a serial mapping, since it would describe
precisely the data layout; but it's not what either the actual or
dummy arguments were described as in the program source.
     The point is that there is really no need for the results of these
inquiry intrinsics to be tied syntactically to the actual argument.
All the user needs is the information necessary to write explicit
message-passing code.  The compiler should be able to use whatever
internal representation is consistent with what the user has written,
and to report information based on this internal representation.
I think that's what the user really wants, in any case.
      Here is the idea of what we would like to say:  Since the ONLY reason
that a user would ever want to call such a routine is to extract
information in order to perform explicit message-passing, we think
that the routines ought to be defined to return information about
        "the global object of which the dummy argument is a
        local section"
with the understanding that this "global object" is implementation-defined.

The problem, of course, is that I don't know a standard
computer-language formal way to state this.

                        --Carl Offner
                        [email-masked]

--------
reply from Chuck
I'll try to get straight to the punch line.  Let me admit up front,
however, that I'm a little hazy on some aspects of EXTRINSIC.

>Here is the idea of what we would like to say:  Since the ONLY reason
...
>with the understanding that this "global object" is
>implementation-defined.
>
>The problem, of course, is that I don't know a standard
>computer-language formal way to state this.
I thought this is what the original language meant.  (Or was intended to
mean...)
    I think of it this way:
The EXTRINSIC callee gets  an area of memory (the dummy argument) and a
data descriptor (not directly visible) from the caller.
    GLOBAL_FOO and FOO_GLOBAL routines extract information otherwise
inaccessible from the descriptor.  [Suddenly I'm suspicious that these
routines should be intrinsics - else it's hard to explain where they get
their arguments...]
    The "actual HPF global array argument" is an array with (a copy of) that
descriptor and area of memory.  This may not correspond to any particular
syntactic element of the program.
     I would agree that the current wording isn't crystal clear (despite being
written by Guy Steele, if I remember right).  But then the whole idea of
EXTRINSIC procedures has always been kind of hard for me to grasp...
     Starting from that premise, let's see if any of these problems get clearer.

At 08:24 03/08/96, Carl Offner wrote:
>
>The problems we see are these:
>
>1.  The actual argument might be an array expression.  This wouldn't
>...
>used to extract global mapping information.
The mental model seems to handle this OK in common cases- the caller has to
build a descriptor, which may not correspond to any array variable or
section thereof.  Indirection arrays break the model by making the
descriptor poorly defined.  But they also break INTENT(OUT) and
assumed-shape dummies.
    HPF has rules for when descriptive mappings can be used for dummy arguments
(i.e. when mappings are identical).  Cant we amend the standard so that
GLOBAL_TO_LOCAL & friends are implementation-dependent when a descriptive
mapping cannot be used?  A note to implementors might say "We suggest that
the compiler create the simplest possible descriptor for the other cases -
for example, align the result of an array expression with one of its
inputs."
>2.  The actual argument might be remapped by the compiler in the
>course of processing the subroutine call.  (This would commonly happen
>if the dummy argument had a different mapping.)  The user really is
>then interested in the "remapped actual".

This is definitely unclear in the current language.

Using something along the lines of the mental model, we have to say "do the
remapping before constructing the descriptor".

>3.  This could also happen if the compiler (for whatever reasons of its
>....
>object, not about whatever appears syntactically specified.

The same problem (theoretically) occurs in regular HPF code if the compiler
chooses its own distributions.  HPF_DISTRIBUTE may then return misleading
information.
      The onus is on the compiler to ensure that if it ignores user advice, then
the results remain consistent.  In particular, GLOBAL_TO_LOCAL should
produce an index that gets you to the expected element.  (In regular HPF,
REDISTRIBUTE must move the right elements to the right places.)  Two
mechanisms have been suggested for doing this:

1. Have the HPF library routines return the truth about what the compiler
is doing.  This basically presumes run-time descriptors are queried (and
updated).

2. Before any HPF library routine is called, remap the data back where the
user said it should go, return the "expected" result, then remap data to
where the compiler thinks it should be.  As an optimization, the compiler
can eliminate the remappings as dead code.
     The first option is considered preferable.
    You seem to assume that the "actual" is the syntactic element - I think it
is the runtime object.
    >4.  Finally, there is an additional problem:  There are mappings that
>can be specified in more than one way.  A compiler will typically not
>store the source of the mapping directives -- it will pick some
>internal representation for the mapping.  When it reports information
>from this internal representation, it may look superficially different
>to the programmer, even though it amounts to the same thing.
      You can say the same thing about HPF_DISTRIBUTE in regular HPF.
       My interpretation is that the HPF library inquiries can return any valid
representation of the mapping.  Since the representations correspond to
equivalent mappings, this should not matter to the user.  (I suppose some
users will try to reverse-engineer the optimizations on address arithmetic
this way... It will not matter to the *normal* user.)
       Is there something in the writeup that says "the mapping specified in the syntax"?
>The point is that there is really no need for the results of these
>inquiry intrinsics to be tied syntactically to the actual argument.
      I agree that there is no need to tie them to the syntax.
Maybe it's just my warped point of view, but I don't think they are...
                                                Chuck

=================
=================
New item - not yet in database  Larry Meadows

Since we're on the topic of hpf_local:

! The question is, should this global_lbound return 1 or -104?
     program xc06
      interface
        extrinsic (hpf_local) function vet(array2,results)
        integer vet
        integer, dimension (:,:,:,:,:,:,:)    :: array2
!HPF$ DISTRIBUTE(BLOCK,BLOCK,BLOCK,BLOCK,BLOCK,BLOCK,BLOCK)::array2
        integer,intent(out):: results
        end function
      end interface

      integer, dimension(-104:-100,2,0:1,-1:1,1:3,-1:0,1:2):: a2
      integer results
!HPF$ DISTRIBUTE(BLOCK,BLOCK,BLOCK,BLOCK,BLOCK,BLOCK,BLOCK)::a2
      integer ::a_res
      integer:: num_procs
      a_res=vet(a2,results)
      print *,results
      end

      extrinsic (hpf_local) function vet(array2,results)
      use hpf_library
      use hpf_local_library
      integer vet
      integer, dimension (:,:,:,:,:,:,:)    :: array2
      integer,intent(out):: results
      results=global_lbound(array2, 1)
      return
      end
---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------

---

From: Rob Schreiber <[email-masked]>
Date: Mon, 11 Mar 1996 13:49:51 -0600 (CST)

REAL status of Group C CCI:

      Questions:

              integer a (10, 2)
              integer b (11)

      ------------
      Carol answers ....

>>>>>>    That takes care of that.  Thanks, Carol.   --- Rob