Subject: hpff-external: HPFF proposal for an F77_LOCAL extrinsic interface

From: Carol Munroe <[email-masked]>
Date: Fri, 26 Apr 1996 10:55:15 -0500 (CDT)

---------------------------------------------------------------------------
[email-masked] is a mailing list for discussion of external
interfaces between HPF and other languages/systems.  Instructions for
adding or deleting yourself from this list appear at the bottom of this
message.
---------------------------------------------------------------------------

The following is a proposal (prepared here at Thinking Machines) that I
would like to have considered for next week's meeting. I apologize for
presenting it at the last minute, especially since it may seem long, but
the essence of the interface is described in the first few pages, while
over half of it consists of examples and specifications for utility
routines.

When you look this over, please note that there is an important option left
open concerning how descriptor information should be made available to
local array inquiry routines, similar to the F90 array inquiry intrinsics
and the HPF_LOCAL_LIBRARY ones, assuming that at least some such routines
should be provided.

The question is whether or not to have users pass descriptor handles
explicitly, in light of possible ambiguities resulting from passing local
array data by address only. The main method presented in this proposal
hides the descriptors even when local inquiry functions are used, but an
alternate method of passing them explicitly, when arrays will be used in
inquiry functions, is also suggested in a highlighted paragraph on page
four.

I would appreciate feedback on this topic and on the proposal as a whole.
the sooner the better, to have a sense of what HPFF members really want in
this interface, since there are very few meetings remaining to present this
in.

******cut here for f77_local.tex*********************************************
\documentstyle[twoside,11pt]{article}

\pagestyle{myheadings}
\sloppy
\footheight=-.60in
\headheight=0in
\textwidth=5.50in
\topmargin=-.20in
\oddsidemargin=0.5in
\evensidemargin=0.5in

\itemsep=.05in
\topsep=.05in
\parsep=.05in
\parskip=.10in
\textheight=9in

\newcounter{dofigures}  \setcounter{dofigures}{0}
\newcounter{dotables}        \setcounter{dotables}{0}
\newcommand{\havefigures}{\setcounter{dofigures}{1}}
\newcommand{\havetables} {\setcounter{dotables}{1}}

\newcommand{\cccdocid}[5]{
  \bibliographystyle{alpha}
% ******************** Title Page ********************
  \vspace*{.5in}
  \centerline{\Large \bf {#1}}
  \vspace*{.02in}
  \centerline{{#2}}
  \vspace*{.02in}
  \centerline{{#3}}
  \vspace*{.02in}
  \centerline{{#4}}
  \vspace*{.02in}
  \centerline{{#5}}
  \vspace*{.5in}

  % ******************** List of Figures ********************
  % Note: Remove the following line if there are no figures
%  \ifodd\value{dofigures} \listoffigures \newpage \fi
%  \ifodd\value{dotables}  \listoftables  \newpage \fi
%  \markboth{#1} {#3}
%  \pagenumbering{arabic}
  }

% ******************** Special macros ********************
%\catcode`^^Z=9                % Make TeX ignore IBMPC EOF character

\newcommand{\ital}[1]{{\it #1}}
\newcommand{\type}[1]{{\tt #1}}
\newcommand{\bold}[1]{{\bf #1}}
\newcommand{\dfn}[1]{{\it #1}\index{#1}}
\newcommand{\comment}[1]{}

\def\fineprint{\relax}
\def\stopfine{\relax}

\newcommand{\beginprog}{\begin{tabbing}
xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=\kill
}
\newcommand{\tb}{\>\tt}
\newcommand{\finprog}{\end{tabbing}}
\newcommand{\progtxt}[1]{``{\tt #1}''}
\newcounter{save_enum}
\newcommand{\hilite}[1]{ \item {\bf #1\\} }
\newcommand{\morelater}{\centerline{$\ldots$\ital{Fill this in later}$\ldots$}}

% ******************** Start of Document ********************
\begin{document}

\cccdocid
  {EXTRINSIC(F77\_LOCAL) Interface}
  {Ver 1.1.2}
  {Eugene Loh}
  {Thinking Machines Corporation}
  {April, 1996}

\begin{abstract}

The legacy of FORTRAN~77 serial programming and the power and popularity of
message-passing programming on parallel machines makes it incumbent on
HPF to support a standard \type {EXTRINSIC(F77\_LOCAL)} interface.  Here, an
F77\_LOCAL interface is proposed.  It preserves the basic nature of the
HPF\_LOCAL interface but is also designed to be very straightforward for
FORTRAN~77 programmers.

\end{abstract}

\section{Introduction}

The HPF standard specifies the nature of \type {EXTRINSIC} interfaces in general
and for HPF\_LOCAL in particular.  Nevertheless, specification of an
F77\_LOCAL interface poses challenges due chiefly to two differences between
HPF and FORTRAN~77:
\begin{itemize}
\item Arguments are passed differently.  In FORTRAN~77,
     arguments are typically passed between subprograms
     by address (reference).  That is, no other information
     about the actual argument is passed --- for example,
     data type, size, distribution, etc.  In contrast, HPF
     implementations often pass by descriptor in order to
     make such information available to the subprogram.

\item Very different information about how array
     elements are to be assigned to specific memory
     locations is available to FORTRAN~77 and HPF programmers.

     In FORTRAN~77, the declaration of an array prescribes
     exactly the mapping of array elements to the linear
     sequence of storage locations.  In HPF, the
     mapping of array elements to different processors
     may be controlled (e.g., with DISTRIBUTION and
     ALIGN directives) and queried (e.g., with
     HPF\_ALIGNMENT, HPF\_DISTRIBUTION, and HPF\_TEMPLATE)
     but there is absolutely no information about how
     array elements on a given processor are organized
     within local, serial memory.

     Indeed, different HPF compilers may organize the data
     locally in different manners --- perhaps including
     border cells for ``stencil'' optimizations, or global
     padding to ensure equal-size subgrids on all processors.
     Certainly, different HPF compilers are not \ital {bound} to
     organize local data in any particular manner, and some
     may choose imaginative orderings in such cases as SMP's,
     for example.
\end{itemize}

One could argue that there already exists a well-defined interface
from HPF to F77\_LOCAL if the programmer goes through HPF\_LOCAL.
Quite likely, this would involve SEQUENCE directives in the HPF\_LOCAL
code.  While the formal elegance of going to F77\_LOCAL only through
HPF\_LOCAL is attractive, it is insufficient in several respects and
it is certainly clumsy for FORTRAN~77 programmers.

A direct F77\_LOCAL interface for global HPF is proposed here.  Its
characteristics include:
\begin{itemize}
\item Preservation of the basic nature of the HPF\_LOCAL
     interface.

\item A simplified mechanism for calling
     local routines that eliminates much of the boilerplate
     code that is inherent to an HPF\_LOCAL approach.
\end{itemize}

The proposal is consistent with HPF's specification with regards to
\type {EXTRINSIC} interfaces --- particularly, with regards to local procedures
--- and is consistent with HPF\_LOCAL when appropriate.

For example, before calling and upon returning from the \type {EXTRINSIC}
routine, the HPF code should synchronize all processors.  As necessary,
the compiler should copy incorrectly mapped arguments to temporaries
as indicated by INTERFACE blocks before invocation of and after return
from an F77\_LOCAL subroutine, just as for any other extrinsic or
non-extrinsic routine.  Such remapping deals with what array elements
are assigned to which processors.  Local routines should not share
names of COMMON blocks with global routines.  A local routine should
not have alternate returns.  Any explicit message-passing calls within
a local routine must be resolved before control is returned to the
global caller.  And so on.

Otherwise, this proposal focuses on the differences between HPF\_LOCAL
and this F77\_LOCAL interface.

\section{PROPOSAL}

This proposal may be presented in four parts:
\begin{itemize}
\item Argument passing.
\item HPF-style utility subroutines.
\item Subgrid-inquiry routine.
\item Message-passing interface.
\end{itemize}

\subsection{Argument passing.}

An argument is passed from HPF to an F77\_LOCAL routine by passing the
base address of that section of local memory that has been allocated
to it.  If needed, an argument is first replicated to the processors.

The restrictions in HPF\_LOCAL suffice to ensure that each physical
processor contains a subset of array elements that can be locally
arranged in a rectangular configuration.  This rectangular configuration
is reorganized in local, serial memory according to the \type {MAP\_TO}
attribute specified for that argument in the \type {INTERFACE} block for
the \type {EXTRINSIC(F77\_LOCAL)} routine:
\begin{itemize}
\item \type {MAP\_TO([LAYOUT=]F77\_ARRAY)}
     indicates that the rectangular configuration
     should be FORTRAN~77 sequence associated in local,
     serial memory.

     For example, many compilers
     add border elements for ``stencil''
     optimizations or pad array allocations
     on particular processors so that all processors
     allocate equal amounts of memory for each array.
     Local reordering eliminates such padding and
     provides FORTRAN~77 sequence association for actual data
     values.

     Any local reordering is in addition to
     any global remapping that may be dictated by
     \type {DISTRIBUTION} or \type {ALIGN} directives
     in the \type {INTERFACE} block.

     If no \type {MAP\_TO()} attribute is specified,
     then \type {MAP\_TO(F77\_ARRAY)} is assumed.

\item \type {MAP\_TO([LAYOUT=]NO\_CHANGE)}
     indicates that no local reordering of the data
     should occur.

     This option is desirable when the
     programmer decides that the overhead of local
     reordering should be eliminated or that certain
     characteristics of the global HPF compiler's ordering
     (border cells, equal-size allocations among processors,
     etc.) should be preserved at the local level.  It forces
     the local programmer to access the local data in
     whatever implementation-dependent style the global HPF
     compiler employs.
\end{itemize}

[{\em {Note}}:  The syntax of this attribute corresponds to that of Meltzer's
``HPF calling C Interoperability Proposal.''  It should be modified according
to the fate of that proposal.]

\subsection{HPF-style utility subroutines.}

Of course, the global HPF caller may
continue to call standard HPF array-inquiry routines and intrinsics.  At the
local level, FORTRAN~77 versions of particular HPF intrinsics and HPF\_LOCAL\_LIBRARY
routines are supported.  Since these routines must be called from FORTRAN~77 and not
HPF, there are a couple of important distinctions.

\begin{itemize}
\item The prefix ``F77\_'' is prepended to the utility name.

\item All arguments are required.  None are optional.  The
     effect of omitting an optional \type {DIM} or \type {PROC} argument may
     be attained by specifying a value of -1 for that argument.

\item Arguments are FORTRAN~77 sequence associated.

\item The utilities are subroutines --- none are functions.
     When the HPF analogs are functions, the F77\_ counterparts
     return values in a new argument that is prepended to the
     argument list.

     [{\em {Note}}:  In great part, this distinction arises
     due to FORTRAN~77's inability to deal with array-valued
     functions.  An important consequence, however, is that
     local programmers need not \type {INCLUDE} any header files
     or \type {USE} any modules since no functions, keywords, or
     interfaces need to be declared.]

\item Arguments corresponding to global arrays should use
     the dummy argument passed in from the HPF caller.

     [{\em {Note to implementors}}:  Note that the F77\_LOCAL
     routine accepts arrays passed by address rather than by
     descriptor.  Hence, inquiry information is no longer
     directly available from the argument itself.

     [One implementation approach would be to construct
     a table of the descriptors for the arguments that are
     present in the argument list when the HPF program enters
     F77\_LOCAL mode, along with the corresponding base
     addresses that are actually passed to FORTRAN~77.  A
     utility routine that is called from FORTRAN~77 could check
     this table for the base address and pick up the
     corresponding descriptor.]

     [{\bf Note:  There is a question as to whether or
     not any ambiguities could arise.  That is, perhaps
     two arguments in the call from HPF to F77\_LOCAL share
     the same base address but not the same descriptor
     --- perhaps because pointers are being used or
     because the HPF compiler passes certain array
     sections in-place.  It may be necessary to document
     this restriction and note that an FORTRAN~77 subprogram
     assumes that distinct dummy arguments do not overlap
     in memory.}

     [{\bf These complications arise due to the need to provide
     programmers with inquiry routines at the local level without exposing
     implementation-dependent descriptors to them.  An
     alternative is to have local programmers explicitly
     pass descriptors through to utility routines.  For
     example, instead of the \type {MAP\_TO()} attribute,
     one might have \type {PASS(F77\_ARRAY)}, \type {PASS(NO\_CHANGE)},
     and \type {PASS(DESCRIPTOR)}.  The local routine would declare
     a descriptor object to be \type {INTEGER DESCRX(*)} and the
     local utility routines would expect such objects to be passed
     for global arrays.  Though programmers would not be expected
     to manipulate descriptor objects directly, these descriptors
     would be exposed.}

     [{\bf Feedback on this point is welcome.}]
\end{itemize}

Otherwise, these routines should behave the same as their counterparts,
without the ``F77\_'' prefaces, would in HPF\_LOCAL.  The list of FORTRAN~77, locally
supported, HPF-style routines is
\begin{itemize}
\item \type {F77\_GLOBAL\_ALIGNMENT},
      \type {F77\_GLOBAL\_DISTRIBUTION},
      \type {F77\_GLOBAL\_TEMPLATE}

\item \type {F77\_ABSTRACT\_TO\_PHYSICAL},
      \type {F77\_PHYSICAL\_TO\_ABSTRACT}
\item \type {F77\_LOCAL\_TO\_GLOBAL},
      \type {F77\_GLOBAL\_TO\_LOCAL}

\item \type {F77\_LOCAL\_BLKCNT},
      \type {F77\_LOCAL\_LINDEX},
      \type {F77\_LOCAL\_UINDEX}

\item \type {F77\_GLOBAL\_SHAPE},
      \type {F77\_GLOBAL\_SIZE}

\item \type {F77\_SHAPE},
      \type {F77\_SIZE}

\item \type {F77\_MY\_PROCESSOR}
\end{itemize}

[{\em {Note}}:  There are no \type {F77\_GLOBAL\_\{L,U\}\_BOUND} routines since it is
anticipated that \type {GLOBAL\_\{L,U\}\_BOUND} will be removed from the
HPF\_LOCAL\_LIBRARY.  Tied to their HPF\_LOCAL counterparts, routines
such as \type {F77\_LOCAL\_TO\_GLOBAL} use one-based global indices.]

\subsection{Subgrid-inquiry routine.}

The HPF\_LOCAL style of programming is to
determine bounds of the local portion of a distributed array within the
local routine.  This complicates indexing tremendously, particularly for
multidimensional arrays, for two reasons:

\begin{itemize}
\item The programmer must perform the multidimensional
     indexing explicitly, instead of leaving this job
     to the FORTRAN~77 compiler.

\item One must call utility subroutines and then construct
     global indices if global indices are of interest.
\end{itemize}

Thus, a globally callable subgrid-inquiry routine, which describes local
subgrids in terms of global indices, is provided.  There is also a locally
callable version.  These routines, \type {HPF\_SUBGRID\_INFO} and
\type {F77\_SUBGRID\_INFO},
are described in a later section.  Only the motivation is discussed here.

Consider the HPF code

\begin{verbatim}
              INTEGER X(NX,NY)
        !HPF$ DISTRIBUTE X(BLOCK,BLOCK)
              FORALL ( I=1:NX, J=1:NY ) X(I,J) = I + (J-1) * NX
\end{verbatim}

Writing this in FORTRAN~77 using an HPF\_LOCAL style, one might have

\begin{verbatim}
              SUBROUTINE SUB( X )

              REAL X(*)
              INTEGER L_INDEX(2), G_INDEX(2), L_SHAPE(2)

        ! DETERMINE NX
              CALL F77_GLOBAL_SIZE( NX , X , 1 )

        ! GET LOCAL SHAPE
              CALL F77_SHAPE( L_SHAPE , X )

        ! IDENTIFY OFFSET OF LOCAL DATA WITHIN GLOBAL ARRAY
              L_INDEX(1) = 1
              L_INDEX(2) = 1
              CALL F77_LOCAL_TO_GLOBAL( X , L_INDEX, G_INDEX)

        ! PERFORM OPERATION
              III = 0
              DO J = 0, L_SHAPE(2) - 1
                DO I = 0, L_SHAPE(1) - 1
                  III = III + 1
                  X(III) = (I + G_INDEX(1)) + (J + G_INDEX(2) - 1) * NX
                END DO
              END DO

              END
\end{verbatim}

In contrast, had the subgrid parameters in terms of global indices been passed
in from the global caller, one might have written more easily and clearly

\begin{verbatim}
              SUBROUTINE SUB( X , LB1 , UB1 , LB2 , UB2 )

              INTEGER LB1 , UB1 , LB2 , UB2
              REAL X( LB1:UB1 , LB2:UB2 )

        ! DETERMINE NX
              CALL F77_GLOBAL_SIZE( NX , X , 1 )

        ! PERFORM OPERATION
              DO J = LB2, UB2
                DO I = LB1, UB1
                  X(I,J) = I + (J - 1) * NX
                END DO
              END DO

              END
\end{verbatim}

\subsection{Message-passing interface.}

Many important, local subroutines will
deal only with local data.  Others, however, will require access to
typical message-passing functionality.

This proposal takes the approach of the HPF\_LOCAL specification, which
is to claim that such communications ``are beyond the scope of this
specification.''

This proposal also recommends that no ``generic interface'' to message-passing
functionality from local subroutines be adopted by the HPFF.  New calls
would simply require HPFF to invent and users to learn ``yet another''
message-passing interface.

This proposal also \ital {highly} recommends that HPF compiler providers support
and F77\_LOCAL programmers use the MPI interface for message-passing calls.

\section{SUBGRID INQUIRY}

As mentioned above, a globally callable subgrid-inquiry routine, \type {HPF\_SUBGRID\_INFO},
which describes the local subgrids in terms of global indices, is provided.  The
routine is intended to simplify F77\_LOCAL programming in slightly restricted,
but very common, cases.

\subsection{``Embedding arrays''}

First, a note on ``embedding arrays.''

In this proposal, the phrase ``embedding array'' refers to an array that
is formed by taking a multidimensional, rectangular volume of data and
extending it in each direction in each dimension (possibly by zero amounts).

For example, the two-dimensional volume of data

\begin{verbatim}
             A  B  C  D
             E  F  G  H
             I  J  K  L
\end{verbatim}

may be embedded in any one of

\begin{verbatim}
     *  *  *  *
     *  *  *  *
     A  B  C  D         A  B  C  D         *  *  A  B  C  D  *  *
     E  F  G  H         E  F  G  H         *  *  E  F  G  H  *  *
     I  J  K  L         I  J  K  L         *  *  I  J  K  L  *  *
     *  *  *  *
     *  *  *  *
                                          *  *  *  *  *  *  *
                                          *  *  *  *  *  *  *
             *  *  *  *  *  *             *  *  *  A  B  C  D
             *  A  B  C  D  *             *  *  *  E  F  G  H
             *  E  F  G  H  *             *  *  *  I  J  K  L
             *  I  J  K  L  *             *  *  *  *  *  *  *
             *  *  *  *  *  *
\end{verbatim}

While FORTRAN~77 compilers use sequence association to store rectangular volumes of
data in serial memory, HPF compilers often embed local, rectangular subgrids
in larger arrays and use sequence association for the larger, embedding arrays.
The paddings are typically introduced to facilitate ``stencil'' optimizations
or ensure equal-size memory allocations across the all processors.

Whether due to user-specified alignments or variations among implementations
regarding row-major versus column-major orderings, the axes of the embedding
array may be ordered differently than those of the data volume.  Thus, even if there were
no extra padding, sequence association of the embedding array may lead to
different storage of the data in serial memory than sequence association
of the data volume would.

\subsection{Restrictions}

The description of local data provided by \type {HPF\_SUBGRID\_INFO} assumes a
slightly restricted model over that chosen by HPF for local subprograms:
\begin{itemize}
\item Restriction 1.  Mapping of elements among processors

      The first restriction is that the subset of array elements that
      is mapped to any processor must be expressible as an array section
      of the global array.  This case includes most \type {*},
      \type {BLOCK}, and \type {CYCLIC} distributions but precludes
      most \type {CYCLIC(N)} distributions as well
      as peculiar alignments.

\item Restriction 2.  Organization of elements on a given processor

      The second restriction is that the global compiler must store
      the local subgrid by extending it in each direction in each
      dimension (perhaps by zero amounts) and sequence associating
      the larger, ``embedding array.''  This restriction applies
      \ital {only} in the case that the
      \type {LB\_EMBED}, \type {UB\_EMBED}, or \type {AXIS\_MAP}
      arguments to \type {HPF\_SUBGRID\_INFO} are used.  These arguments
      tell an F77\_LOCAL routine how a subgrid, passed from HPF with
      the \type {MAP\_TO(NO\_CHANGE)} attribute, is ordered by the global HPF compiler.

      In most cases, however, arrays will be passed to F77\_LOCAL subprograms
      with the default \type {MAP\_TO(F77\_ARRAY)} attribute so
      that \type {LB\_EMBED}, \type {UB\_EMBED}, and \type {AXIS\_MAP} will be of no
      interest and this second restriction will not apply.
      Instead, the local subgrid will be sequence associated in
      serial memory.  All padding due to embedding will be removed.
\end{itemize}

If the local programmer does not use \type {HPF\_SUBGRID\_INFO} or
\type {F77\_SUBGRID\_INFO}, then neither restriction applies at all.

\subsection{HPF-callable subgrid-inquiry subroutine}

The subgrid-inquiry subroutine is

\type {HPF\_SUBGRID\_INFO
(ARRAY, IERR, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)}
\begin{itemize}
\item \type {ARRAY} is an array of any type, size, shape, or
     distribution

\item \type {IERR}  is an error-status return variable, which
     is zero upon successful return and nonzero
     otherwise --- say, in the case that the appropriate
     restrictions are not satisfied

\item \type {DIM}   is an optional argument indicating the
     axis along which the parameters are desired
     --- if no axis is specified, parameters are
     returned for all axes

\item \type {LB}, \type {UB}, \type {STRIDE}
     The remaining arguments are all optional,
     \type {INTENT (OUT)}, integer arrays.  If a \type {DIM} argument
     is given, they are rank-one arrays of size
     \type {NUMBER\_OF\_PROCESSORS()} and \type {BLOCK} distribution.
     If no \type {DIM} argument is given, there is a
     second, collapsed axis whose extent is at least the rank
     of the global array.  The arguments \type {LB}, \type {UB}, and
     \type {STRIDE} use indices of the global array to return the
     lower bounds, upper bounds, and strides of the
     array sections that describe the local subgrids.

\item \type {LB\_EMBED}, \type {UB\_EMBED}, \type {AXIS\_MAP}
     These arguments are also optional arguments
     that should be declared like LB, UB, and STRIDE.
     They return the lower and upper bounds of the
     local embedding arrays using indices of the
     global array and the axes of the embedding array
     to which the axes of \type {ARRAY} are mapped.  Thus, they are of interest only
     when the programmer will pass the array using
     the \type {MAP\_TO(NO\_CHANGE)} attribute.
\end{itemize}

[{\em {Note}}:  It is unclear whether local programmers would prefer global indices
to be one-based or to be however they declare them in the global
caller.  In any case, this proposal is predicated on and follows
the convention established by HPF\_LOCAL that global indices should
be one-based.]

Consistent with the HPF model for local subprograms, it is
assumed that array axes are mapped independently to axes of the
rectangular processor grid.  Thus, values for \type {LB}, \type {UB},
and \type {STRIDE} should
be determined on an axis-by-axis basis.  Results should be the same
whether they are queried one-by-one, using a \type {DIM} argument, or all at
once.  In particular, if an axis does not have any index values mapped
to a particular processor, then \type {UB = LB - STRIDE} on that processor for
that axis.  Values for other axes should be determined independently.

If the array has no elements mapped to a particular processor,
the choices for \type {UB\_EMBED}, \type {LB\_EMBED}, and \type {AXIS\_MAP}
on that processor are somewhat
arbitrary.  There are certainly no guidelines within HPF ---
in contrast to the situation for for \type {LB}, \type {UB}, and \type {STRIDE} ---
since HPF only addresses which elements are mapped
to which processors, but not how those elements are locally organized on
a particular processor.  The only strict requirement is that the values
of \type {UB\_EMBED} and \type {LB\_EMBED}, given the value of \type {STRIDE},
should give the correct
overall size of the locally allocated subgrid, even if none of its elements
are actually used.  It is suggested that \type {UB\_EMBED}, \type {LB\_EMBED}, and
\type {AXIS\_MAP} continue to
reflect the axis-by-axis mapping of the array to processors and if no
memory is locally allocated due to a particular axis mapping, then
\type {UB\_EMBED = LB\_EMBED - STRIDE} for that axis.

\subsection{Example}

Consider, for example,

\begin{verbatim}
              REAL X(7,5)
        !HPF$ PROCESSORS P(2,2)
        !HPF$ DISTRIBUTE X(BLOCK,CYCLIC) ONTO P
\end{verbatim}

Then the subgrids on the various processors are X(1:4,1:5:2), X(5:7,1:5:2),
X(1:4,2:4:2), and X(5:7,2:4:2).  The subgrids can all written as array
sections of the global array, fulfilling the first restriction.  If one
passed X to the programmer's F77\_LOCAL routine using the default
\type {MAP\_TO(F77\_ARRAY)} attribute, one could call \type {HPF\_SUBGRID\_INFO}
to describe the local, sequence-associated subgrids.

If X were to be passed in to the programmer's F77\_LOCAL routine with the
\type {MAP\_TO(NO\_CHANGE)} attribute, then \type {HPF\_SUBGRID\_INFO} could
describe the local subgrids if the compiler used ``embedding arrays''.  As examples,

\begin{itemize}
\item It may allocate only enough room for the subgrids ---
     in this case, 4x3, 3x3, 4x2, and 3x2 elements per processor,
     respectively.

\item It may allocate same-sized subgrids on each processor.
     Then the data on each node is embedded in a larger array
     --- perhaps X(1:4,1:5:2), X(5:8,1:5:2), X(1:4,2:6:2), and
     X(5:8,2:6:2), respectively, or 4 x 3 elements per processor.

\item It may also allocate extra border cells for ``stencil''
     optimizations.  The allocated arrays on the respective
     processors might be X(0:5,1:5:2), X(4:9,1:5:2), X(0:5,2:6:2),
     and X(4:9,2:6:2), or (1+4+1) x 3 elements per processor.
\end{itemize}

All of these allocations fulfill the second restriction so long as these larger
embedding arrays are sequence associated in local, serial memory.  For example,
processor P(2,2) might have data values X(5:7,2:4:2) embedded in a storage array
X(4:9,2:6:2) that is sequence associated in local, serial memory, leading the
F77\_LOCAL routine to see

\begin{verbatim}
                         MAP_TO(NO_CHANGE)       MAP_TO(F77_ARRAY)

base address of X  ---->         *                     X(5,2)
                              X(5,2)                   X(6,2)
                              X(6,2)                   X(7,2)
                              X(7,2)                   X(5,4)
                                 *                     X(6,4)
                                 *                     X(7,4)
                                 *                        *
                              X(5,4)                      *
                              X(6,4)                      *
                              X(7,4)                      *
                                 *                        *
                                 *                        *
                                 *                        *
                                 *                        *
                                 *                        *
                                 *                        *
                                 *                        *
                                 *                        *
\end{verbatim}

\subsection{FORTRAN~77-callable subgrid-inquiry subroutine}

The locally callable version of \type {HPF\_SUBGRID\_INFO} is

\type {F77\_SUBGRID\_INFO
(ARRAY, IERR1, IERR2, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)}
\begin{itemize}
\item all arguments are required --- none is optional

\item \type {ARRAY} is a dummy argument passed in from global HPF

\item \type {IERR1} is zero upon successful determination of \type {LB}, \type {UB},
     and \type {STRIDE} and nonzero otherwise (e.g., subgrids are
     not expressible as array sections of the global array)

\item \type {IERR2} is zero upon successful determination of \type {LB\_EMBED}
     and \type {UB\_EMBED} and nonzero otherwise (e.g., the global
     compiler does not organize data locally by sequence
     associating an ``embedding'' array)

\item \type {DIM = -1} behaves the same as omitting the
     optional \type {DIM} argument in the global version

\item the subgrid parameters \type {LB}, \type {UB}, \type {STRIDE}, \type {LB\_EMBED},
     \type {UB\_EMBED}, and \type {AXIS\_MAP} should be declared as integer scalars
     if an axis is specified by the \type {DIM} argument or
     as rank-one integer arrays of size equal to at
     least the rank of the \type {ARRAY} if \type {DIM = -1}
\end{itemize}

\section{PROGRAMMING EXAMPLES}

\subsection{\type {MAP\_TO(F77\_ARRAY)}}

This example illustrates F77\_LOCAL programming using the default
\type {MAP\_TO(F77\_ARRAY)} attribute.

\subsubsection{HPF caller}

\begin{verbatim}
              program example

        ! declare the data array and a verification copy
              integer, parameter :: nx = 100, ny = 100
              real, dimension(nx,ny) :: x, y
        !hpf$ distribute(block,block) :: x, y

        ! the global sum will be computed
        ! by forming partial sums on the processors
              real partial_sum(number_of_processors())
        !hpf$ distribute partial_sum(block)

        ! local subgrid parameters are declared per processor
        ! for a rank-two array
              integer, dimension(number_of_processors(),2) ::
             & lb, ub, number
        !hpf$ distribute(block,*) :: lb, ub, number

        ! define interfaces

              interface

                extrinsic(f77_local) subroutine local1
             &   ( lb1, ub1, lb2, ub2, x )
                integer, dimension(:) :: lb1, ub1, lb2, ub2
                real x(:,:)
        !hpf$ distribute(block) :: lb1, ub1, lb2, ub2
        !hpf$ distribute(block,block) :: x
                end

                extrinsic(f77_local) subroutine local2(n,x,r)
                integer n(:)
                real x(:,:), r(:)
        !hpf$ distribute n(block)
        !hpf$ distribute x(block,block)
        !hpf$ distribute r(block)
                end

              end interface

        ! determine result using only global HPF

              ! initialize values
              forall (i=1:nx,j=1:ny) x(i,j) = i + (j-1) * nx
              ! determine and report global sum
              print *, 'global HPF result: ',sum(x)

        ! determine result using local subroutines

              ! initialize values ( assume stride = 1 )
              call HPF_subgrid_info( y, ierr, lb=lb, ub=ub )
              if (ierr.ne.0) stop 'error!'
              call local1( lb(:,1), ub(:,1), lb(:,2), ub(:,2), y )

              ! determine and report global sum
              number = ub - lb + 1
              call local2 ( number(:,1) * number(:,2) , y , partial_sum )
              print *, 'F77_LOCAL result #1 : ',sum(partial_sum)

              end
\end{verbatim}

\subsubsection{FORTRAN~77 callee}

\begin{verbatim}
              subroutine local1( lb1, ub1, lb2, ub2, x )

              real x ( lb1 : ub1 , lb2 : ub2 )

        ! get the global extent of the first axis
        ! this is an HPF_LOCAL type of inquiry routine with an ``F77_'' prefix
              call F77_global_size(nx,x,1)

        ! initialize elements of the array
              do j = lb2, ub2
                do i = lb2, ub2
                  x(i,j) = i + (j-1) * nx
                end do
              end do

              end

              subroutine local2(n,x,r)

        ! here, the correspondence to the global indices is not important
        ! only the total size of the subgrid is passed in
              real x(n)

              r = 0.
              do i = 1, n
                r = r + x(i)
              end do

              end
\end{verbatim}

\subsection{\type {MAP\_TO(NO\_CHANGE)}}

This example performs only the initialization of the above example.  It
illustrates use of the \type {MAP\_TO(NO\_CHANGE)} attribute and the
addressing of data in terms of ``embedding arrays.''

\subsubsection{HPF caller}

\begin{verbatim}
              program example

              integer, parameter :: nx = 100, ny = 100
              real, dimension(nx,ny) :: y
        !hpf$ distribute(block,block) :: y

        ! local subgrid parameters are declared per processor
        ! for a rank-two array
              integer, dimension(number_of_processors(),2) ::
             & lb, ub, lb_embed, ub_embed
        !hpf$ distribute(block,*) :: lb, ub, lb_embed, ub_embed

        ! define interfaces

              interface

                extrinsic(f77_local) subroutine local1(
             &   lb1, ub1, lb_embed1, ub_embed1,
             &   lb2, ub2, lb_embed2, ub_embed2, x )
                integer, dimension(:) ::
             &   lb1, ub1, lb_embed1, ub_embed1,
             &   lb2, ub2, lb_embed2, ub_embed2
                real, dimension(:,:), map_to(no_change) :: x
        !hpf$ distribute(block) :: lb1, ub1, lb_embed1, ub_embed1
        !hpf$ distribute(block) :: lb2, ub2, lb_embed2, ub_embed2
        !hpf$ distribute(block,block) :: x
                end

              end interface

        ! initialize values
        ! ( assume stride = 1 and no axis permutation )

              call HPF_subgrid_info( y, ierr,
             & lb=lb, lb_embed=lb_embed,
             & ub=ub, ub_embed=ub_embed)
              if (ierr.ne.0) stop 'error!'

              call local1(
             & lb(:,1), ub(:,1), lb_embed(:,1), ub_embed(:,1),
             & lb(:,2), ub(:,2), lb_embed(:,2), ub_embed(:,2), y )

              end
\end{verbatim}

\subsubsection{FORTRAN~77 callee}

\begin{verbatim}
              subroutine local1(
             & lb1, ub1, lb_embed1, ub_embed1,
             & lb2, ub2, lb_embed2, ub_embed2, x )

        ! the subgrid has been passed in its "embedded" form
              real x ( lb_embed1 : ub_embed1 , lb_embed2 : ub_embed2 )

        ! get the global extent of the first axis
        ! this is an HPF_LOCAL type of inquiry routine with an ``F77_'' prefix
              call F77_global_size(nx,x,1)

        ! otherwise, initialize elements of the array
        ! loop only over actual array elements
              do j = lb2, ub2
                do i = lb2, ub2
                  x(i,j) = i + (j-1) * nx
                end do
              end do

              end
\end{verbatim}

\section{APPENDIX A.  NEW HPF UTILITY SUBROUTINES}

\subsection{
\type {HPF\_SUBGRID\_INFO
(ARRAY, IERR, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)}
}
\begin{itemize}
\item Description.  Subgrid-inquiry subroutine.

\item \type {ARRAY}  is an array of any type, size, shape, or distribution.
  It is an \type {INTENT (IN)} argument.

\item \type {IERR}  is a scalar integer.  It is an INTENT (OUT) argument.
  Its return value is zero upon successful return and nonzero
  otherwise.  Errors result if local subgrids cannot be expressed
  as array sections of \type {ARRAY}.  If LB\_EMBED, UB\_EMBED, or AXIS\_MAP is specified,
  then errors also result if the compiler does not organize the
  local data in serial memory by sequence associating a larger
  ``embedding'' array.

\item \type {DIM} (optional)  is a scalar integer.  Is is an \type {INTENT (IN)}
  argument.

\item \type {LB}, \type {UB}, \type {STRIDE},
  \type {LB\_EMBED}, \type {UB\_EMBED}, \type {AXIS\_MAP}
  (all optional)  are
  integer arrays.  They are \type {INTENT (OUT)} arguments.  If
  \type {DIM} is specified, they are rank-one arrays of size
  \type {NUMBER\_OF\_PROCESSORS()} and \type {BLOCK} distribution.  If \type {DIM}
  is not specified, there is a second, collapsed axis
  whose extent is at least the rank of the global array.  The return
  values are the lower and upper bounds and strides of the
  array sections describing the local data (in terms of global indices),
  the lower and upper bounds of the embedding arrays (again, in terms of
  global indices), and the axes of the embedding arrays to
  which the axes of \type {ARRAY} are mapped.
\end{itemize}

\section{APPENDIX B.  NEW FORTRAN~77 UTILITY SUBROUTINES}

In all of the following:
\begin{itemize}
\item \type {ARRAY}  is some dummy argument passed in from
  the HPF caller corresponding to some global array or its
  local subgrid.  It is an \type {INTENT (IN)} argument.

\item \type {DIM}  is a scalar integer.  Is is an \type {INTENT (IN)} argument.
  This argument specifies a particular axis of the global
  array associated with \type {ARRAY} or, if \type {DIM = -1}, inquiry is
  for all axes.

\item An ``inquiry result'' is an \type {INTENT (OUT)} argument.  If \type {DIM = -1},
  it is a rank-one array of size equal to at least the rank of
  the global array associated with \type {ARRAY}, returning information
  associated with all axes.  If \type {DIM} is positive, the ``inquiry
  result'' is a scalar, returning information only for the axis
  indicated by \type {DIM}.

\item The arguments are defined in the same way as for the
  corresponding HPF or HPF\_LOCAL routines unless otherwise
  noted.
\end{itemize}

\subsection{
\type {F77\_SUBGRID\_INFO
(ARRAY, IERR1, IERR2, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF subroutine
  \type {HPF\_SUBGRID\_INFO}.

\item \type {IERR1}  is a scalar integer.  It is an \type {INTENT (OUT)} argument.
  Its return value is zero if \type {LB}, \type {UB}, and \type {STRIDE} were determined
  successfully and nonzero otherwise.

\item \type {IERR2}  is a scalar integer.  It is an \type {INTENT (OUT)} argument.
  Its return value is zero if \type {LB\_EMBED} and \type {UB\_EMBED} were
  determined successfully and nonzero otherwise.

\item \type {LB}, \type {UB}, \type {STRIDE},
  \type {LB\_EMBED}, \type {UB\_EMBED}, \type {AXIS\_MAP}
  are ``inquiry results'' of
  type integer.  They are the lower and upper bounds and strides
  of the array sections describing the local data (in terms of global indices),
  the lower and upper bounds of the embedding arrays (again, in terms of
  global indices), and the axes of the embedding arrays to which the
  axes of \type {ARRAY} are mapped.
\end{itemize}

\subsection{
\type {F77\_GLOBAL\_ALIGNMENT
(ALIGNEE, LB, UB, STRIDE, AXIS\_MAP, IDENTITY\_MAP, DYNAMIC, NCOPIES)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {GLOBAL\_ALIGNMENT}.  All but the first are \type {INTENT (OUT)}
  arguments whose return values are as specified by the
  corresponding HPF routine.

\item \type {ALIGNEE}  is a dummy argument passed in from global
  HPF.  It is an \type {INTENT (IN)} argument.

\item \type {LB}, \type {UB}, \type {STRIDE}, \type {AXIS\_MAP}  are
  integer arrays of rank one.  Their
  size must be at least equal to the rank of the global HPF array
  associated with \type {ALIGNEE}.

\item \type {IDENTITY\_MAP}, \type {DYNAMIC}  are scalar logicals.

\item \type {NCOPIES}  is a scalar integer.
\end{itemize}

\subsection{
\type {F77\_GLOBAL\_DISTRIBUTION
(DISTRIBUTEE, AXIS\_TYPE, AXIS\_INFO, PROCESSORS\_RANK, PROCESSORS\_SHAPE)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {GLOBAL\_DISTRIBUTION}.  All but the first are \type {INTENT (OUT)}
  arguments whose return values are as specified by the
  corresponding HPF routine.

\item \type {DISTRIBUTEE}  is a dummy argument passed in from global
  HPF.  It is an \type {INTENT (IN)} argument.

\item \type {AXIS\_TYPE}  is a \type {CHARACTER*9} array of rank one.  Its size must be
  at least equal to the rank of the global HPF array associated with
  \type {DISTRIBUTEE}.

\item \type {AXIS\_INFO}  is an integer array of rank one.  Its size must be at
  least equal to the rank of the global HPF array associated with
  \type {DISTRIBUTEE}.

\item \type {PROCESSORS\_RANK}  is an integer scalar.

\item \type {PROCESSORS\_SHAPE}  is an integer array of rank one.  Its size must
  be at least equal to the value returned by \type {PROCESSORS\_RANK}.
\end{itemize}

\subsection{
\type {F77\_GLOBAL\_TEMPLATE
(ALIGNEE, TEMPLATE\_RANK, LB, UB, AXIS\_TYPE, AXIS\_INFO, NUMBER\_ALIGNED, DYNAMIC)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {GLOBAL\_TEMPLATE}.  All but the first are \type {INTENT (OUT)} arguments
  whose return values are as specified by the corresponding HPF
  routine.

\item \type {ALIGNEE}  is a dummy argument passed in from global
  HPF.  It is an \type {INTENT (IN)} argument.

\item \type {TEMPLATE\_RANK}  is a scalar integer.

\item \type {LB}, \type {UB}, \type {AXIS\_INFO}  are integer arrays of rank one.
  Their size must
  be at least equal to the rank of the align-target to which the global
  HPF array associated with \type {ALIGNEE} is ultimately aligned.

\item \type {AXIS\_TYPE}  is a \type {CHARACTER*10} array of rank one.  Its size must be
  at least equal to the rank of the align-target to which the global
  HPF array associated with \type {ALIGNEE} is ultimately aligned.

\item \type {NUMBER\_ALIGNED}  is a scalar integer.

\item \type {DYNAMIC}  is a scalar logical.
\end{itemize}

\subsection{
\type {F77\_ABSTRACT\_TO\_PHYSICAL(ARRAY, INDEX, PROC)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {ABSTRACT\_TO\_PHYSICAL}.

\item \type {INDEX}  is a rank-one, \type {INTENT (IN)}, integer array.

\item \type {PROC}  is a scalar, \type {INTENT (OUT)}, integer.
\end{itemize}

\subsection{
\type {F77\_PHYSICAL\_TO\_ABSTRACT(ARRAY, PROC, INDEX)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {PHYSICAL\_TO\_ABSTRACT}.

\item \type {PROC}  is a scalar, \type {INTENT (IN)}, integer.

\item \type {INDEX}  is a rank-one, \type {INTENT (OUT)}, integer array.
\end{itemize}

\subsection{
\type {F77\_LOCAL\_TO\_GLOBAL(ARRAY, L\_INDEX, G\_INDEX)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {LOCAL\_TO\_GLOBAL}.

\item \type {L\_INDEX}  is a rank-one, \type {INTENT (IN)}, integer array.

\item \type {G\_INDEX}  is a rank-one, \type {INTENT (OUT)}, integer array.
\end{itemize}

\subsection{
\type {F77\_GLOBAL\_TO\_LOCAL(ARRAY, G\_INDEX, L\_INDEX, LOCAL, NCOPIES, PROCS)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL subroutine
  \type {GLOBAL\_TO\_LOCAL}.

\item \type {G\_INDEX}  is a rank-one, \type {INTENT (IN)}, integer array.

\item \type {L\_INDEX}  is a rank-one, \type {INTENT (OUT)}, integer array.

\item \type {LOCAL}  is a scalar, \type {INTENT (OUT)}, logical.

\item \type {NCOPIES}  is a scalar, \type {INTENT (OUT)}, integer.

\item \type {PROCS}  is a rank-one, integer array whose size is at
  least the number of processors that hold copies of the
  identified element.
\end{itemize}

\subsection{
\type {F77\_LOCAL\_BLKCNT(L\_BLKCNT, ARRAY, DIM, PROC)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL function
  \type {LOCAL\_BLKCNT}.

\item \type {L\_BLKCNT} is an ``inquiry result'' of type integer.

\item \type {PROC} is a scalar integer.  It must be a valid processor
  number or, if \type {PROC = -1}, the value returned by \type {F77\_MY\_PROCESSOR()}
  is implied.
\end{itemize}

\subsection{
\type {F77\_LOCAL\_LINDEX(L\_LINDEX, ARRAY, DIM, PROC)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL function
  \type {LOCAL\_LINDEX}.

\item \type {L\_LINDEX} is a rank-one, integer array of size equal to
  at least the value returned by \type {F77\_LOCAL\_BLKCNT()}.

\item \type {DIM} may not be -1.

\item \type {PROC} is a scalar integer.  It must be a valid processor
  number or, if \type {PROC = -1}, the value returned by \type {F77\_MY\_PROCESSOR()}
  is implied.
\end{itemize}

\subsection{
\type {F77\_LOCAL\_UINDEX(L\_UINDEX, ARRAY, DIM, PROC)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL function
  \type {LOCAL\_UINDEX}.

\item \type {L\_UINDEX} is a rank-one, integer array of size equal to
  at least the value returned by \type {F77\_LOCAL\_BLKCNT()}.

\item \type {DIM} may not be -1.

\item \type {PROC} is a scalar integer.  It must be a valid processor
  number or, if \type {PROC = -1}, the value returned by \type {F77\_MY\_PROCESSOR()}
  is implied.
\end{itemize}

\subsection{
\type {F77\_GLOBAL\_SHAPE(SHAPE, ARRAY)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL function
  \type {GLOBAL\_SHAPE}.

\item \type {SHAPE}  is a rank-one, integer array of size equal to at
  least the rank of the global array associated with \type {ARRAY}.
  Its return value is the shape of that global array.
\end{itemize}

\subsection{
\type {F77\_GLOBAL\_SIZE(SIZE, ARRAY, DIM)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL function
  \type {GLOBAL\_SIZE}.

\item \type {SIZE}  is a scalar integer equal to the extent of axis \type {DIM}
  of the global array associated with \type {ARRAY} or, if \type {DIM = -1},
  the total number of elements in that global array.
\end{itemize}

\subsection{
\type {F77\_SHAPE(SHAPE, ARRAY)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF intrinsic \type {SHAPE()},
  as it would behave as called from HPF\_LOCAL.

\item \type {SHAPE}  is a rank-one, integer array of size equal to at
  least the rank of the subgrid associated with \type {ARRAY}.
  Its return value is the shape of that subgrid.
\end{itemize}

\subsection{
\type {F77\_SIZE(SIZE, ARRAY, DIM)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF intrinsic \type {SIZE()},
  as it would behave as called from HPF\_LOCAL.

\item \type {SIZE}  is a scalar integer equal to the extent of axis \type {DIM}
  of the subgrid associated with \type {ARRAY} or, if \type {DIM = -1},
  the total number of elements in that subgrid.
\end{itemize}

\subsection{
\type {F77\_MY\_PROCESSOR(MY\_PROC)}
}
\begin{itemize}
\item Description.  FORTRAN~77-callable version of HPF\_LOCAL function
  \type {MY\_PROCESSOR}.

\item \type {MY\_PROC}  is a scalar, \type {INTENT (OUT)}, integer.  Its value is
  the identifying number of the physical processor from which
  this call is made.
\end{itemize}

\end{document}
---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------