Subject: hpff-doc: mapping-ext.tex

From: [email-masked] (Carl Offner)
Date: Fri, 18 Oct 1996 13:17:54 -0500 (CDT)

---------------------------------------------------------------------------
[email-masked] is a mailing list for HPF 2.0 language specification
authors and editors.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------
A small change in conformance with one of the changes to mapping-subr.tex.

% File: mapping-ext.tex

% Contents:
% Approved Extensions for data mapping for HPF 2.0 document, including
%        GenBlock
%        indirect
%        range
%        shadow
%        subsets
%        derived types
%        more on pointers.

% Revision history:
% May-10-96        Created by Charles Koelbel, Rice University
%                (From HPF 2.0 proposals)
% May-22-96        Modified by Piyush Mehrotra to include all of
%               the above topics
% Aug-8-96         Modified by Piyush Mehrotra:
%               modified subsection on remapping across procedure boundaries
%               modified the section on pointers based on proposals passed
%                       in the July meeting
%               added clarifications and examples to the range section
% Oct-18-96        Modified by Carl Offner:
%               added a few words to the beginning of item 2 in the
%               definition of ``specialization'' in conformance with a
%               similar change in the mapping-subr.tex section.

\chapter{Approved Extensions for Data Mapping}
\label{ch-mapping-ext}

This section describes a set of data mapping features which extend
the capabilities provided by the base set as described in
Section~\ref{ch-mapping-base}. These extensions can be
divided into two categories. The first set of extensions
provide the user greater control over the mapping of the data.
These include directives for dynamic remapping of data
which allow the user to redistribute and realign at runtime data which
has been declared {\tt DYNAMIC}.
The {\tt ONTO} clause used in the {\tt DISTRIBUTE} directive is extended
to allow direct distribution to subsets of processors.
Explicit mapping of pointers and components of derived
types are also introduced.
Two new distributions are included, the {\tt GEN\_BLOCK}
distribution which generalizes the block distribution and
the {\tt INDIRECT} which allows the mapping of individual array
elements to be specified through a mapping array.

The programmer can use the second set of extensions to provide the
compiler with information useful for generating efficient
code. This category includes the \texttt{RANGE} directive, which
allows the user to specify the range of distributions that
a dynamically distributed array, a pointer, or a formal
argument may have. The \texttt{SHADOW} directive allows the user
to specify the amount of additional space required on a
processor to accommodate non-local elements in a
nearest-neighbor computation.

Since this section deals with extensions, we repeat some of the
sections of Sections~\ref{ch-mapping-base}
and~\ref{ch-mapping-subr}, providing new rules and extending old ones
where necessary.
In particular, subsections~\ref{ext:PartialOrderOnMaps},
\ref{ext:ExplicitInterfaces} and
\ref{ext:ProcChars} extend t eh corresponding subsections
in Section~\ref{ch-mapping-base} based on the approved
extensions described here.

%When extending a rule we use \textit{extended-xx} as the new rule name
%with the understanding that it will replace the old rule name,
%\textit{xx}, everywhere.

\section{Extended Model}

The fundamental model for allocation of data to abstract
processors still remains a two-level mapping as described in
Section~\ref{ch-mapping-base}. However, it is extended to
allow the dynamic remapping of the data objects as
illustrated by the following diagram:

\begin{center}

\setlength{\unitlength}{0.01in}
\begin{picture}(600,240)(0,30)
\thicklines
\put(100,150){\circle{50}}
\put(242,150){\circle{50}}
\put(383,150){\circle{50}}
\put(525,150){\circle{50}}
\put(125,150){\vector(1,0){92}}
\put(267,150){\vector(1,0){91}}
\put(408,150){\vector(1,0){92}}
\put(56,190){\shortstack{Arrays or\strut\\other
objects\strut}}
\put(192,190){\shortstack{Group of\strut\\aligned
objects\strut}}
\put(328,190){\shortstack{Abstract\strut\\processors as
a\strut
                \\user-declared\strut\\Cartesian
mesh\strut}}
\put(494,190){\shortstack{Physical\strut\\processors\strut}}
\put(135,50){\shortstack{{\tt ALIGN}\strut\\(static) or\strut
                \\{\tt REALIGN}\strut\\(dynamic)\strut}}
\put(265,50){\shortstack{{\tt DISTRIBUTE}\strut\\(static) or\strut
                \\{\tt REDISTRIBUTE}\strut\\(dynamic)\strut}}
\put(400,50){\shortstack{Optional\strut\\implementation-\strut
                \\dependent\strut\\directive\strut}}
\end{picture}

\end{center}

Thus, objects can now be remapped at execution time using
the executable directives {\tt REALIGN} and {\tt REDISTRIBUTE}
Any object that is the root of
an alignment tree, i.e., is not explicitly aligned to
another object, can be explicitly redistributed.
Redistributing such an object causes all objects ultimately
aligned with it to be also redistributed so as to maintain the
alignment relationships.

Any object that is not a root
of an alignment tree can be explicitly realigned but not explicitly
redistributed. Such a realignment does not change the
mapping of any other object.
Note that such remapping of data may require communication
among the processors.

By analogy with the Fortran 90 {\tt ALLOCATABLE} attribute, HPF includes
the attribute {\tt DYNAMIC}.  It is not permitted
to {\tt REALIGN} an array that has not been declared {\tt DYNAMIC}.
Similarly, it is not permitted to {\tt REDISTRIBUTE} an array or template
that has not been declared {\tt DYNAMIC}.

Saved local variables, variables in common, and variables accessed by
use association must not be implicitly remapped - e.g., by having
variable distribution formats or being aligned with entities having
variable distribution formats.  Of these three categories of
variables, only variables accessed by use association may have the
{\tt DYNAMIC} attribute.

\section{Syntax of Attributed Forms of the New Directives}

As in the case of other mapping directives, the executable directives
{\tt REALIGN} and {\tt REDISTRIBUTE} also come in two forms (statement
form and attribute form) but may not be combined with other attributes
in a single directive.  The \texttt{RANGE} and \texttt{SHADOW}
attributes may be combined with other attributes in a single
directive.

                                                                        \BNF
combined-attribute-extended \IS  ALIGN align-attribute-stuff
                           \OR  DISTRIBUTE dist-attribute-stuff
                           \OR  DYNAMIC
                           \OR  INHERIT
                           \OR  TEMPLATE
                           \OR  PROCESSORS
                           \OR  DIMENSION ( explicit-shape-spec-list )
                           \OR  RANGE range-attr-stuff-list
                           \OR  SHADOW shadow-attr-stuff
                           \OR  ACTIVE

\FNB
\begin{constraints}
\item The \texttt{ACTIVE} attribute may only be applied to a
processors arrangement.
\end{constraints}

The \texttt{ACTIVE} directive is discussed in Section~\ref{ch-parallel-ext}
while the rest are discussed below.

\section{REDISTRIBUTE Directive}
\label{REDISTRIBUTE-SECTION}

The {\tt REDISTRIBUTE} directive is similar to the {\tt DISTRIBUTE}
directive but is considered executable.  An array (or template) may be
redistributed at any time, provided it has been declared {\tt DYNAMIC}
(see Section \ref{DYNAMIC-SECTION}).  Any other arrays currently
ultimately aligned with an array (or template) when it is redistributed
are also remapped to reflect the new distribution, in such a way as to
preserve alignment relationships (see Section \ref{ALIGN-SECTION}).
(This can require a lot of computational and communication effort at
run time; the programmer must take care when using this feature.)

The {\tt DISTRIBUTE} directive may appear only in the {\it specification-part}
of a scoping unit.  The {\tt REDISTRIBUTE} directive may appear
only in the {\it execution-part} of a scoping unit.  The principal
difference between {\tt DISTRIBUTE} and {\tt REDISTRIBUTE} is
that {\tt DISTRIBUTE} must contain only a {\it specification-expr} as the
argument to a
distribution format, e.g.,
{\tt BLOCK} or {\tt CYCLIC}, whereas in {\tt REDISTRIBUTE}
such an argument may be any integer expression.  Another difference is that
{\tt DISTRIBUTE} is an attribute, and so can be combined with other attributes
as part of a {\it combined-directive}, whereas {\tt REDISTRIBUTE} is not an
attribute
(although a {\tt REDISTRIBUTE} statement may be written in the
style of attributed syntax, using ``{\tt ::}'' punctuation).

Formally, the syntax of the {\tt DISTRIBUTE} and
{\tt REDISTRIBUTE} directives is:

                                                                        \BNF
redistribute-directive     \IS  REDISTRIBUTE distributee dist-directive-stuff
                   \OR  REDISTRIBUTE dist-attribute-stuff :: 
 distributee-list
                                                                        \FNB
\begin{constraints}

\item  A {\it distributee} that appears in a {\tt REDISTRIBUTE} directive
must have the {\tt DYNAMIC} attribute (see Section \ref{DYNAMIC-SECTION}).

\item Neither the {\it dist-format-clause} nor the {\it dist-target}
in a {\tt REDISTRIBUTE} may begin with ``{\tt *}''.

\end{constraints}

If a range directive (see Section~\ref{range}) has been used to restrict
the set of distribution formats allowed for a {\it distributee}, then
the new mapping should match one of the formats specified in the range
directive.

A {\tt REDISTRIBUTE} directive must not cause any data object
associated with the {\it distributee} via storage association ({\tt
COMMON} or {\tt EQUIVALENCE}) to be mapped such that storage units of
a scalar data object are split across more than one abstract
processor.  See Section \ref{sequence} for further discussion of
storage association.

The statement form of a {\tt REDISTRIBUTE} directive
may be considered an abbreviation for an attributed form that
happens to mention only one {\it distributee}; for example,
                                                                        \CODE
!HPF$ REDISTRIBUTE distributee ( dist-format-list ) ONTO dist-target
                                                                        \EDOC
is equivalent to
                                                                        \CODE
!HPF$ REDISTRIBUTE ( dist-format-list ) ONTO dist-target :: distributee
                                                                        \EDOC

\section{REALIGN Directive}
\label{REALIGN-SECTION}

The {\tt REALIGN} directive is similar to the {\tt ALIGN} directive but
is considered executable.  An array (or template) may be realigned at
any time, provided it has been declared {\tt DYNAMIC} (see Section
\ref{DYNAMIC-SECTION}) Unlike redistribution (see Section
\ref{DISTRIBUTE-SECTION}), realigning a data object does not cause any
other object to be remapped.  (However, realignment of even a single
object, if it is large, could require a lot of computational and
communication effort at run time; the programmer must take care when
using this feature.)

The {\tt ALIGN} directive may appear only in the {\it
specification-part} of a scoping unit.  The {\tt REALIGN} directive is
similar but may appear only in the {\it execution-part} of a scoping
unit.  The principal difference between {\tt ALIGN} and {\tt REALIGN}
is that {\tt ALIGN} must contain only a {\it specification-expr} as a
{\it subscript} or in a {\it subscript-triplet}, whereas in {\tt
REALIGN} such subscripts may be any integer expressions.  Another
difference is that {\tt ALIGN} is an attribute, and so can be combined
with other attributes as part of a {\it combined-directive}, whereas
{\tt REALIGN} is not an attribute (although a {\tt REALIGN} statement
may be written in the style of attributed syntax, using ``{\tt ::}''
punctuation).

Formally, the syntax of {\tt REALIGN} is as follows:

                                                                        \BNF
realign-directive          \IS  REALIGN alignee align-directive-stuff
                           \OR  REALIGN align-attribute-stuff :: alignee-list
                                                                        \FNB
\begin{constraints}

\item  Any {\it alignee} that appears in a {\tt REALIGN} directive
must have the {\tt DYNAMIC} attribute (see Section \ref{DYNAMIC-SECTION}).

\item If the {\tt align-target} specified in the {\tt align-with-clause}
has the {\tt DYNAMIC} attribute, then each {\tt alignee} must also have the
{\tt DYNAMIC} attribute.

\item An object may not have both the {\tt INHERIT} attribute and
the {\tt ALIGN} attribute.  (However, an object with the {\tt INHERIT}
attribute may appear as an {\it alignee} in a {\tt REALIGN} directive,
provided that it does not appear as a {\it distributee} in a {\tt DISTRIBUTE}
or {\tt REDISTRIBUTE} directive.)
\end{constraints}

If a range directive (see Section~\ref{range}) has been used to restrict
the set of distribution formats allowed for an {\it alignee}, then
the new mapping should match one of the formats specified in the range
directive.

\section{DYNAMIC Directive}
\label{DYNAMIC-SECTION}

The {\tt DYNAMIC} attribute specifies that an object may be dynamically
realigned or redistributed.
                                                                        \BNF
dynamic-directive      \IS  DYNAMIC  alignee-or-distributee-list

alignee-or-distributee \IS  alignee
                       \OR  distributee
                                                                        \FNB

\begin{constraints}

\item An object in {\tt COMMON} may not be declared {\tt DYNAMIC} and
may not be aligned to an object (or template) that is {\tt DYNAMIC}.
(To get this kind of effect,  Fortran 90 modules must be used instead
of {\tt COMMON} blocks.)

\item An object with the {\tt SAVE} attribute may not be declared {\tt DYNAMIC}
and may not be aligned to an object (or template) that is {\tt DYNAMIC}.

\end{constraints}

A {\tt REALIGN} directive may not be applied to an {\it alignee} that does not
have the {\tt DYNAMIC} attribute.  A {\tt REDISTRIBUTE} directive may not be
applied to a {\it distributee} that does not have the {\tt DYNAMIC} attribute.

A {\tt DYNAMIC} directive may be combined with other directives, with the
attributes stated in any order, consistent with the Fortran
90 attribute syntax.

Examples:
                                                                        \CODE
!HPF$ DYNAMIC A,B,C,D,E
!HPF$ DYNAMIC:: A,B,C,D,E
!HPF$ DYNAMIC, ALIGN WITH SNEEZY:: X,Y,Z
!HPF$ ALIGN WITH SNEEZY, DYNAMIC:: X,Y,Z
!HPF$ DYNAMIC, DISTRIBUTE(BLOCK, BLOCK) :: X,Y
!HPF$ DISTRIBUTE(BLOCK, BLOCK), DYNAMIC :: X,Y
                                                                        \EDOC
The first two examples mean exactly the same thing.
The next two examples mean exactly the same second thing.
The last two examples mean exactly the same third thing.

The three directives
                                                                        \CODE
!HPF$ TEMPLATE A(64,64),B(64,64),C(64,64),D(64,64)
!HPF$ DISTRIBUTE(BLOCK, BLOCK) ONTO P:: A,B,C,D
!HPF$ DYNAMIC A,B,C,D
                                                                        \EDOC
may be combined into a single directive as follows:
                                                                        \CODE
!HPF$ TEMPLATE, DISTRIBUTE(BLOCK, BLOCK) ONTO P,   &
!HPF$   DIMENSION(64,64),DYNAMIC :: A,B,C,D
                                                                        \EDOC

An {\tt ALLOCATABLE} object may also be given the {\tt DYNAMIC} attribute.
If an {\tt ALLOCATE} statement is immediately followed by {\tt REDISTRIBUTE}
and/or {\tt REALIGN} directives,
the meaning in principle is that the array is first created
with the statically declared mapping, if any, then immediately remapped.
In practice there is an obvious optimization: create the array
in the processors to which it is about to be remapped, in a single
step.  HPF implementors are strongly encouraged to implement this
optimization and HPF programmers are encouraged to rely upon it.
Here is an example:
                                                                       \CODE
      REAL,ALLOCATABLE(:,:) :: TINKER, EVERS
!HPF$ DYNAMIC :: TINKER, EVERS
      REAL, ALLOCATABLE :: CHANCE(:)
!HPF$ DISTRIBUTE(BLOCK),DYNAMIC :: CHANCE
      ...
      READ 6,M,N
      ALLOCATE(TINKER(N*M,N*M))
!HPF$ REDISTRIBUTE TINKER(CYCLIC, BLOCK)
      ALLOCATE(EVERS(N,N))
!HPF$ REALIGN EVERS(:,:) WITH TINKER(M::M,1::M)
      ALLOCATE(CHANCE(10000))
!HPF$ REDISTRIBUTE CHANCE(CYCLIC)
                                                                       \EDOC
While {\tt CHANCE} is by default always allocated with a {\tt BLOCK}
distribution, it should be possible for a compiler to notice that it will
immediately be remapped to a {\tt CYCLIC} distribution.  Similar remarks
apply to {\tt TINKER} and {\tt EVERS}.  (Note that {\tt EVERS}
is mapped in a thinly-spread-out manner onto {\tt TINKER};
adjacent elements of {\tt EVERS} are mapped to elements of
{\tt TINKER} separated by a stride {\tt M}.  This thinly-spread-out
mapping is put in the lower left corner of {\tt TINKER},
because {\tt EVERS(1,1)} is mapped to {\tt TINKER(M,1)}.)

\section{Remapping and Subprogram Interfaces}
\label{DYNAMIC-DUMMY-SECTION}

If the dummy argument of any subprogram has the {\tt DYNAMIC} attribute,
then an explicit interface is required for the subprogram
(see subsection~\ref{ext:ExplicitInterfaces}).
The rules on the interaction of the {\tt REALIGN} and {\tt
REDISTRIBUTE} directives with a subprogram argument interface are:
\begin{enumerate}

\item A dummy argument may be declared {\tt DYNAMIC}.  However, it is subject
to the general restrictions concerning the use of the name of an
array to stand for its associated template.

The effect of any redistribution of the dummy after the procedure returns
to the caller is dependent on the attribute of the actual argument.
If the actual argument associated with the dummy has also been declared
{\tt DYNAMIC}, then any explicit remapping of the dummy is visible in the
caller after the procedure returns.
If a range directive (see Section~\ref{range}) has been used to restrict
the set of distribution formats allowed for
the actual argument, then
the new mapping should match one of the formats specified in the range
directive.

If the actual argument associated with the dummy has not been declared
{\tt DYNAMIC} then the original mapping of the actual has to be restored
on return.
When the subprogram returns
and the caller resumes execution, all objects accessible to the caller
after the call which are not declared
{\tt DYNAMIC},  are mapped exactly as they were before the call.

\item If an array or any section thereof is accessible by two or more
paths, it is not HPF-conforming to remap it through any of those paths.  For
example, if an array is passed as an actual argument, it is forbidden
to realign that array, or to redistribute an array or template to which
it was aligned at the time of the call, until the subprogram has
returned from the call.  This prevents nasty aliasing problems.  An
example:

                                                                        \CODE
      MODULE FOO
      REAL A(10,10)
!HPF$ DYNAMIC ::  A
      END

      PROGRAM MAIN
      USE FOO
      CALL SUB(A(1:5,3:9))
      END

      SUBROUTINE SUB(B)
      USE FOO
      REAL B(:,:)
!HPF$ DYNAMIC ::  B
      ...
!HPF$ REDISTRIBUTE A            !Nonconforming
      ...
      END
                                                                        \EDOC
Situations such as this are forbidden, for the same reasons that an assignment
to {\tt A} at the statement marked ``Nonconforming'' would also be forbidden.
In general, in {\it any} situation where assignment to a variable
would be nonconforming by reason of aliasing, remapping of that variable
by an explicit {\tt REALIGN} or {\tt REDISTRIBUTE} directive is also forbidden.

Note that it is legal to remap a host-associated or use-associated variable
in a subprogram if it has been declared {\tt DYNAMIC} and is accessible
only through a single path. Such remappings stay in effect even after
the subprogram has returned to its caller.

\end{enumerate}

%----------------------------------------------------------------------------
% Mapping to subsets proposal
%--------------------------------------------------------------------------
\section{Mapping to Processor Subsets}
\label{mapping-proc-subset}

This extension allows  objects  to  be  directly  distributed  to  processor
subsets  by allowing a processor subset to be specified
where  a  processor  could  be  named,  e.g.,  in  a   {\tt DISTRIBUTE}
directive.   The  specified subset must be a proper subset of the
named processor arrangement.

Formally the syntax of the extended {\it dist-target} is as
follows:

                                                                        \BNF
extended-dist-target    \IS processor-name [( section-subscript-list )]
                        \OR * processor-name [( section-subscript-list )]
                        \OR *
                                                                        \FNB
\begin{constraints}

\item The {\it section-subscript}s in the
{\it section-subscript-list} may not be
{\it vector-subscript}s and are restricted to be
either {\it subscript}s or {\it subscript-triplet}s.

\item In the {\it section-subscript-list},
the number of {\it section-subscript}s
must equal the rank of the {\it processor-name}.

\end{constraints}

                                                                        \CODE
!Example 1
!HPF$ PROCESSORS P(10)
        REAL A(100)
!HPF$ DISTRIBUTE A(BLOCK) ONTO P(2:5)

!Example 2
!HPF$ PROCESSORS Q(10,10)
        REAL A(100,100)
!HPF$ DISTRIBUTE B(BLOCK,BLOCK) ONTO Q(5:10,5:10)
                                                                        \EDOC

In Example 1, the array A is distributed by block across the
processors P(2) to P(5) while in the second example, the array B
is distributed across the lower right quadrant of the processor array Q.

\begin{users}
This extension is most useful in conjunction with the
tasking construct, see Section~\ref{sec-tasking}, which allows
multiple independent phases of a computation to execute simultaneously
on different subsets of processors. A similar situation arises when
the code uses multiple data structures which can be computed in
parallel where the computation on each individual object also
exhibits parallelism, e.g., the multiple blocks in
a multi-block grid used in some fluid dynamics calculation. Here, the
individual blocks have to be distributed over subsets of processors
to exploit both levels of parallelism.
\end{users}

\section{Pointers}
\label{POINTERS-SECTION}

As an approved extension to HPF, pointers can be explicitly mapped.
Formally, this implies that the constraints that a {\it distributee}
and an {\it alignee} may not have the {\tt POINTER} attribute as
stated in Sections~\ref{DISTRIBUTE-SECTION} and~\ref{ALIGN-SECTION}
respectively, have to be removed.

As in the case of an allocatable object, the mapping specification
for a pointer does not take effect immediately but plays a role when
the pointer becomes pointer associated with a target either through allocation
or through pointer assignment.

When a pointer with an explicit mapping is used in an {\tt ALLOCATE}
statement, the data is allocated with the specified mapping.

For example:
                                                                        \CODE
      REAL, POINTER, DIMENSION(:) :: A, B
!HPF$ ALIGN B(I) WITH A(I)
!HPF$ DISTRIBUTE A(BLOCK)
      ...
      ALLOCATE(A(100))
      ALLOCATE(B(50))
      ...
      ALLOCATE(B(200))                  ! Nonconforming
                                                                        \EDOC

Pointer {\it A} is declared to have a {\tt BLOCK} distribution while
pointer {\tt B} is declared to be identically aligned with {\it A}.
When {\it A} is allocated, it is created with a block distribution.
When {\it B} is allocated, it is aligned with the first 50 elements of
{\it A}. Note, that the allocation statements may not occur in the
opposite order, since an object may be aligned to another only if it
has already been created or allocated.  Also, the second allocation
for {\it B} is nonconforming, since a larger object, {\it B} here,
cannot be aligned with a smaller object, {\it A} in this case.

A pointer \texttt{P} with an explicit mapping can be pointer
associated with a target \texttt{T} through a pointer assignment
statement under the following conditions:

\begin{enumerate}

\item The mapping of \texttt{T} is a specialization of the mapping of
  \texttt{P}; and

\item If \texttt{P} is explicitly aligned, its ultimate align target
  has a fully-specified non-transcriptive distribution; and

\item \label{PandTdynamic} \texttt{P} and \texttt{T} are either both
  \texttt{DYNAMIC} or neither is.

\end{enumerate}

Here are some examples:
                                                                        \CODE
      REAL, POINTER, DIMENSION(:,:) :: P
!HPF$ DISTRIBUTE P(BLOCK,BLOCK)
      REAL, TARGET, DIMENSION (100, 100) ::  B, C, D
!HPF$ DISTRIBUTE B(BLOCK, BLOCK)
!HPF$ DISTRIBUTE C(BLOCK, CYCLIC)
      ...
      P => B                 ! Conforming
      P => B(1:50, 1:50)     ! Nonconforming: target must be  a whole array.
      P => C                 ! Nonconforming: the distribution in the
                             ! second dimension does not match
      P => D                 ! Nonconforming: D is not explicitly mapped
      ...
                                                                        \EDOC
The following pointer assignment is valid even though no processor
arrangement is specified for the pointer; in this case, the mapping of
\texttt{B} is a specialization of the mapping of \texttt{P}:
                                                                        \CODE
      REAL, POINTER, DIMENSION(:) :: P
      REAL, TARGET, DIMENSION(100) ::  B
!HPF$ DISTRIBUTE P(BLOCK)
!HPF$ DISTRIBUTE (BLOCK) ONTO P :: B
      ...
      A => B                 ! Conforming
      ...
                                                                        \EDOC

                                                                        \CODE
      REAL, POINTER, DIMENSION(:) :: P
!HPF$ DISTRIBUTE * :: P
      REAL, TARGET, DIMENSION(100) ::  B, C
!HPF$ DISTRIBUTE B(BLOCK), C(CYCLIC)
      ...
      P => B                 ! Conforming
      P => C                 ! Conforming
      P => C(1:50)           ! Nonconforming: target must be a whole array
      ...
                                                                        \EDOC

Here, the {\tt *} is used to indicate a transcriptive distribution for the
pointer {\it A} and thus it can be pointer associated with both
targets {\it B} and {\it C} distributed by {\tt BLOCK}
and {\tt CYCLIC} respectively.  However, it still cannot be used to
point to an array section such as \texttt{C(1:50)}.  To do that, the
pointer must have the \texttt{INHERIT} attribute:

                                                                        \CODE
      REAL, POINTER, DIMENSION(:) :: P
!HPF$ INHERIT :: P
      REAL, TARGET, DIMENSION(100) ::  B, C
!HPF$ DISTRIBUTE B(BLOCK), C(CYCLIC)
      ...
      P => B                 ! Conforming
      P => C                 ! Conforming
      P => C(1:50)           ! Conforming
      ...
                                                                        \EDOC

To allow pointers to have transcriptive distributions, we have
to change the constraint for {\it dist-format-clause} as specified
in Section~\ref{DISTRIBUTE-SECTION}, to read as follows:

\begin{constraints}
\item If either the {\it dist-format-clause} or the {\it dist-target}
in a {\tt DISTRIBUTE} directive begins with ``{\tt *}''
then every {\it distributee} must be a dummy argument,
{\em
except if the {\em distributee}
has the {\tt POINTER} attribute.}
\end{constraints}

The constraint for {\it align-spec} as specified
in Section~\ref{ALIGN-SECTION}, should be changed to read as follows:

\begin{constraints}
\item If the {\it align-spec}
in an {\tt ALIGN} directive begins with ``{\tt *}''
then every {\it alignee} must be a dummy argument,
{\em
except if the {\em alignee}
has the {\tt POINTER} attribute.}
\end{constraints}

The constraint for {\it inheritee} as specified
in Section~\ref{INHERIT-SECTION}, should be changed to read as follows:

\begin{constraints}
\item An {\it inheritee} must be a dummy argument,
{\em
except if the {\em alignee}
has the {\tt POINTER} attribute.}
\end{constraints}

When pointers with such transcriptive mappings
are used in an {\tt ALLOCATE} statement,
the compiler may choose any arbitrary mapping for the allocated data.
A range declaration (see Section~\ref{range}) can be used to restrict
the set of distribution formats.

If a pointer has the \texttt{DYNAMIC} attribute, then any target
associated with the pointer (which must therefore also have the
\texttt{DYNAMIC} attribute) may be remapped using a
\texttt{REDISTRIBUTE} or \texttt{REALIGN} statement under the
following restriction:

A pointer may be used in {\tt REALIGN} and {\tt REDISTRIBUTE} as an
{\it alignee}, {\it align-target}, or {\it distributee} if and only if
it is currently associated with a whole array, not an array section.

Note that when an object is remapped, the new mapping is
visible through any pointer that may be associated with the object.

\subsection{Pointers and Subprograms}

If a dummy pointer argument is not explicitly mapped, then the
actual pointer argument must not also be explicitly mapped.

If a dummy pointer argument has an explicit mapping, then the actual
argument must follow the rules for pointer assignment as stated above,
with one exception:  If the actual argument has the \texttt{DYNAMIC}
attribute, it is not necessary that the corresponding dummy argument
have the \texttt{DYNAMIC} attribute.  That is, item~\ref{PandTdynamic}
on page~\pageref{PandTdynamic} is weakened to

\begin{enumerate}
\item[3.] If a dummy pointer argument has the \texttt{DYNAMIC}
  attribute, then the corresponding actual argument must also have the
  \texttt{DYNAMIC} attribute.
\end{enumerate}

Again a range declaration (see Section~\ref{range}) can be used to
restrict the set of distribution formats of the actual.

A dummy pointer argument may have the \texttt{DYNAMIC} attribute.  In
this case, the actual pointer argument must also have the
\texttt{DYNAMIC} argument. The target associated with the dummy
argument may be redistributed under the restrictions stated in the
last subsection.  Following Fortran rules, if the actual is also
visible (through host- or use-association), the target may be
redistributed only through the dummy argument.  If the dummy argument
is redistributed, then the actual argument has the new mapping on
return from the procedure.  In such a case, the new mapping must match
the range restrictions (if any) of the actual.

\section{Mapping of Derived Type Components}
\label{derived-type}

An {\tt ALIGN}, {\tt DISTRIBUTE}, or {\tt DYNAMIC} directive may appear
within a
{\it derived-type-def} wherever a {\it component-def-stmt} may appear.  Every
{\it alignee} or {\it distributee} within such a directive must be the name of
a component defined within that {\it derived-type-def}.
To allow mapping of the structure components, the rules have to be extended
as follows:

                                                                        \BNF

distributee-extended    \IS  object-name
                        \OR  template-name
                        \OR  component-name
                        \OR  structure-component
                                                                        \FNB

A derived type is said to be an {\it explicitly mapped type}
if any of its components is explicitly mapped or if any of its
components is of an explicitly mapped type.

\begin{constraints}
\item A component of a derived type may be explicitly distributed
only if the type of the component is not an explicitly mapped type.

\item A variable or array of a derived type may be explicitly
distributed only if the derived type is not an explicitly mapped type.

\item A {\it distributee} in a {\tt DISTRIBUTE} directive may not be a
{\it structure-component}.

\item A {\it distributee}
in a {\tt DISTRIBUTE} directive which occurs in a {\it derived-type-def}
must be the {\it component-name} of a component of the derived type.

\item A {\it component-name} may occur as a {\it distributee}
in a {\it DISTRIBUTE} directive occuring within the derived type definition
only.

\item A {\it distributee} that is a {\it structure-component} may occur
only in a {\tt REDISTRIBUTE} directive
and every {\it part-ref}
except the rightmost must be scalar (rank zero).
The rightmost {\it part-name}
in the {\it structure-component} must have the {\tt DYNAMIC} attribute.
\end{constraints}

                                                                        \BNF
alignee-extended        \IS  object-name
                        \OR  component-name
                        \OR  structure-component
                                                                        \FNB

\begin{constraints}
\item A component of a derived type may be explicitly aligned
only if the type of the component is not an explicitly mapped type.

\item A variable or array of a derived type may be explicitly
aligned only if the derived type is not an explicitly mapped type.

\item An {\it alignee} in an {\tt ALIGN} directive may not be a
{\it structure-component}.

\item An {\it alignee}
in a {\tt ALIGN} directive which occurs in a {\it derived-type-def}
must be the {\it component-name} of a component of the derived type.

\item A {\it component-name} may occur as an {\it alignee}
in an {\it ALIGN} directive occuring within the derived type definition
only.

\item An {\it alignee} that is a {\it structure-component} may occur
only in a {\tt REALIGN} directive
and every {\it part-ref}
except the rightmost must be scalar (rank zero).
The rightmost {\it part-name} in the {\it structure-component} must
have the {\tt DYNAMIC} attribute.

\end{constraints}

                                                                        \BNF
align-target-extended   \IS  object-name
                        \OR  template-name
                        \OR  component-name
                        \OR  structure-component
                                                                        \FNB

\begin{constraints}

\item A {\it component-name} may appear as an align target
only in an {\it ALIGN} directive occuring within the derived type definition.

\item A {\it component-name} may occur as an {\it align-target}
in an {\it ALIGN} directive occuring within the derived type definition
only.

\item In an {\it align-target} that is a {\it structure-component}
every {\it part-ref}
except the rightmost must be scalar (rank zero).
\end{constraints}

The above constraints imply that components of derived type can be
mapped within the derived type definition itself such that
when any objects of that type are created the components
will be created with the specified mapping.

An align directive inside a derived type definition may
align a component of the derived type
with another component of the same derived
type or with another object.
A structure component can be used as a target to align other objects
including components of derived types.

If a component of a derived type has been declared {\tt DYNAMIC},
then the component of an object of the type may be
remapped using the {\tt REDISTRIBUTE} or {\tt REALIGN} directive.

Examples:

                                                                        \CODE
      TYPE SIMPLE
        REAL S(100)
!HPF$   DISTRIBUTE S(BLOCK)
      END TYPE SIMPLE

!HPF$ TEMPLATE, DISTRIBUTE(BLOCK, *) :: HAIRY_TEMPLATE(47,73)

      TYPE COMPLICATED
        INTEGER SIZE
        REAL RV(100,100), KV(100,100), QV(47,73)
! Arrays RV, KV, and QV may be mapped
!HPF$   DISTRIBUTE (BLOCK, BLOCK):: RV, KV
!HPF$   ALIGN WITH HAIRY_TEMPLATE :: QV
        TYPE(SIMPLE) SV(100)
! The following directive is not valid because SIMPLE
!  is an explicitly mapped type.
!HPF$   DISTRIBUTE SV(BLOCK)
      END TYPE COMPLICATED

      TYPE(COMPLICATED) LOTSOF(20)

! The following directive is not valid because COMPLICATED
! is an explicitly mapped type.
!HPF$ DISTRIBUTE LOTSOF(BLOCK)
                                                                        \EDOC

Here, a component of the derived type {\tt SIMPLE} has been mapped;
thus objects of this type, e.g., {\tt SV} in type {\tt COMPLICATED},
cannot be distributed. The array {\tt LOTSOF} cannot be distributed for the
same reason.

Components of structures can be remapped using the {\tt REDISTRIBUTE}
or {\tt REALIGN} directive if they have been declared {\tt DYNAMIC}.
For example, the following code fragment can be used to allocate and map
multiple blocks (called {\tt SUBGRID} here) of a multi-block grid:

                                                                        \CODE
!HPF$ PROCESSORS P( number_of_processors() )

      TYPE SUBGRID
        INTEGER SIZE
        INTEGER LO, HI          ! target subset of processors
        REAL, POINTER BL(:)
!HPF$   DYNAMIC BL
      END TYPE SUBGRID

      TYPE (SUBGRID), ALLOCATABLE :: GRID(:)

      READ (*,*) SUBGRID_COUNT
      ALLOCATE GRID(SUBGRID_COUNT)
      DO I = 1, SUBGRID_COUNT
        READ(*,*) GRID(I)%SIZE
      END DO

! Compute processor subsets for each subgrid, setting the LO and HI values
      CALL FIGURE_THE_PROCS ( GRID, number_of_processors())
! Allocate each subgrid and distribute to the computed processors subset
      DO I = 1, SUBGRID_COUNT
        ALLOCATE( GRID(I)%BL( GRID(I)%SIZE ) )
!HPF$   REDISTRIBUTE GRID(I)%BL(BLOCK) ONTO P( GRID(I)%LO : GRID(I)%HI )
     END DO
                                                                        \EDOC

%-----------------------------------------------------------------------------
% Indirect mapping proposal, apparent date Jan 18, 1996
%----------------------------------------------------------------------------

\section{New Distribution Formats}
\label{dist-formats}

This section describes two new
distribution formats. Formally, we can extend the syntax as follows:

                                                                \BNF
extended-dist-format    \IS  BLOCK  [ ( int-expr ) ]
                        \OR  CYCLIC [ ( int-expr ) ]
                        \OR GEN\_BLOCK (int-array)
                        \OR INDIRECT (int-array)
                        \OR  *
                                                                \FNB

\begin{constraints}
\item
    An {\it int-array} appearing in a {\it extended-dist-format} of a
    {\tt DISTRIBUTE} or {\tt REDISTRIBUTE} directive must be an
integer array of rank 1.

\item
    An {\it int-array} appearing in a {\it extended-dist-format} of a
    {\tt DISTRIBUTE} directive must be a {\it restricted-expr}.

\item
    The size of any
 {\it int-array} appearing with a {\tt GEN\_BLOCK} distribution must be
equal to the extent of the corresponding dimension of the target processor
arrangement.

\item
    The size of any
 {\it int-array} appearing with an {\tt INDIRECT} distribution must be
equal to the extent of the corresponding dimension of the {\it distributee}
to which the distribution is to be applied.

\end{constraints}

The ``generalized'' block distribution, {\tt GEN\_BLOCK}, allows
contiguous segments of an array, of possibly unequal sizes, to be mapped
onto processors. The sizes of the segments
are specified by values of a user-defined integer mapping array,
one value per target processor of the mapping.
That is,  the {\it ith} element of the mapping array specifies the
size of the block to be stored on the {\it ith} processor
of the target processor arrangement.
Thus, the values of the mapping arrays are restricted to be
non-negative numbers and their sum must be greater than or equal to
the extent of the corresponding dimension the array being distributed.

The mapping array has to be a restricted expression when used in  the
{\tt DISTRIBUTE}   directive,  but  can  be  an  array  variable  in  a
{\tt REDISTRIBUTE} directive. In the latter case, changing the value of
the  map  array  after  the  directive has been executed will not
change the mapping of the distributed array.

Let {\it l} and {\it u} be the lower and upper bounds of the dimension
of the {\it distributee}, {\it MAP} be the mapping array
 and let {\it BS(i):BE(i)} be the resultant
elements mapped to the {\it ith} processor in the corresponding
dimension of the target processor arrangements.
Then,
\begin{eqnarray*}
BS (1) & = & l,\\
BE (i) & = & max( BS(i)+MAP(i)-1, u),\\
BS (i) & = & BE(i-1) + 1.\\
\end{eqnarray*}

Example:
                                                                        \CODE
        PARAMETER (S = /2,25,20,0,8,65/)
!HPF$ PROCESSORS P(6)
        REAL A(100), B(200), new(6)
!HPF$ DISTRIBUTE A( GEN_BLOCK( S) ) ONTO P
!HPF$ DYNAMIC  B
        ...
       new = ...
!HPF$  REDISTRIBUTE ( B( GEN_BLOCK(new) )
                                                                        \EDOC
Given the above specification, array elements A(1:2) are mapped on P(1),
A(3:27) are mapped on P(2), A(28:37) are mapped on P(3),
no elements are mapped on P(4), A(37:45) are mapped on P(5), and
A(46:100) are mapped on P(6).
The array {\it B} is distributed based on the array {\it new} whose
values are computed at runtime.

\begin{implementors}
Accessing elements of an array distributed using the generalized
block distribution may require accessing the values of the mapping
array at runtime. However, since the size of such an array
is equal to that of the processor arrangement, it can in most
cases be replicated over all processors.

For dynamic arrays, an independent copy of the mapping array
will have to be maintained internally so that a change
in the values of the mapping array does not affect the access
of the distributed array.
\end{implementors}

There are many scientific applications in which the structure  of
the  underlying domain is such that it does not map directly onto
Fortran data structures. For example, in many CFD applications  an
unstructured mesh (consisting of triangles in 2D or tetrahedra in
3D) is used to represent the underlying domain. The nodes of such
a mesh are generally represented by a one-dimensional array while
another is used to  represent  their  interconnections.   Mapping
such  arrays  using the structured distribution mechanisms, {\tt BLOCK}
and {\tt CYCLIC}, results in mappings in which unrelated  elements  are
mapped  onto  the  same  processor. This in turn leads to massive
amounts of unnecessary communication.   What  is  required  is  a
mechanism  to  map a related set of arbitrary array elements onto
the same processor.  The  {\tt INDIRECT}  distribution
provides  such  a  mechanism.

The  {\tt INDIRECT}  distribution  allows  a  many-to-one  mapping   of
elements  of  a  dimension  of a data array to a dimension of the
target processor arrangement. An integer array is used to specify
the  target  processor  of  each  individual element of the array
dimension being distributed.  That is, the {\it ith} element of the mapping
array  provides  the  processor  number  onto which the {\it ith} array
element is to be mapped.  Since the mapping array maps array elements
onto  processor  elements, the extent of the mapping array must match
the extent of the dimension of the array it is distributing. Also,
the  values of the mapping array must lie between the lower and upper
bound of the target dimension of the processor arrangement.

The mapping array has to be a restricted expression when used in  the
{\tt DISTRIBUTE}   directive,  but  can  be  an  array  variable  in  a
{\tt REDISTRIBUTE} directive. In the latter case, changing the value of
the  mapping  array  after  the  directive has been executed will not
change the mapping of the distributed array.

Example:

                                                                        \CODE
!HPF$ PROCESSORS P(4)
        REAL A(100), B(50)
        INTEGER map1(100), map2(50)
        PARAMETER (map1 = /1,3,4,3,3,2,1,4, ..../)
!HPF$ DYNAMIC B
!HPF$ DISTRIBUTE A( INDIRECT(map1) ) ONTO P
!HPF$ DISTRIBUTE map2(BLOCK) ONTO P

        map2 = ...
!HPF$ DISTRIBUTE B( INDIRECT(map2) ) ONTO P
        ....
                                                                        \EDOC

Here, the array {\it A} is distributed statically using the constant
array {\it map1}.  Thus:

\hspace{0.5in}
\parbox{3in}{
A(1) is mapped onto P(1),\\
A(2) is mapped onto P(3),\\
A(3) is mapped onto P(4),\\
A(4) is mapped onto P(3),\\
A(5) is mapped onto P(3),\\
A(6) is mapped onto P(2),\\
A(7) is mapped onto P(1),\\
A(5) is mapped onto P(4), and so on.\\
}

The array {\it B} is declared  dynamic and is redistributed using
the mapping array {\it map2}.

\begin{implementors}
In general, the {\tt INDIRECT} distribution is going to be used in  the
{\tt REDISTRIBUTE}  directive  with an array variable as the map array.
Also, since the size of the mapping array must be  the  same  as  the
array  being  distributed,  it  will  itself  be distributed most
likely using the {\tt BLOCK} distribution.  This raises several issues.
To  correctly  implement  this  distribution,  the runtime system
should maintain a (distributed) copy of the mapping array so that  if
the  program  modifies  the  mapping array, the distribution does not
change.  Using an array variable as a mapping array implies that  the
location  of  each  element  of the array will not be known until
runtime. Thus, a communication maybe required to figure  out  the
location  of  a  specific array element.
\end{implementors}

%-----------------------------------------------------------------------------
% Shadow regions proposal, apparent date Feb 7, 1996
%-----------------------------------------------------------------------------

\section{Range Directive}
\label{range}

The {\tt RANGE} attribute is used to restrict the possible distribution
formats for an object or template that has the {\tt DYNAMIC} attribute or
a transcriptive distribution format (including pointers).

                                                                \BNF
range-directive         \IS RANGE ranger range-attr-stuff-list

ranger                  \IS object-name
                        \OR template-name

range-attr-stuff        \IS ( range-attr-list )

range-attr              \IS range-dist-format
                        \OR ALL

range-dist-format       \IS BLOCK [ ()]
                        \OR  CYCLIC [ () ]
                        \OR GEN\_BLOCK
                        \OR INDIRECT
                        \OR  *

                                                                \FNB

\begin{constraints}

\item
The {\it ranger} must have the {\tt DYNAMIC} attribute,
the {\tt INHERIT} attribute and no explicit {\tt DISTRIBUTE},
or it must have been specified with a {\it dist-format-clause}
of {\tt *} in a {\tt DISTRIBUTE} or combined directive.

\item
The length of any {\it range-attr-list} must be equal to the rank of
the {\it ranger}.

\item
The {\it ranger} must not appear as an alignee in an {\tt ALIGN}
or {\tt REALIGN} directive.
\end{constraints}

Since the length of each {\it range-attr-list} is the same as the rank
of the {\it ranger}, each {\it range-attr}, {\it R}, in each {\it
range-attr-stuff} corresponds positionally to a dimension {\it D} of
the ranger.  This dimension {\it D} in turn either corresponds (though
not necessarily positionally) to an axis {\it A} of the template with
which the ranger is ultimately aligned, or there is no corresponding
axis in that template.

With this notation, a {\tt RANGE} attribute on a {\it ranger} is equivalent to
the following restriction:

For at least one {\it range-attr-stuff} in the {\it
    range-attr-stuff-list}, every {\it range-attr}, {\it R}, must either
  \begin{itemize}
  \item be compatible with the distribution
    format of the corresponding axis {\it A}, if such a corresponding axis
    exists, or
  \item be either {\tt *} or {\tt ALL}, if no such corresponding axis
    exists.
  \end{itemize}

This compatibility must be maintained by any {\tt DISTRIBUTE}
or {\tt REDISTRIBUTE} directive in which the {\it ranger} appears as
a distributee, or if the ranger has the {\tt POINTER}
attribute and is transcriptively distributed,
for any target with which the {\it ranger}
becomes associated.

A distribution format of

\begin{enumerate}

\item
{\tt BLOCK} is compatible with a {\it range-dist-format} of {\tt BLOCK},
{\tt BLOCK()} or {\tt CYCLIC()};

\item
{\tt BLOCK(n)} is compatible with a {\it range-dist-format} of
{\tt BLOCK()} or {\tt CYCLIC()};

\item
{\tt CYCLIC} is compatible with a {\it range-dist-format} of {\tt CYCLIC}
or {\tt CYCLIC()};

\item
{\tt CYCLIC(n)} is compatible with a {\it range-dist-format} of {\tt CYCLIC()};

\item
{\tt GEN\_BLOCK} is compatible with a {\it range-dist-format} of
{\tt GEN\_BLOCK};

\item
{\tt INDIRECT} is compatible with a {\it range-dist-format} of {\tt INDIRECT};

\item
{\tt *} is compatible with a {\it range-dist-format} of {\tt *}.

\end{enumerate}

All distribution formats are compatible with a {\it range-dist-format} of
{\tt ALL}.

Note that the possibility of a {\tt RANGE} directive of the form
                                                                        \ICODE
!HPF$ RANGE range-attr-stuff-list :: ranger-list
                                                                        \EDOC
is covered by syntax rule \ref{combined-directive-rule} for a
{\it combined-directive} using {\it combined-attribute-extended}
as defined in rule \ref{combined-attribute-extended-rule}.

Examples:

                                                                \CODE
!HPF$    DISTRIBUTE T(BLOCK)
!HPF$    ALIGN A(I,J) WITH T(I)

         CALL SUB(A)
            ....

         SUBROUTINE SUB(X)
!HPF$    INHERIT X
!HPF$    RANGE X (BLOCK, *),  (CYCLIC, *)
                                                                \EDOC

Since the ultimate align target of X, the inherited template T in this case,
does not have a second dimension, only a {\tt *} or {\tt ALL} can be
used in the second dimensions of the {\tt range-attr-stuff} for X.

                                                                \CODE
         REAL A(100, 100, 100)
!HPF$    DISTRIBUTE A(BLOCK, *, CYCLIC)

         CALL SUB( A(:,,:,1) )          ! Conforming
         CALL SUB( A(:,,1,:) )          ! Nonconforming
         CALL SUB( A(1,,:,:) )          ! Nonconforming
            ....

         SUBROUTINE SUB(X)
         REAL A(:, :)
!HPF$    INHERIT X
!HPF$    RANGE X (BLOCK, *)
                                                                \EDOC

Given the range directive in the subroutine SUB, only the first call
to SUB is conforming. However, all three calls can be made conforming if
the range directive above is replaced by the following directive:

                                                                \CODE
!HPF$    RANGE (BLOCK, *), (BLOCK, CYCLIC), (*, CYCLIC) :: X
                                                                \EDOC

\section{Shadow Width Declarations}
\label{mapext:shadow}

In compiling nearest-neighbor code, for example, in
discretizing partial differential equations or implementing
convolutions, a standard technique is to allocate storage on each
processor for the local array section that includes additional space
for the elements that have to be moved in from its neighboring
processors.  This additional storage is referred to as ``shadow
edges.''  There are conceptually two shadow edges for each array
dimension:  one at the low end of the local array section, and the
other at the high end.

In a single routine, the compiler can tell which arrays require shadow
edges, and allocate this additional space accordingly.  However, since
the width of the shadow area is dependent on the size of the
computational stencil being used, an array may require different
shadow widths in different routines. Thus, without interprocedural
analysis, an array argument may need to be copied into a space with
the appropriate shadow width on each procedure call. A similar data
motion would be required to copy the data back to its original
location on exit from the subroutine.  This unnecessary data motion
can be avoided by allowing the user to specify the required shadow
width when the array is declared.

Formally, the syntax for declaring shadow widths is as follows:

                                                                \BNF
shadow-directive        \IS SHADOW shadow-target shadow-attr-stuff

shadow-target               \IS object-name
                        \OR component-name

shadow-attr-stuff       \IS ( shadow-spec-list )

shadow-spec             \IS width
                        \OR low-width : high-width

width                   \IS int-expr
low-width               \IS int-expr
high-width              \IS int-expr
                                                                \FNB

\begin{constraints}
\item The \textit{int-expr} representing a \textit{width}, \textit{low-width},
  or \textit{high-width} must be a constant \textit{specification-expr} with
  value greater than equal to 0.

\item
The absence of a \textit{shadow-directive} is equivalent to a {\it
  shadow-directive} having each \textit{shadow-spec} equal to 0.

\item A \textit{shadow-spec} of {\it width} is equivalent to a
  \textit{shadow-spec} of {\it width}:{\it width}
\end{constraints}

Thus, the directive

                                                                \CODE
!HPF$    DISTRIBUTE (BLOCK) :: A
!HPF$    SHADOW (w) :: A
                                                                \EDOC

\noindent
specifies that the array \textit{A} is distributed \texttt{BLOCK}
and is to have a shadow width of \textit{w} on both sides.  If
\textit{A} is a dummy argument, this gives the compiler enough
information to inhibit unnecessary data motion at procedure calls.

Alternatively, different
shadow widths can be specified for the low end and high end
of a dimension. For example:
                                                                \CODE
         REAL, DIMENSION (1000) :: A
!HPF$    DISTRIBUTE(BLOCK), SHADOW(1:2) ::  A
         ....
         FORALL (i = 2, 998)
            A(i) = 0.25* (A(i) + A(i-1) + A(i+1) + A(i+2))
                                                                \EDOC
\noindent
specifies that only one non-local element is needed at the lower end
while two are needed at the high end.

\section{Equivalence and Partial Order on the Set of Mappings}
\label{ext:PartialOrderOnMaps}

Section~\ref{mapsub:PartialOrderOnMaps} has to be changed to
accomodate the new distributions, the \texttt{SHADOW} attribute, and
mapping of components of derived types, all introduced as approved
extensions.  The relevant text now reads as follows; additions are in
\textbf{bold-face} type:

First, we define a notion of equivalence for \textit{dist-format}
specifications:

\begin{enumerate}
\item Each \textit{dist-format} is equivalent to itself.

\item  Using the notation \(\equiv\) for the phrase ``is equivalent to'',

  \[\begin{array}{rclcl}
    \texttt{BLOCK}(n)  &\equiv& \texttt{BLOCK}(m) & \mbox{iff } m
      \mbox{ and } n \mbox{ have the same value} \\
    \texttt{CYCLIC}(n) &\equiv& \texttt{CYCLIC}(m) & \mbox{iff } m
      \mbox{ and } n \mbox{ have the same value} \\
    \texttt{CYCLIC}    &\equiv& \texttt{CYCLIC}(1) \\
    \texttt{GEN\_BLOCK(V)} &\equiv& \texttt{GEN\_BLOCK(W)} &
      \mbox{\textbf{iff the values of corresponding}} \\
       &&& \mbox{\textbf{  elements of
           \texttt{V} and \texttt{W} are equal}} \\
    \texttt{INDIRECT(V)} &\equiv&\texttt{INDIRECT(W)} &
      \mbox{\textbf{iff the values of corresponding}} \\
      &&& \mbox{\textbf{  elements of
          \texttt{V} and \texttt{W} are equal}}
  \end{array}\]

\item Other than this, no two lexically distinct \textit{dist-format}
  specifications are equivalent.
\end{enumerate}

Equivalence, thus defined, is an equivalence relation in the usual
sense.

\textbf{Similarly, we say that two \texttt{SHADOW} attributes (see
  Section~\ref{mapext:shadow} for the syntax) are equivalent iff the
  \textit{shadow-spec-list} of one is element wise equivalent to the
  \textit{shadow-spec-list} of the other.}

Now the partial order on mappings is defined: Let \texttt{S} (``special'')
and \texttt{G} (``general'') be two data objects.

The mapping of \texttt{S} is a \emph{specialization} of the mapping of
\texttt{G} if and only if either

\begin{enumerate}

\item \texttt{G} has the \texttt{INHERIT} attribute, or

\item \texttt{S} does not have the \texttt{INHERIT} attribute, and the
  following constraints all hold:

  \begin{enumerate}
  \item \texttt{S} is a named object \textbf{or structure component}, and
  \item The shapes of the ultimate align targets of \texttt{S} and \texttt{G}
    are the same, and
  \item The dimensions of \texttt{S} and \texttt{G} each correspond to the same
    dimension of their respective ultimate align targets, and
    corresponding elements of \texttt{S} and \texttt{G} are aligned with the
    same corresponding elements of their respective ultimate align
    targets, and
  \item Either

    \begin{enumerate}
    \item The ultimate align targets of \texttt{S} and \texttt{G} are not
      explicitly distributed, or
    \item Both ultimate align targets of \texttt{S} and \texttt{G} are
      explicitly distributed.  In this case, the distribution
      directive specified for the ultimate align target of \texttt{G} must
      satisfy one of the following conditions:

      \begin{enumerate}
      \item It must have no \textit{dist-onto-clause}, or
      \item It must have a \textit{dist-onto-clause} of
        ``\texttt{ONTO} *'', or
      \item It must have a \textit{dist-onto-clause} specifying a
        processor arrangement having the same shape as that
        explicitly specified in a distribution directive for the
        ultimate align target of \texttt{S}.
      \end{enumerate}

      and the distribution directive for the ultimate align target of
      \texttt{G} must also satisfy one of the following conditions:

      \begin{enumerate}
      \item It must have no \textit{dist-format-clause}, or
      \item It must have a \textit{dist-format-clause} of ``*'', or
      \item Each \textit{dist-format} must be equivalent (in the sense
        defined above) to the \textit{dist-format} in the
        corresponding position of the \textit{dist-format-clause} in
        an explicit distribution directive for the ultimate align
        target of \texttt{S}.
      \end{enumerate}

    \end{enumerate}

  \item \textbf{Either \texttt{S} and \texttt{G} have no
      \texttt{SHADOW} attribute or they have equivalent
      \texttt{SHADOW} attributes.}

  \end{enumerate}

\end{enumerate}

\section{Explicit Interfaces}
\label{ext:ExplicitInterfaces}

The requirements in Section~\ref{mapsub:ExplicitInterfaces} are
extended as follows to account for the possible presence of the
\texttt{DYNAMIC} attribute; the addition is in \textbf{bold-face}
type:

An explicit interface is required \emph{except} when all of the
following conditions hold:

\begin{enumerate}

\item  Fortran does not require one, \emph{and}

\item No dummy argument is distributed transcriptively or with the
  \texttt{INHERIT} attribute, \emph{and}

\item \textbf{No dummy argument has the
  \texttt{DYNAMIC} attribute, \emph{and}}

\item For each pair of corresponding actual and dummy arguments, either:

  \begin{enumerate}

  \item They are both implicitly mapped, or

  \item They are both explicitly mapped and

    \begin{enumerate}
    \item The mapping of the actual argument is a specialization of
      the mapping of the dummy argument, and
    \item If the ultimate align targets of the actual and dummy
      arguments are both explicitly distributed, then the
      \textit{dist-onto-clause} of each must specify processor
      arrangements with the same shape.
    \end{enumerate}

  \end{enumerate}

  \emph{and}

\item For each pair of corresponding actual and dummy arguments, either:

  \begin{enumerate}

  \item Both are sequential, or

  \item Both are nonsequential.

  \end{enumerate}

\end{enumerate}

\section{Characteristics of Procedures}
\label{ext:ProcChars}

The \texttt{SHADOW} and \texttt{DYNAMIC} attributes, if present, are
hpf-characteristics of dummy arguments and procedure return values.
To be precise, the definitions in Section~\ref{mapsub:ProcChars} are
rewritten as follows; additions are in \textbf{bold-face} type:

\begin{itemize}
\item A processor arrangement has one hpf-characteristic: its shape.

\item A template has up to three hpf-characteristics:

\begin{enumerate}
\item its shape;
\item its distribution, if explicitly stated;
\item the hpf-characteristic (i.e., the shape) of the processor
  arrangement onto which it is distributed, if explicitly stated.
\end{enumerate}

\item A dummy data object has the following hpf-characteristics:

\begin{enumerate}
\item its alignment, if explicitly stated, as well as all
  hpf-characteristics of its align target;
\item its distribution, if explicitly stated, as well as the
  hpf-characteristic (i.e., the shape) of the processor arrangement
  onto which it is distributed, if explicitly stated;
\item \textbf{its \texttt{SHADOW} attribute, if explicitly stated.}
\item \textbf{its \texttt{DYNAMIC} attribute, if explicitly stated.}
\end{enumerate}

\item A function result has the same hpf-characteristics as a dummy
  data object.  Specifically, it has the following
  hpf-characteristics:

\begin{enumerate}
\item its alignment, if explicitly stated, as well as all
  hpf-characteristics of its align target;
\item its distribution, if explicitly stated, as well as the
  hpf-characteristic (i.e., the shape) of the processor arrangement
  onto which it is distributed, if explicitly stated;
\item \textbf{its \texttt{SHADOW} attribute, if explicitly stated.}
\item \textbf{its \texttt{DYNAMIC} attribute, if explicitly stated.}
\end{enumerate}

\end{itemize}

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------