Subject: hpff-doc: hpf-doc: hpf-local-ext.tex (includes my corrections of Oct 15)

From: [email-masked]
Date: Tue, 29 Oct 96 09:33:09 -0500

---------------------------------------------------------------------------
[email-masked] is a mailing list for HPF 2.0 language specification
authors and editors.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------
% File: hpf-local-ext.tex

% Contents:
% Approved Extension for EXTRINSIC(HPF_LOCAL) for HPF 2.0 document

% Revision history:
% May-10-96     Created by Charles Koelbel, Rice University
%               (from HPF 1.2 document)
% July 10-96  Major Revision to make new chapter by Mary Zosel and more Aug. 11
% Plus new section on C by Henry Zongaro and Andy Meltzer.
% Sept 18 Zosel and Munroe - continue generation of integrated chapters
% Oct 4 Munroe - continued minor changes re global-to-local argument passing
%                and input revised section on extrinsic(f77_local)
%Oct 29 David Loveman - clarification of argument passage, correction of example

\chapter{Approved Extensions for HPF Extrinsics}
\label{ch-hpf-local-ext}

%{\em
%Comments on this chapter should be directed to
%Mary Zosel ({\tt [email-masked]})
%and {\tt [email-masked]}.
%Also copy Henry Zongaro ({\tt [email-masked]})
%for any comments on
%the C language interface.
%Please use ``{\tt Comments on HPF\_LOCAL}'' as the {\tt Subject:}
%line.
%\par
%}
%

This Section builds on Section ~\ref{ch-extrinsics} by defining specific
interfaces for HPF
with different models of parallelism ({\tt LOCAL} and {\tt SERIAL}
(Section~\ref{ext-models})
and different languages HPF (Section~\ref{ext-lang-hpf}),
C (Section~\ref{ext-lang-c}), and
Fortran (Section~\ref{ext-lang-f}).
Library routines
useful in the extrinsic models are defined in Section~\ref{ext-lib}.
These are defined with Fortran bindings.  An implementation may chose to define
similar routines for use with C.
The intent of the HPF extrinsic mechanism is to generalize.
These definitions may
serve as a model for other interfaces.
Some additional extrinsic interfaces are
given after Annex~\ref{ch-extrinsic-app}.

In HPF 1.1, specific extrinsics were
defined for {\tt HPF}, {\tt HPF_LOCAL}, and {\tt HPF_SERIAL}. There was also
a short discussion of {\tt Fortran90_Local}.
In this more general setting, the  {\tt HPF} keywords are retained for
compatibility.  As defined in Section~\ref{ch-extrinsics}, a model
of {\tt HPF} refers to the global language,
{\tt HPF_LOCAL} is the same as {\tt LANGUAGE = "hpf" , MODEL = "local"}, and
{\tt HPF_SERIAL} is the same as {\tt LANGUAGE = "hpf", MODEL = "serial"}. The
{\tt Fortran90_Local} extrinsic is now addressed by the
{\tt LANGUAGE = "fortran"} extrinsic kind.

>From the caller's standpoint, an invocation of an extrinsic
procedure from a ``global'' HPF program has the same semantics as an
invocation of a regular procedure.  The callee may see a different
picture.  The following sections describe sets of conventions for
coding callees in the various extrinsic options.  The set of extrinsic options
supported by a particular  HPF compiler is implementation-dependent.  They are
not limited to those descibed in this chapter. Furthermore, the language
processor used to compile the actual extrinsic subprogram need not be an HPF
compiler.  More
specifically, it need not actually be a compiler for the language noted in the
{\tt LANGUAGE}
specification, as long as the executing extrinsic code conforms to the
conventions defined
for the language.  We define these interfaces to promote portability and
interoperability,
but a given implementation and the programmer are free to create other
combinations of models and languages.

\section{Alternative Extrinsic  Models:  LOCAL and SERIAL}
\label{ext-models}
A global HPF program may be thought of as a set of processors cooperating in
a loosely synchronous fashion on a single logical thread of program control.
Section~\ref{ch-extrinsics}
defines two additional models that may be invoked from global HPF:
{\tt LOCAL} where the model is single-processor  "node" code, where all active
processors participate, but with only the
data that is mapped to a given physical processor directly accessible, and
{\tt SERIAL} where the model is a single-processor, operating alone, with all
necessary data aggregated by the caller before the {\tt SERIAL} subprogram is
invoked. As examples of use,the  {\tt LOCAL} model is useful for programs that
drop into code with explicit message passing for data communications, while the
{\tt SERIAL} model may be needed for calls to system libraries or
specialized I/O routines.

Both the {\tt LOCAL} and the  {\tt SERIAL} model can be invoked from a
global HPF program, but in general, these models may not be mixed.
Calling {\tt GLOBAL} HPF from {\tt LOCAL} or {\tt SERIAL} is not
defined.  Furthermore, calling  {\tt SERIAL} from {\tt LOCAL} is not
defined.  Section~\ref{ext-access} gives more detail about how
various models can interact with each other.

Some additionl restrictions are placed on all  {\tt LOCAL} and
{\tt SERIAL} subprograms
invoked from global HPF:

\begin{itemize}
\item An {\tt EXTRINSIC("HPF","LOCAL")} routine may not be invoked, either
directly or indirectly, in the body of a {\tt FORALL} construct or in
the body of an {\tt INDEPENDENT} loop.

\item The subprogram directly invoked by global HPF must not be
recursive.

\item The subprogram directly invoked by global HPF must not use
any alternate return mechanism.

\item An  {\tt "HPF","LOCAL"} or {\tt "HPF","SERIAL"}
procedure referenced in global HPF must have an
accessible explicit interface.

\end{itemize}

The behavior of I/O statements in a  {\tt LOCAL} and {\tt SERIAL} subprogram is
implementation-dependent.

\subsection{The LOCAL Model}
\label{ext-local-model}
An extrinsic procedure can be defined as explicit SPMD code by
specifying the local procedure
code that is to execute on each processor.  In this section, we describe the
contract between the caller and a callee that is a local procedure.
It is important not to confuse the extrinsic procedure,
which is conceptually a single procedural entity called
from the HPF program, with the local procedures,
which are executed on each node,
one apiece. An {\it invocation} of an extrinsic procedure results in a
separate invocation of a local procedure on each processor.
The {\it execution} of an extrinsic procedure consists of the concurrent
execution of a local procedure on each executing processor.  Each
local procedure may terminate at any time by executing a {\tt RETURN}
statement.  However, the extrinsic procedure as a whole terminates
only after every local procedure has terminated; in effect, the
processors are synchronized before return to a global HPF caller.

With the exception of returning from a local procedure to the global
caller that initiated local execution, there is no implicit
synchronization required of the locally executing processors.  A local
procedure may use any control structure whatsoever.  To access data
outside the processor requires either preparatory communication to
copy data into the processor before running the local code, or
communication between the separately executing copies of the local
procedure.  Individual implementations may provide
implementation-dependent means for communicating, for example through
a message-passing library or a shared-memory mechanism.  Such
communication mechanisms are beyond the scope of this specification.
Note, however, that many useful portable algorithms that require only
independence of control structure can take advantage of local
routines, without requiring a communication facility.

The local  model assumes only that nonsequential
array axes are mapped independently to
axes of a rectangular processor grid, each array axis to at most one
processor axis (no ``skew'' distributions) and no two array axes to
the same processor axis.  This restriction suffices to ensure that
each physical processor contains a subset of array elements that can
be locally arranged in a rectangular configuration.  (Of course, to
compute the global indices of an element given its local indices, or
vice versa, may be quite a tangled computation---but it will be
possible.)  In the case of cyclic distributions, multiple sections of
the array may be mapped to the local processors.

It is recommended that if, in any given implementation, an interface
kind does not obey the conventions described in this section, then the
name of that interface kind should not use the {\tt LOCAL} keyword as
the name of the model.

\subsubsection{Conventions for Calling Local Subprograms}

The default mapping of scalar dummy arguments,
of scalar function results, and of sequential arrays is such that the argument
is replicated on each physical processor.  These mappings may, optionally,
be explicit in the interface (except in the case of sequential arrays,
which may not be explicitly mapped), but any other explicit mapping is not
HPF conforming. Dummy arguments may not be of explicitly mapped derived
types or have explicitly mapped structure components.

As in the case of non-extrinsic subprograms, actual arguments may be
mapped in any way; if necessary, they are copied automatically to
correctly mapped temporaries before invocation of and after return
from the extrinsic procedure.

It should be noted that the conventions for calling local
subprograms apply only at the interface between the global and
local models. The conventions do not propagate to further
subprograms called from within the local model.

\subsubsection{Calling Sequence}

The actions detailed below have to occur prior to the invocation of
the local procedure on each processor.  These actions are enforced by
the compiler of the calling routine, and are not the responsibility of
the programmer, nor do they impact the local procedure.

\begin{enumerate}

\item The processors are synchronized.  In other words, all actions
that logically precede the call are completed.

\item Each actual argument is remapped, if necessary, according to the
directives (explicit or implicit) in the declared interface for the
extrinsic procedure.  Thus, HPF mapping directives appearing in the
interface are binding---the compiler must obey these directives in
calling local extrinsic procedures.  (The reason for this rule is that
data mapping is explicitly visible in local routines).  Actual
arguments corresponding to sequential arrays and scalar dummy arguments
are replicated (by broadcasting, for example) in all processors. 
Scalars of derived types with explicitly mapped structure components
or of an explicitly mapped derived type cannot be passed from global
HPF to an extrinsic local procedure.

\item If a variable accessible to the called routine has a replicated
representation, then all copies are updated prior to the call to
contain the correct current value according to the sequential
semantics of the source program.

\end{enumerate}

After these actions have occurred, the local procedure is invoked on
each processor.  The information available to the local invocation is
described below in Section~\ref{local-info-section}.

The following actions  occur before control is transferred back to
the caller.

\begin{enumerate}

\item All processors are synchronized after the call.  In other words,
execution of every copy of the local routine is completed before
execution in the caller is resumed.

\item The original distribution of arguments (and of the result of an
extrinsic function) is restored, if necessary.

\end{enumerate}

\begin{implementors}
An implementation might check, before returning from the local
subprogram, to make sure that replicated variables have been updated
consistently by the subprogram.  However, there is certainly no
requirement---perhaps not even any encouragement---to do so.  This is
the responsibilty of the local subprogram, and any checks in the
caller are a tradeoff between speed and, for instance, debuggability.
\end{implementors}

\subsubsection{Information Available to the Local Procedure}
\label{local-info-section}

The local procedure invoked on each processor is passed a {\em local
argument} for each {\em global argument} passed by the caller to the
(global) extrinsic procedure interface. Each global argument is an HPF
array or scalar. The corresponding local argument is the part of the
global array stored locally, or a local copy of a scalar argument
or sequential array
replicated across processors. Note that if the HPF array is
replicated, each local procedure receives a copy of the entire
actual. An array actual argument passed by an HPF caller is called a
{\em global array}; the subgrid of that global array passed to one
copy of a local routine (because it resides in that processor) is
called a {\it local array}.

If the extrinsic procedure is a function, then the local procedure is
also a function.  Each local invocation of that function will return
the local part of the extrinsic function return value. Only scalar-valued
extrinsic function are allowed.  All local functions must return the
same value.

If a global HPF program calls local subprogram {\tt A} with
an actual array argument {\tt X}, and {\tt A} receives a portion of
array {\tt X} as dummy argument {\tt P}, then {\tt A} may call another
local subprogram {\tt B} and pass {\tt P} or a section of {\tt P} as
an actual argument to {\tt B}.

The run-time interface must provide enough information that each
local function can discover for each local argument the mapping of the
corresponding global argument, translate global indices to local
indices, and vice-versa.  A specific set of procedures that provide
this information is listed in the HPF LOCAL Library
Section~\ref{HPF-LOCAL-LIBRARY}.  The
manner in which this information is made available to the local
routine depends on the implementation and the programming language
used for the local routine.

\subsection{The SERIAL Model}
\label{ext-serial-model}
This section defines a set of conventions for writing code in which an
instance of a subprogram executes on only one processor (of which
there may be more than one).

If a program unit has extrinsic model
{\tt SERIAL}, an HPF compiler should assume
that the subprogram is coded to be executed on a single processor.
>From the point of view of a global HPF caller, the {\tt SERIAL} procedure
behaves the same as an identically coded HPF procedure would.  Differences
might only arise in implementation-specific behavior (such as the performance).

There is currently no manner in which to specify which processor is to
execute an "HPF","SERIAL" procedure.

\subsubsection{Serial Calling Sequence}

Prior to invocation of a {\tt SERIAL} procedure from global HPF, the
behavior of the program will be as if the following actions take place:

\begin{enumerate}

\item The processors are synchronized.  All actions that logically precede the
call are completed.

\item All actual arguments are remapped to the processor that will actually
execute the {\tt SERIAL} procedure. Each argument will appear to the
{\tt SERIAL} procedure as a sequential argument.

\end{enumerate}

The behavior of the {\tt SERIAL} procedure will be as if it was executed
on only one processor.  After the instance of the {\tt SERIAL} procedure
invoked from global HPF has completed, the behavior will be as if the
following happens:

\begin{enumerate}

\item All processors are synchronized after the call.

\item The original mappings of actual arguments are restored.

\end{enumerate}

\section{Extrinsic Language Bindings}
\label{ext-langs}

The previous section defines the rules and considerations for execution
models defined for HPF extrinsics.
The HPF extrinsic interface is also used to tell the compiler what the
language
conventions of a called subprogram are. Four language bindings are defined
here:
{\tt HPF, Fortran, F77, and C}.  A given implemtation may support additional
interfaces or allow a user to construct custom interfaces.  Taken together,
these sections define the special
extrinsics {\tt "HPF","LOCAL"} and {\tt "HPF","SERIAL"}.

The key feature of the language interface is the {\tt MAP_TO} feature for
explicit interfaces which is used to give the programmer control over
aspects of argument passing.  This mechanism is used extensively in the
interfaces to {\tt C} and {\tt Fortran 77}, but it is defined in this more
general context, because it can also apply to other language interfaces.

\subsection{Control of Arguments}

\BNF

map-to                  \IS map-to-kind-arg-spec-list

map-to-kind-arg-spec    \IS map-to-type-spec

			\OR layout-spec

			\OR pass-by-spec

map-to-type-spec        \IS MAP_TO ( char-initialization-expr )

layout-spec             \IS  LAYOUT ( char-initialization-expr )

pass-by-spec            \IS  PASS_BY ( char-initialization-expr)

\FNB

\begin{constraints}

\item In a {\it map-to-kind-arg-spec-list},  at most
one each of   {\it map-to-type-spec}, {\it layout-spec}, and {\it pass-by-spec}
may occur.

\end{constraints}

The definition of {\em characteristics of a dummy data object} as
given in Section 12.2.1.1 of the May 1995 draft Fortran 95 standard is
extended to include the dummy data object's data mapping (alignment and
distribution) and its map-to characteristics.

In {\it map-to-type-spec}, values of {\it char-initialization-expr}
are intended to describe how the data type of the named actual argument
is mapped to the data type of the dummy argument in the extrinsic
procedure. An example is given in Section~\ref{cexample}

In {\it layout-spec}, values of {\it char-initialization-expr} are intended
to describe how the data layout of the named actual argument is mapped to
the data layout of the dummy argument in the extrinsic procedure. An
example is given in Section ~\ref{fexample}.  If no {\it layout-spec} is
specified for a dummy array argument, the data layout shall be same as if
it were being passed to an HPF procedure of the same model, unless another
default layout is defined for the given extrinsic type. 

In {\it pass-by-spec}, values of {\it char-initialization-expr}
are intended to describe the mechanism used to associate the named
actual argument with the dummy argument in the extrinsic procedure. 
Examples are given in Sections~\ref{fexample} and~\ref{cexample}. 
If no {\it pass-by-spec}
is specified, the associate mechanism is implementation-dependent, based
on the compiler's knowlege of the extrinsic language processor.

For a given language interface,  that uses the {\it map-to} mechanism
the set of permitted values for
{\it map-to-type-spec}, {\it layout-spec}, and  {\it pass-by-spec} will be
specified.
If the range of permitted values of the type and mapped to type differ and
the value of the actual argument or some subobject of the actual argument is
not within the permitted range of the mapped to type, the value of the
associated dummy argument or subobject becomes undefined.  Conversely, if
the value of the dummy argument or some subobject of the dummy is not within
the permitted range of values of the associated dummy argument, and the
associated actual argument is a variable, the value of the associated actual
argument or subobject of the actual becomes undefined.

If there is a mismatch between the precision, representation method, range of
permitted values or storage sequence between the type of the dummy argument and
the permitted mapped to type of the dummy argument, the compiler shall ensure
that, for the duration of the reference to the extrinsic, the dummy
argument is represented in
a manner that is compatible with the expectations of the callee for an
object of the permitted {\tt map-to-type}.  Upon return from the procedure, the
compiler shall ensure that the value of an actual argument which is a variable
is restored to the specified type and kind.

\begin{users}
This rule was created to ensure the portability of interoperability.  However,
it should be noted that for large objects, a significant overhead may be
incurred if there is a mismatch between the representation method used for
the data type versus the representation method used for the permitted mapped to
type.
\end{users}

\section{HPF Bindings}
\label{ext-lang-hpf}

HPF is the default language assumption.
It requires no {\it map-to} attributes in explicit interface definitions.
All required subprogram binding information can be accomplished via the
standard Fortran explicit interface.

The rules stated in section 14.7 of the Fortran
standard will apply to
variables defined in Fortran-based {\tt SERIAL} and {\tt LOCAL}
 scoping units.  In particular, if the
definition status, association status, or allocation status of a variable is
defined upon execution of a {\tt RETURN} statement or an {\tt END} statement
in a Fortran  subprogram, such a variable in an {\tt SERIAL} or {\tt LOCAL}
 subprogram will be defined upon execution of a {\tt RETURN} statement or an
{\tt END} statement.

Any I/O performed within an
extrinsic  subprogram of a different model, and the correspondence of file
names
and unit numbers used to those used in global HPF and local or serial
subprogram code will be implementation-defined.

\subsection{Additional Special Considerations for {\tt "HPF","LOCAL"}}
\label{hpf-local-restrictions}

There are some considerations about what HPF features may be used in
writing a local, per-processor procedure.

Local program units can use all HPF constructs except for {\tt
REDISTRIBUTE} and {\tt REALIGN}.  Moreover, {\tt DISTRIBUTE}, {\tt ALIGN},
and {\tt INHERIT} directives may be applied only to dummy arguments; that
is, every {\tt alignee} and {\it distributee} must be a dummy argument, and
every {\it align-target} must be a template or a dummy argument. Mapping
directives in local HPF program units are understood to have global
meaning, as if they had appeared in global HPF code, applying the global
array of which a portion is passed on each processor.  (The principal use
of such mapping directives is in an {\tt "HPF","LOCAL"} module that is used by
a global HPF module.)

{\tt HPF_ALIGNMENT}, {\tt HPF_TEMPLATE}, and {\tt HPF_DISTRIBUTION},
the distribution query library subroutines,
may be applied to non-sequential local
arrays.  Their outcome is the same as for a global array that happens
to have all its elements on a single node.

As introduced in Section~\ref{ext-access},
a local HPF program unit may not access global HPF data other than
data that is accessible, either directly or indirectly, via the actual
arguments.  In particular, a local HPF program unit does not have
access to global HPF {\tt COMMON} blocks; {\tt COMMON} blocks
appearing in local HPF program units are not identified with global
HPF {\tt COMMON} blocks.  The same name may not be used to identify a
{\tt COMMON} block both within a local HPF program unit and an HPF
program unit in the same executable program.

Local HPF {\tt COMMON} blocks define storage that is contained on
every processor; each processor has its own private copy of the entire
{\tt COMMON} block and may access only its own private copy.

Just as is the case for local variables in {\tt LOCAL} subprograms,
{\tt COMMON} in local subprograms contains local data, held in local
storage, on each processor.  This storage will in general contain data
that is different on each processor.  Indeed, the size of a local {\tt
COMMON} block can be different on each processor.

According to the Fortran specification, a {\tt COMMON} block that goes
out of scope is not preserved, unless it has the {\tt SAVE} attribute.
This is true of local {\tt COMMON} blocks, which should be given the
{\tt SAVE} attribute if they are intended to convey information
between calls to the local subprogram.

Scalers of an explicitly mapped derived type cannot be passed to an
{\tt "HPF","LOCAL"} subprogram.

The attributes (type, kind, rank, optional, intent) of the dummy
arguments must match the attributes of the corresponding dummy
arguments in the explicit interface.  A dummy argument of an {\tt
EXTRINSIC("HPF","LOCAL")} routine may not be a procedure name.

A dummy argument of an {\tt EXTRINSIC("HPF","LOCAL")} routine may not have
the {\tt POINTER} attribute.

A nonsequential dummy array argument of an
{\tt EXTRINSIC("HPF","LOCAL")} routine must have assumed shape.
% even when it is explicit shape in the interface.
Note that, in general, the shape of a dummy array argument differs
from the shape of the corresponding actual argument, unless there is a
single executing processor.

Explicit mapping directives for dummy arguments may appear in a local
procedure.  Such directives are understood as applying to the global array
whose local sections are passed as actual arguments or results on each
processor.  If such directives appear, corresponding mapping directives
must be visible to every global HPF caller.  This may be done by providing
an interface block in the caller, or by placing the local procedure in a
module of extrinsic kind {\tt "HPF","LOCAL"} that is then used by the global
HPF program unit that calls the local procedure.

A local procedure may have several {\tt ENTRY} points. A global HPF
caller must contain a separate extrinsic interface for each entry
point that can be invoked from the HPF program.

\subsection{Argument Association}

If a dummy argument of an {\tt EXTRINSIC("HPF","LOCAL")} routine is a
scalar then the corresponding dummy argument of the local procedure
must be a scalar of the same type and type parameters.
Only scalars of intrinsic types
or of derived types that are not explicitly mapped may be passed
from a global to an "HPF","LOCAL" routine.  When the
extrinsic procedure is invoked then the local procedure is passed an
argument that consists of the local copy of the replicated
scalar. This copy will be a valid HPF scalar.

If a dummy argument of an {\tt EXTRINSIC("HPF","LOCAL")} routine is an
array, then the corresponding dummy argument in the specification of
the local procedure must be an array of the same rank, type, and type
parameters.

If the array is sequential in the extrinsic interface,
the corresponding actual argument will be passed by replicatation
on all processors, just as scalar arguments are passed.
Each local dummy argument will be
associated with a full copy of the actual array argument.
The dummy argument in the extrinsic interface and the corresponding
dummy argument in the specification of the local
procedure may be declared with the same explicit shape.
All sequential dummy arguments
passed by replication to an EXTRINSIC("HPF","LOCAL") procedure must either be
INTENT(IN) arguments or should be updated consistently across processors.

If the dummy argument is a nonsequential array, then,
when the extrinsic procedure is invoked, the local dummy
argument is associated with the local array that consists of the
subgrid of the global array that is stored locally.   This local array
will be a valid HPF array.

If an {\tt EXTRINSIC("HPF","LOCAL")} routine is a function, then the local
procedure is a function that returns a scalar of the same type and
type parameters as the HPF extrinsic function.  The value returned by each
local invocation must be the same.

Each physical processor has at most one copy of each HPF variable.

Consider the following extrinsic interface:
\CODE
  INTERFACE
    EXTRINSIC("HPF","LOCAL") FUNCTION MATZOH(X, Y) RESULT(Z)
      REAL, DIMENSION(:,:) :: X
      REAL, DIMENSION(:) :: Y
      REAL Z
      !HPF$ ALIGN WITH X(:,*) :: Y(:)
        ! note that this asserts that size(Y) = size(X,1)
      !HPF$ DISTRIBUTE X(BLOCK, CYCLIC)
    END FUNCTION
  END INTERFACE
\EDOC

The corresponding local HPF procedure is specified as follows.

\CODE
EXTRINSIC("HPF","LOCAL") FUNCTION MATZOH(XX, YY) RESULT(ZZ)
  REAL, DIMENSION(:,:) :: XX
  REAL, DIMENSION(5:) :: YY ! assumed shape with lower bound of 5
  REAL ZZ
  NX1 = SIZE(XX, 1)
  LX1 = LBOUND(XX, 1)
  UX1 = UBOUND(XX, 1)
  NX2 = SIZE(XX, 2)
  LX2 = LBOUND(XX, 2)
  UX2 = UBOUND(XX, 2)
  NY  = SIZE(YY, 1)
  LY  = LBOUND(YY, 1)
  UY  = UBOUND(YY, 1)
  ...
END FUNCTION
\EDOC

Assume that the function is invoked with an actual (global) array {\tt
X} of shape \( 3 \times 3 \) and an actual vector {\tt Y} of length \(
3 \) on a 4-processor machine, using a \( 2 \times 2 \) processor
arrangement (assuming one abstract processor per physical processor).

Then each local invocation of the function {\tt MATZOH} receives the
following actual arguments:
\begin{center}
\begin{tabular}{ll@{\hskip.5in}ll}
\multicolumn{2}{l}{Processor (1,1)}&\multicolumn{2}{@{}l}{Processor (1,2)}
\\[4pt]
{\tt X(1,1)} & {\tt X(1,3)}        & {\tt X(1,2)} \\
{\tt X(2,1)} & {\tt X(2,3)}        & {\tt X(2,2)} \\[4pt]
{\tt Y(1)}   &                     & {\tt Y(1)} \\
{\tt Y(2)}   &                     & {\tt Y(2)} \\[8pt]
\multicolumn{2}{l}{Processor (2,1)}&\multicolumn{2}{@{}l}{Processor (2,2)}
\\[4pt]
{\tt X(3,1)} & {\tt X(3,3)}        & {\tt X(3,2)} \\[4pt]
{\tt Y(3)}   &                     & {\tt Y(3)}
\end{tabular}
\end{center}
Here are the values to which each processor would set {\tt NX1},
{\tt LX1}, {\tt UX1}, {\tt NX2}, {\tt LX2}, {\tt UX2}, {\tt NY}, {\tt LY}, and
{\tt UY}:
\begin{center}
\begin{tabular}{lll@{\hskip.5in}lll}
\multicolumn{3}{l}{Processor (1,1)}&\multicolumn{3}{@{}l}{Processor (1,2)}
\\[4pt]
\({\tt NX1}={\tt 2}\)&\({\tt LX1}={\tt 1}\)&\({\tt UX1}={\tt 2}\)&
\({\tt NX1}={\tt 2}\)&\({\tt LX1}={\tt 1}\)&\({\tt UX1}={\tt 2}\) \\
\({\tt NX2}={\tt 2}\)&\({\tt LX2}={\tt 1}\)&\({\tt UX2}={\tt 2}\)&
\({\tt NX2}={\tt 1}\)&\({\tt LX2}={\tt 1}\)&\({\tt UX2}={\tt 1}\) \\
\({\tt ~NY}={\tt 2}\)&\({\tt ~LY}={\tt 5}\)&\({\tt ~UY}={\tt 6}\)&
\({\tt ~NY}={\tt 2}\)&\({\tt ~LY}={\tt 5}\)&\({\tt ~UY}={\tt 6}\) \\[8pt]
\multicolumn{3}{l}{Processor (2,1)}&\multicolumn{3}{@{}l}{Processor (2,2)}
\\[4pt]
\({\tt NX1}={\tt 1}\)&\({\tt LX1}={\tt 1}\)&\({\tt UX1}={\tt 1}\)&
\({\tt NX1}={\tt 1}\)&\({\tt LX1}={\tt 1}\)&\({\tt UX1}={\tt 1}\) \\
\({\tt NX2}={\tt 2}\)&\({\tt LX2}={\tt 1}\)&\({\tt UX2}={\tt 2}\)&
\({\tt NX2}={\tt 1}\)&\({\tt LX2}={\tt 1}\)&\({\tt UX2}={\tt 1}\) \\
\({\tt ~NY}={\tt 1}\)&\({\tt ~LY}={\tt 5}\)&\({\tt ~UY}={\tt 5}\)&
\({\tt ~NY}={\tt 1}\)&\({\tt ~LY}={\tt 5}\)&\({\tt ~UY}={\tt 5}\) \\
\end{tabular}
\end{center}

An actual argument to an extrinsic procedure may be a pointer.  Since
the corresponding dummy argument may not have the {\tt POINTER}
attribute, the dummy argument becomes associated with the target of
the HPF global pointer.  In no way may a local pointer become pointer
associated with a global HPF target.  Therefore, an actual argument
may not be of a derived-type containing a pointer component.

\begin{rationale}
It is expected that global pointer variables will have a different
representation from that of local pointer variables, at least on
distributed memory machines, because of the need to carry additional
information for global addressing.\ This restriction could be lifted
in the future.
\end{rationale}

Other inquiry intrinsics, such as {\tt ALLOCATED} or {\tt PRESENT},
should also behave as expected.  Note that when a global array is
passed to a local routine, some processors may receive an empty
set of elements.

\subsection{Special Considerations for "HPF","SERIAL"}

There are restrictions that apply to an {\tt "HPF","SERIAL"} subprogram.

No {\it specification-directive}, {\it realign-directive}, or
{\it redistribute-directive} is permitted to be appear in an {\tt "HPF","SERIAL"}
subprogram or interface body.

\begin{rationale}
An HPF mapping directive would likely be meaningless in an {\tt
"HPF","SERIAL"} subprogram.  Note, however, the {\it
independent-directive} may appear in an {\tt "HPF","SERIAL"} subprogram,
since it may provide meaningful information to a compiler about a {\tt
DO} loop or a {\tt FORALL} statement or construct.
\end{rationale}

Any dummy data objects and any function result variables of an {\tt "HPF","SERIAL"}
procedure will be considered to be sequential.

An {\tt "HPF","SERIAL"} subprogram must not contain a definition of a common block
that has the same name as a common block defined in an HPF or "HPF","LOCAL"
program unit.  In addition, an {\tt "HPF","SERIAL"} subprogram must not contain a
definition of the blank common block if an HPF or "HPF","LOCAL" program unit has a
definition of the blank common block.

A dummy argument or function result variable of an {\tt "HPF","SERIAL"} procedure
that is referenced in global HPF must not have the {\tt POINTER} attribute.
A subobject of a dummy argument or function result of an {\tt "HPF","SERIAL"}
procedure that is referenced in global HPF, must not have the {\tt POINTER}
attribute.

A dummy argument of an {\tt "HPF","SERIAL"} procedure that is referenced in global
HPF and any subobject of such a dummy argument must not have the {\tt TARGET}
attribute.

A dummy procedure argument of an {\tt "HPF","SERIAL"} procedure must be an
{\tt "HPF","SERIAL"} procedure.

\CODE
      PROGRAM MY_TEST
        INTERFACE
          EXTRINSIC("HPF","SERIAL") SUBROUTINE GRAPH_DISPLAY(DATA)
            INTEGER, INTENT(IN) :: DATA(:, :)
          END SUBROUTINE GRAPH_DISPLAY
        END INTERFACE

        INTEGER, PARAMETER :: X_SIZE = 1024, Y_SIZE = 1024

        INTEGER DATA_ARRAY(X_SIZE, Y_SIZE)
!HPF$   DISTRIBUTE DATA_ARRAY(BLOCK, BLOCK)

!  Compute DATA_ARRAY
        ...
        CALL DISPLAY_DATA(DATA_ARRAY)
      END PROGRAM MY_TEST

! The definition of a graphical display subroutine.
! In some implementation-dependent fashion,
! this will plot a graph of the data in DATA.

      EXTRINSIC("HPF","SERIAL") SUBROUTINE GRAPH_DISPLAY(DATA)
        INTEGER, INTENT(IN) :: DATA(:, :)
        INTEGER :: X_IDX, Y_IDX

        DO Y_IDX = LBOUND(DATA, 2), UBOUND(DATA, 2)
          DO X_IDX = LBOUND(DATA, 1), UBOUND(DATA, 1)
          ...
          END DO
        END DO
      END SUBROUTINE GRAPH_DISPLAY
\EDOC

%\section{C Language Bindings}
%\label{ext-lang-c}
\input{c-interop-ext}

\section{Fortran Language Bindings}
\label{ext-lang-f}

When the language specified in an extrinsic definition is {\tt Fortran}
the rules are bascially the same as those for HPF because HPF is based
on the Fortran standard.  There are a few issues to consider in this case:
\begin{itemize}
\item Only Fortran constructs should be used.  Features such as
asynchronous I/O or the HPF library may not be supported.

\item It is recommended that Fortran language processors to be used
for this purpose be extendedto support the HPF local distribution
query routines and the associated {\tt HPF_LOCAL_LIBRARY}

\item Assuming the intent is to compiler the extrinsic routines with
a Fortran processors, these routines should be in separate files rather
than incorporated into  files with HPF source code.

\item The programmer should expect any HPF directives may be ignored.

\end{itemize}

%put section on F77 here when it materializes
%\section{Fortran 77 Language Bindings}
%\label{ch-f77-local-ext}
\input{f77-local-ext}

\section{The Extrinsic Library}
\label{ext-lib}

Following are Fortran bindings  for routines useful in intrinsic subprograms.

%\subsection {The Local Extrinsic Library}
\input{local-library}

\subsection {Library Access from Serial Extrinsics}

A {\tt SERIAL} subprogram may contain references to any
{\tt HPF_LIBRARY} procedure or HPF intrinsic function,
except {\tt HPF_ALIGNMENT}, {\tt HPF_DISTRIBUTION} or {\tt HPF_TEMPLATE}.
Within a {\tt SERIAL} scope the
{\tt HPF_LOCAL_LIBRARY} module must not be used.

References to the intrinsic functions {\tt NUMBER_OF_PROCESSORS} and
{\tt PROCESSORS_SHAPE} will return the same value as if the function
reference appeared in global HPF.
---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------