Subject: hpff-doc: hpf-craft-app.tex

From: Thomas MacDonald <[email-masked]>
Date: Mon, 30 Dec 1996 16:19:07 -0600 (CST)

---------------------------------------------------------------------------
[email-masked] is a mailing list for HPF 2.0 language specification
authors and editors.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------
I finished making the changes to the hpf-craft-app.tex Annex.  Basically
this involved corrections, wording changes, adding more examples, adding a
subsection that was left out (NO BARRIER).  I ran it throuh LaTeX and
found that all the paragraphs disappeared unless I put in blank lines.  I
used some things Andy set up fro me before he left so, I most likely am
doing something wrong.  Anyway, it's 99% complete :^)

Tom MacDonald
[email-masked]

=============== delete this line and everything above it ===============
% File: hpf-craft-app.tex
% Contents:
% Extrinsic Interfaces Appendix for HPF 2.0 document, including
%	policy
%	mechanism
%	HPF_CRAFT
%       proposed SPMD interface
% Revision history:
% May-10-96	Created by Charles Koelbel, Rice University
%		(from HPF 2.0 proposals)
% May 29-96	Mary Zosel, LLNL
%		fill in the extrinsic policy and procedures section
% Aug 12        Mary Zosel, LLNL
%               Update with HPF-CRAFT from Andy Meltzer  
% Dec 30        Tom MacDonald, SGI/Cray Research
%               Corrections and more examples
 
\chapter{HPF-Craft}
\label{ch-hpf-craft}
%\section*{\centering Abstract}
% IEEE allows italicized abstract
%{\em
HPF\_CRAFT is a hybrid language, combining an SPMD execution model with 
high performing HPF features.
The model combines the multi-threaded execution of HPF\_LOCAL and the
HPF syntax.  The goal of HPF\_CRAFT is to 
attain the potential performance of an SPMD programming model with 
access to HPF features and a well-defined extrinsic interface to HPF.
%}
\section{Introduction\label{sec:intro}}
HPF\_CRAFT is a hybrid language, combining an SPMD execution 
model with high performing and portable HPF features.
The model combines the multi-threaded execution of HPF\_LOCAL and the
HPF syntax and features.  The goal of HPF\_CRAFT is to 
attain the potential performance of an SPMD programming model with 
access to HPF features and a well-defined extrinsic interface to HPF.
It is built on top of the HPF\_LOCAL extrinsic environment.

SPMD features and a multi-threaded model allow the user to take
advantage of the performance and opportunity for low level access of 
a more general purpose programming model.  Including HPF data
distribution features gives the programmer access to high
performing aspects of both models, but with the added responsibility of 
working with a more low-level execution model.
HPF\_CRAFT is best suited for platforms that support
one way communication features, but is consistent with HPF
and easily targeted for platforms that have HPF and can support 
SPMD programming styles.

The HPF features included in HPF\_CRAFT are a subset of the full HPF 
language chosen for their performance and their broad portability and
ease of use.  HPF\_CRAFT contains additional features to support SPMD
programming styles.  There are some differences from HPF, however. 
For example, I/O causes differences; in HPF\_CRAFT different
processors are allowed to read from different files at the same time, in HPF
the processors must all read from the same file.   The differences in the
models are principally caused by the multi-threaded execution model and
the introduction of HPF\_LOCAL data rules.

HPF\_CRAFT allows for the notion of {\em private data}.  Data defaults
to a mapping in which data items are allocated so that each processor
has a unique copy.  The values of the individual
data items and the flow of control may vary from processor to
processor within HPF\_CRAFT. This behavior is consistent with the behavior
of HPF\_LOCAL.  In HPF\_CRAFT a processor
may be individually named and code executed based upon which processor
it is executing on.
HPF\_CRAFT also allows for the notion of {\em private loops}.  A private loop 
is executed in entirety by each processor.

The rules governing the interface to HPF\_CRAFT subprograms
are similar to those for the HPF\_LOCAL interface.  Dummy arguments
use a hybrid of the interfaces between HPF and
itself and that of HPF and HPF\_LOCAL.  Explicitly mapped dummy arguments
behave just as they do in HPF, while default (private) dummy arguments
use the HPF\_LOCAL calling convention.

HPF\_CRAFT will be initially made available on Cray MPP systems
and may also be available on Cray vector architectures.
Future versions of HPF\_CRAFT are possible on other vendor's 
architectures as well.   

HPF\_CRAFT is being implemented for Cray Research by The Portland Group, Inc.
For Cray systems, HPF\_CRAFT may be obtained through the Cray Research Inc.
Orderdesk,
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> Cray Research Inc. \\
\> [email-masked] \\
\> (612) 683-5907 \\
\end{tabbing}
Additional formal documentation, requests, and suggestions can be made to 
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> The Portland Group \\
\> 9150 SW Pioneer Ct., Suite H \\
\> Wilsonville, OR 97070 \\
\> (503) 682-2806 \\
\> [email-masked] \\
\end{tabbing}
	
\section{Examples of Use}
HPF\_CRAFT is intended for use in circumstances where greater control
and performance are desired for MIMD style architectures.   Since data
may be declared to be private, local control is made more available and
since processor information is available message passing and direct
memory access programming styles can be seamlessly integrated with
explicitly mapped data.

The following examples show some of the capabilities of HPF\_CRAFT that
are different from those of HPF.  Others, such as integrated message
passing and synchronization primitives are not shown.  Much of HPF can
also be used within HPF\_CRAFT.

Example 1 illustrates the difference between the default distribution
for data and the distribution of mapped data.

\begin{tabbing}
----------\=----\=----\=----\=---\=------------\=\kill
{\tt ! Example 1} \\
\\
\> {\tt INTEGER PRIVATE\_A(100, 20), PRIVATE\_B(12, 256), PRIVATE\_C} \\
\> {\tt INTEGER MAPPED\_A(100, 20), MAPPED\_B(12, 256), MAPPED\_C} \\
{\tt !HPF\$} \> {\tt DISTRIBUTE MAPPED\_A(BLOCK, BLOCK), MAPPED\_B(BLOCK, *), MAPPED\_C} \\
\end{tabbing}
In the above example, given
8 processors, there would be 8 * 100 * 20 (or  16,000) elements in the 
array {\tt PRIVATE\_A}.  Each processor contains an entire array named
{\tt PRIVATE\_A}. The elements of {\tt PRIVATE\_A} on processor 1
cannot be referenced using implicit syntax by any other processor.  There are
only 100 * 20 (or 2000) elements of array {\tt MAPPED\_A}, however, and these
elements are distributed about the machine in a {\tt (BLOCK, BLOCK)} fashion.

The difference between the {\tt PRIVATE\_A} declaration in HPF\_CRAFT and that
in HPF is the most instructive.   In HPF\_CRAFT
each processor contains one copy of the array, and the values of the elements
of the array may vary from processor to processor.   HPF
implementations are permitted to make
one copy of the array per processor the default, but
the values of these copies must remain coherent across all processors.
In HPF there is no way 
to write a conforming program in which different processors
have different values for the same array.

Example 2 shows the usefulness of the {\tt ON} clause
for the {\tt INDEPENDENT} loop as well as giving an example of how private
data may be used.

\begin{tabbing}
----------\=----\=----\=----\=---\=------------\=\kill
{\tt ! Example 2} \\
\\
\> {\tt PRIVATE\_C = 0} \\
{\tt !HPF\$} \> {\tt INDEPENDENT (I, J) ON MAPPED\_B(I, J)} \\
\>  {\tt DO J=1,256} \\
\>\>  {\tt DO I=1,12} \\
\>\>\>  {\tt MAPPED\_B(I, J) = MAPPED\_B(I, J) + 5} \\
\>\>\>  {\tt PRIVATE\_C = PRIVATE\_C + MAPPED\_B(I, J)} \\
\>\>  {\tt ENDDO} \\
\>  {\tt ENDDO} \\
\end{tabbing}
In this example, each iteration is executed on the
processor containing the data that is mapped to it.  The user was allowed
to specify this.

In addition, the private variable {\tt PRIVATE\_C} is used to compute a total
for each processor.  At the end of execution of the loop, the values
of {\tt PRIVATE\_C} may be different on each processor depending upon the
values in the elements of the array on each processor.  This data may 
be used as is, or it can be quickly summed using a barrier or an 
{\tt ATOMIC UPDATE}.

Example 3 shows the final total value being combined into the variable
{\tt MAPPED\_C} whose value is available to all processors.

\begin{tabbing}
----------\=----\=----\=----\=---\=------------\=\kill
{\tt ! Example 3} \\
\\
\> {\tt MAPPED\_C = 0} \\
{\tt !HPF\$} \> {\tt ATOMIC UPDATE} \\
\> {\tt MAPPED\_C = MAPPED\_C + PRIVATE\_C} \\
\>
\end{tabbing}

Example 4 shows how the language allows private data to vary from
processor to processor.

\begin{tabbing}
----------\=----\=----\=----\=---\=------------\=\kill
{\tt ! Example 4} \\
\\
\> {\tt IF (MY\_PE() .EQ. 5) THEN } \\
\>\> {\tt PRIVATE\_C = } {\em some-big-expression} \\
\> {\tt ENDIF} \\
\end{tabbing}
In this example, {\tt PRIVATE\_C} on processor 
5 will have the result of {\em some-big-expression}.  Each processor can do
distinctly different work and communicate through mapped data.

The code fragment in Example 5 is from an application and shows a few
features of the language.

\begin{tabbing}
----------\=----\=----\=----\=---\=------------\=\kill
{\tt ! Example 5} \\
\\
{\tt !HPF\$} \> {\tt GEOMETRY G(*, CYCLIC) }\\
\> {\tt REAL FX(100,100), FY(100,100), FZ(100,100) } \\
{\tt !HPF\$} \> {\tt DISTRIBUTE (G) :: FX,FY,FZ } \\
\> {\tt REAL FXP(100,16,100), FYP(100,16,100) } \\
{\tt !HPF\$} \> {\tt DISTRIBUTE FXP(*,*, BLOCK) FYP(*,*, BLOCK) } \\
\> {\tt INTEGER CELL, ATOM, MAP(1000), NACELL(1000) } \\
\\
{\tt !HPF\$} \> {\tt INDEPENDENT (CELL) ON FX(1,CELL) } \\
\> {\tt DO CELL=1,100 } \\
\>\>  {\tt JCELL0 = 16*(CELL-1) } \\
\>\>  {\tt DO NABOR = 1, 13 } \\
\>\>\>  {\tt JCELL = MAP(JCELL0+NABOR) } \\
\>\>\>  {\tt DO ATOM=1, NACELL(CELL) } \\
\>\>\>\>    {\tt FX(ATOM, CELL) = FX(ATOM, CELL) + FXP(ATOM, NABOR, JCELL) } \\
\>\>\>\>    {\tt FY(ATOM, CELL) = FY(ATOM, CELL) + FYP(ATOM, NABOR, JCELL) } \\
\>\>\>  {\tt ENDDO } \\
\>\>  {\tt ENDDO } \\
\>  {\tt ENDDO } \\
\end{tabbing}

The {\tt GEOMETRY} directive allows the user
to generically specify a mapping and use it to apply to many arrays (they
need not have the same extents.) 

Example 5 has a single {\tt INDEPENDENT} loop which is the outer loop.
It executes 100 iterations total.  Within this loop the 
private value of {\tt JCELL0}
is set for each processor (ensuring that it is a local computation everywhere.)
Nested inside the {\tt INDEPENDENT} loop is a private loop;  this loop executes
13 times {\em per} processor.  Inside this loop {\tt JCELL} is computed locally
on each processor, minimizing unnecessary communication.  Finally the innermost
loop is also private.  
\section{External Interface\label{sec:extern-interf}}
This section describes the behavior when an HPF\_CRAFT routine
is called from HPF.

The calling convention and argument passing rules for HPF\_CRAFT are
a hybrid of those for HPF calling HPF\_LOCAL and HPF calling
HPF.  Explicit interfaces are required.  Where dummy arguments
are private (default) storage, the HPF calling HPF\_LOCAL 
conventions are used.  Where dummy arguments are explicitly mapped,
the calling convention matches HPF calling HPF.  

There are a number of constraints on HPF\_CRAFT routines
that are called from HPF.  The following is a list of restrictions
placed on HPF\_CRAFT routines called from HPF:
\begin{itemize}
\item Recursive HPF\_CRAFT routines cannot be called from HPF.
\item HPF\_CRAFT routines called from HPF may only enter the routine at a
      single place (no alternate entries).
\item An HPF\_CRAFT supprogram may not be invoked directly or
      indirectly from within the body of a {\tt FORALL} construct or
      within the body of an {\tt INDEPENDENT DO} loop that is 
      inside an HPF program.
\item The attributes (type, kind, rank, optional, intent) of the dummy
      arguments in a supprogram called by HPF
      must match the attributes of the corresponding dummy 
      arguments in the explicit interface.  
\item A dummy argument of an HPF\_CRAFT supprogram called by HPF 
\begin{itemize}
\item     must not be a procedure name.
\item     must not have the {\tt POINTER} attribute.
\item     must not be sequential, unless it is also {\tt PE_PRIVATE}.
\item     must have assumed shape even when it is explicit shape in the 
          interface.
\item	  if scalar, it must be mapped so that each processor has a copy of 
          the argument.
\end{itemize}
\item The default mapping of scalar dummy arguments and of scalar function
      results when an HPF program calls an HPF\_CRAFT routine is that it
      is replicated on each processor.
\end{itemize}

If a dummy argument of an {\tt EXTRINSIC("HPF\_CRAFT")} routine interface
block is an array and the dummy argument of 
the HPF\_CRAFT supprogram has the default
private mapping, then the corresponding dummy argument in the
specification of the HPF\_CRAFT procedure must be an array of the same
rank, type, and type parameters.   When the extrinsic procedure is invoked,
the dummy argument is associated with the local array that consists of the
subgrid of the global array that is stored locally.

If the dummy argument of the HPF\_CRAFT supprogram
is explicitly mapped, it must have the same mapping as the dummy argument
of the {\tt EXTRINSIC("HPF\_CRAFT")} supprogram.   Note that this restriction
does not require actual and dummy arguments to match and is no more stringent
than saying that mappings of dummy arguments in interface blocks must 
match those in the actual routine.
\section{Execution Model\label{sec:exec-model}}
HPF\_CRAFT is built upon the fundamental execution model of HPF\_LOCAL,
augmented with data mapping and work distribution features from HPF.
It is also augmented with explicit low-level control features,
many taken from Cray Research's CRAFT language.

In HPF\_CRAFT there is a single task on each processor and 
all tasks begin executing in parallel, with data defaulting
to a private distribution, the same default distribution used in HPF\_LOCAL.
Each processor gets a copy of the data storage unless specified otherwise by 
the user.
Consequently I/O works identically to I/O in HPF\_LOCAL and 
message passing libraries are easily integrated.

Simply stated, the execution model is that of HPF\_LOCAL.

To provide correct behavior when explicitly mapped data is involved, 
this model defines implicit barrier points at which
the execution model requires that all processors must stop and wait 
for the execution of all
other processors before continuing.  These barriers add additional
semantics to the HPF\_LOCAL behavior.
An implementation may remove 
any of these barriers that are deemed unnecessary, but {\em every} 
processor must participate in the barriers at each one of these points.

The points where there are implicit barriers are conceptually after
those instances
in which the processors in the HPF\_CRAFT program are executing 
cooperatively, as if in an HPF program (e.g., after
an {\tt INDEPENDENT} loop).  An HPF\_CRAFT program treats
operations on explicitly mapped objects as if they were operations in
an HPF program and it treates operations on private data as if they
were executed within the HPF\_LOCAL framework.
It is occasionally useful for an advanced programmer to indicate 
to the compilation system where barriers are not needed; HPF\_CRAFT
has syntax to allow this capability.
\section{HPF\_CRAFT Functional Summary}
HPF\_CRAFT contains a number of features not available in HPF, and 
restricts the usage of many of the features currently available.  
The following is a concise list of the differences.
\begin{itemize}
\item {\tt INDEPENDENT} has been extended to better support an {\tt ON} clause.
\item There are new rules defining the interaction of explicitly 
      mapped and private data.
\item Parallel inquiry intrinsics {\tt IN\_PARALLEL()} and 
      {\tt IN\_INDEPENDENT()} have been added.
\item Serial regions ({\tt MASTER / END MASTER}) have been added.
\item Explicit synchronization primitives are provided.
\item The {\tt ATOMIC UPDATE}, {\tt SYMMETRIC}, and {\tt GEOMETRY} directives have been added.
\item Many other compiler information directives have been added to assist
      the compiler in producing good quality code.
\end{itemize}

\subsection{Data Mapping Features}
Data mapping features provided are those that have been found useful most 
often.  When data is explicitly mapped, only one copy of the data storage
is created unless the explicit mapping directs otherwise.  The
value of explicitly mapped replicated data items must be consistent
between processors as is the case in HPF.  
Storage and sequence association for explicitly mapped arrays is
not guaranteed in HPF\_CRAFT.  For private data, storage and sequence
association follows the Fortran 90 rules.

A new directive is included for completeness: {\tt PE\_PRIVATE}, which
specifies that the data should conform to the default behavior.
The values of private varaibles may vary on different processors.

\subsection{Subprogram Interfaces}
The behavior and requirements of an HPF\_CRAFT program at subprogram 
interfaces may be divided into three cases.  Each case is also
available using some combination of HPF and HPF\_LOCAL.
For dummy arguments that are explicitly mapped, the behavior
is identical to that of HPF.  
All processors must cooperate in a subprogram
invocation that remaps or explicitly maps data.  In other words, if
an explicit interface is required (by the HPF rules) or the 
subprogram declares explicitly mapped data, the subprogram must be called on
all processors.  Processors need not cooperate if there are only 
reads to non-local data.  The {\tt INHERIT} attribute may only be
applied to explicitly mapped data.

Data that has the default private mapping (case two)
the behavior of an HPF\_CRAFT subprogram at subprogram interfaces
is identical to that of HPF\_LOCAL.
Data is passed individually on every processor and the
processors need not interact in any way.   

When a subprogram is passed actual arguments that are a combination
of both explicitly mapped data and private data, the explicitly mapped
data follows the HPF rules and the private data follows the HPF\_LOCAL
rules.

In case three, the user has the option of passing data with explicitly
mapped actual arguments to dummy arguments that are not explicitly 
mapped (i.e., private.)  The mapping rules for this data 
are identical to the mapping
rules when HPF calls an HPF\_LOCAL subprogram.  The data remains ``in-place."
All HPF arrays are logically
carved up into pieces; the HPF\_CRAFT procedure executing on a 
particular physical processor sees an array containing just those
elements of the global array that are mapped to that physical 
processor.   There is implicit barrier synchronization
after an {\tt INDEPENDENT} loop.   Transfer of control into or out of
an {\tt INDEPENDENT} loop is prohibited.

Finally, it is undefined behavior when an actual argument is private and
the dummy argument is explicitly mapped.   A definition could 
be supplied for this interaction, but it is the same solution that 
one might propose for a calling sequence when HPF\_LOCAL subprograms call 
HPF subprograms.  

\subsection{The {\tt INDEPENDENT} directive}
The {\tt INDEPENDENT} directive is part of HPF\_CRAFT with the same semantics
as in HPF.  However, within {\tt INDEPENDENT} loops
the values of private data may vary from processor to processor.
{\tt INDEPENDENT} applied to {\tt FORALL} has identical syntax and 
semantics as in HPF.

An HPF independent loop optionally may have a {\tt NEW} clause. The {\tt NEW}
clause is not required by HPF\_CRAFT for default (not explicitly mapped)
data. In HPF\_CRAFT data defaults to
private so values may differ from processor to processor.

Private data has slightly different behavior than
data specified in the {\tt NEW} clause.  
The value of a private datum on each processor can be used beyond a single
iteration of the loop. 
Private data may be used to compute local sums, for example.  
The values of data items named in a {\tt NEW} clause
may not be used beyond a single iteration. The {\tt NEW} clause asserts that
the {\tt INDEPENDENT} directive would be valid if new objects were created for
the variables named in the clause for each iteration of the loop.
The semantics of the {\tt NEW} clause are identical in
HPF\_CRAFT and HPF. 

The semantics of an {\tt INDEPENDENT} applied to loops containing private data
references
changes with respect to the private data.  The change can be
summarized to say that instead of indicating that iterations have no
dependencies upon one-another, with respect to the private data, iterations 
on different processors have no dependencies upon one-another.

\subsection{{\tt ON} clause}
In addition to the version of {\tt INDEPENDENT} available from HPF,
a new version of {\tt INDEPENDENT} is included that incorporates
the {\tt ON} clause. There are a number of differences between the 
versions of {\tt INDEPENDENT} with and without the {\tt ON} clause.

The new version of the {\tt INDEPENDENT} directive may be applied to the 
first of a group of tightly nested loops and may apply to more than one 
of them. This more easily facilitates the use of the {\tt ON} clause. 
The current {\tt INDEPENDENT} directive applies only to a single loop nest.  
The {\tt INDEPENDENT} directive is extended so that multiple loop nests can
be named.
The general syntax for these new independent loops is as follows:
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$} \> {\tt INDEPENDENT} (I_1,I_2,\ldots,I_n) 
               {\tt ON} {\em array-name}(h_1(I_1),h_2(I_2),\ldots,h_n(I_n)) \\
\> {\tt DO} I_1 = L_1, U_1, S_1           \\
\>\> {\tt DO} I_2 = L_2, U_2, S_2         \\
\>\>\> \ldots                             \\
\>\>\>\> {\tt DO} I_n = L_n, U_n, S_n     \\
\>\>\>\>\> \ldots                         \\
\>\>\>\> {\tt END DO}                     \\
\>\>\> \ldots                             \\
\>\> {\tt END DO}                         \\
\> {\tt END DO}
\end{tabbing}

The syntax and semantics of {\tt INDEPENDENT} with the {\tt ON} 
clause are different from its syntax and semantics without the {\tt ON} 
clause. With the
{\tt ON} clause the directive states that there are no cross-processor
dependencies, but there may be dependencies between iterations on a
processor.   There is an implicit barrier synchronization after an
{\tt INDEPENDENT} loop.   Transfer of control into or out of
an {\tt INDEPENDENT} loop is prohibited.

The iteration space of an {\tt INDEPENDENT} nest must be rectangular.  
That is, the lower loop bound, the upper loop bound, and the step 
expression for each loop indicated by the {\tt INDEPENDENT} induction 
list must be invariant with regard to the {\tt INDEPENDENT} nest.  
Each index expression of {\em array-name} in the
{\tt ON} clause (the functions {\em h_i} above,) 
must be one of the following two forms:
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\bf [ }{\em a }{\tt * loop\_control\_variable + }{\bf ] }{\em b} \\
\> {\bf [ }{\em a }{\tt * loop\_control\_variable - }{\bf ] }{\em b}
\end{tabbing}
where {\em a} and {\em b} must be integer values; they can be 
expressions, constants,
or variables. The values of {\em a} and {\em b} must be
invariant with regard to the {\tt INDEPENDENT} loop nest.  
For example, specifying {\tt A(I,J,K)} is valid.  Specifying {\tt A(3,I+J,K)} 
is not valid.  Specifying {\tt A(I,I,K)} is not valid because {\tt I}
appears twice.   Division is prohibited in any index
expression of the {\tt ON} clause.  

\subsection{Array Syntax}
Array syntax is treated identically in HPF\_CRAFT as in HPF for 
explicitly mapped objects.   
For private objects the behavior is 
identical to that of HPF\_LOCAL.   When private objects and 
explicitly mapped objects are combined the rules are as follows:
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\em result} = {\em rhs_1} op_1 {\em rhs_2} op_2 ... op_m {\em rhs_n}
\end{tabbing}
\begin{itemize}
\item If {\em result} is explicitly mapped and all {\em rhs} arrays are 
      explicitly mapped, the work is distributed as in HPF.
\item If {\em result} is private and all {\em rhs} arrays are private the
      computation is done on all processors as an HPF\_LOCAL program
      would do it.
\item If {\em result} is private and all {\em rhs} arrays are explicitly 
      mapped, the 
      work is distributed as in HPF and the values of the results are 
      broadcast to the {\em result} on each processor.
\item If {\em result} is explicitly mapped and {\em not} all {\em rhs} 
      arrays are explicitly mapped, the results of the operation are undefined,
      unless all corresponding elements of all private {\em rhs} arrays
      have the same values.
\item If {\em result} is private and some, but not all {\em rhs} arrays are 
      explicitly mapped, the value is computed on each processor
      and saved to the local {\em result}.
\end{itemize}

All processors must participate in any array syntax 
statement in which the value of an explicitly mapped array is modified,
and there is implicit barrier synchronization
after the statement executes.

\subsection{{\tt FORALL} and {\tt WHERE} }
The {\tt FORALL} and {\tt WHERE}
statements are treated exactly as in HPF when data is
explicitly mapped.  When private data is modified, the statement is executed
separately on each processor.  Finally, when data in a {\tt FORALL}
or {\tt WHERE} are
mixed, the rules for array syntax apply.   If any explicitly mapped
data item is modified in a {\em forall-stmt} or {\em where-stmt}
then arrays in the 
{\em forall-header} or {\em where-header}
must be explicitly mapped.  In a {\tt FORALL}
construct, if any explicitly mapped array is modified, all modified
arrays must be explicitly mapped.   There is an implicit
barrier synchronization after {\tt FORALL} and {\tt WHERE}
statements if any arrays in the {\em forall-header}
or {\em where-header}
are explicitly mapped.

\subsection{Synchronization Primitives}
A number of synchronization primitives are provided.
These primitives include:
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> Barriers (test, set, wait)\\
\> Locks ( test, set, clear)\\
\> Critical Sections \\
\> Events (test, set, wait, clear)
\end{tabbing}

Barriers provides an explicit mechanism for a task to indicate its arrival
at a program point and to wait there until all other
tasks arrive.   A task may test and optionally wait
at an explicit barrier point.
In the following example, a barrier is used to make sure that {\em block3}
is not entered by any task until all tasks have completed execution of
{\em block1}.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\em block1} \\
\> {\tt CALL SET\_BARRIER()} \\
\> {\em block2} \\
\> {\tt CALL WAIT\_BARRIER()} \\
\> {\em block3} 
\end{tabbing}
The following example performs a similar function as above.  However, while
waiting for all tasks to arrive at the barrier, the early tasks
perform work within a loop.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\em block1} \\
\> {\tt CALL SET\_BARRIER()} \\
\> {\tt DO WHILE (.NOT. TEST\_BARRIER())} \\
\> \> {\em block2} \\
\> {\tt END DO} \\
\> {\em block3}
\end{tabbing}

Locks are used to prevent the simultaneous access of data by multiple tasks.  

The {\tt SET\_LOCK(}{\em lock}{\tt)} intrinsic sets the mapped
integer variable {\em lock} atomically.  If the lock is already set, the task 
that called {\tt SET\_LOCK} is suspended until the lock is cleared by 
another task and then sets it.  Individual locks may be tested or cleared
using {\em result = }{\tt TEST\_LOCK(}{\em lock}{\tt )}
and {\tt CLEAR\_LOCK(}{\em lock}{\tt )} respectively.

A {\em critical section} protects access to a section of code rather
than to a data object.  
The {\tt CRITICAL} directive marks the beginning of a code region in 
which only one task can enter at a time.  The {\tt END CRITICAL} directive 
marks the end of the critical section.   Transfer of control into or out
of a critical section is prohibited.

\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$} \> {\tt CRITICAL} \\
\> {\tt GLOBAL\_SUM = GLOBAL\_SUM + LOCAL\_SUM} \\
{\tt !HPF\$} \> {\tt END CRITICAL} \\
\end{tabbing}

Events are typically used to record the state of a program's execution
and to communicate that state to another task.  Because they do not set
locks, as do the lock routines described earlier, they cannot easily be
used to enforce serial access of data.  They are suited to work such as
signalling other tasks when a certain value has been located in a search
procedure.  There are four routines needed to perform the event functions,
and each requires a mapped argument.

The {\tt SET\_EVENT(}{\em event}{\tt)} routine
sets or {\em posts} an event; it declares that an action has been
accomplished or a certain point in the program has been reached.  A
task can post an event at any time, whether the state of the event 
is cleared or already posted.  The {\tt CLEAR\_EVENT(}{\em event}{\tt)}
routine clears an event, the {\tt WAIT\_EVENT(}{\em event}{\tt)} routine
waits until a particualr event is posted, and the
{\em result = }{\tt TEST\_EVENT(}{\em event}{\tt)} function returns
a logical value indicating whether a particular event has been posted.

\subsection{Barrier Removal}
You can explicitly remove an implicit barrier after
any {\tt INDEPENDENT} loop,
or after any array syntax 
statement that modifies explicitly mapped arrays,
by using the {\tt NO BARRIER} directive.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$} \> {\tt NO BARRIER }
\end{tabbing}

\subsection{Serial Regions}
It is often useful to enter
a region where only one task is executing.  This is particularly 
useful for certain types of I/O.   To facilitate this, two directives
are provided.
In addition, one may optionally attach a {\tt COPY} clause to the 
{\tt END MASTER} directive which specifies the private 
data items whose
values should be broadcast to all processors.  The syntax of this 
directive is:
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$     } \> {\tt MASTER }\\
\>{\em sequential region} \\
\> \ldots \\
{\tt !HPF\$} \> {\tt END MASTER }{\bf [}{\tt , COPY(} {\em var_1 }{\bf [}{\tt,} {\em var_2}{\tt, \ldots , }{\em var_n} {\bf ]}{\tt )}{\bf ]} 
\end{tabbing}
where {\em var} is {\tt SYMMETRIC} private data to be copied to the same named
private data on other processors.

If a routine is called within a serial region, the routine
executes serially; there is no way to get back to parallel execution
within the routine.  All explicitly mapped data is accessible from 
within routines called in a serial region, but a routine called
from within a serial region cannot allocate explicitly mapped data
or remap data.
All processors must participate in the invocation of the serial region.
Transfer of control into or out of a serial region is not permitted.

\subsection{Libraries}
The HPF Local Routine Library is available in HPF\_CRAFT.
The HPF\_LOCAL extrinsic environment contains a number of libraries
that are useful for local SPMD programming and a number of libraries
that allow the user to determine global (rather than local) state
information.  These library procedures take as input the name of
a dummy argument and return information on the corresponding global
HPF actual argument.  They may only be invoked by an HPF\_CRAFT
procedure that was directly invoked by global HPF code.  They may
be called only for private data.   The libraries reside in a module
called {\tt HPF\_LOCAL\_LIBRARY}. 

The HPF Library is available to HPF\_CRAFT when called with data that is
explicitly mapped and all processors are participating in the call.
In addition, as in HPF\_LOCAL, the entire HPF Library is available for
use with private data.  Mixing private and explicitly mapped data in
calls to the HPF library produces undefined behavior.

\subsection{Parallel Inquiry Intrinsics}
These intrinsic functions are provided as an extension to HPF.  They return
a logical value that provides information to the programmer
about the state of execution in a program.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\tt IN\_PARALLEL()} \\
\> {\tt IN\_INDEPENDENT()} 
\end{tabbing}

\subsection{Task Identity}
{\tt MY\_PE()} may be used to return the local processor number.  
The physical processors are
identified by an integer in the range of 0 to {\em n-1} where {\em n}
is the value returned by the global HPF\_LIBRARY function 
{\tt NUMBER\_OF\_PROCESSORS}.  Processor identifiers are returned
by {\tt ABSTRACT\_TO\_PHYSICAL}, which establishes the one-to-one
correspondence between the abstract processors of an HPF processors
arrangement and the physical processors.  Also, the local library
function {\tt MY\_PROCESSOR} returns the identifier of the task 
executing the call.

\subsection{Parallelism Specification Directives}
These directives allow a user to assert that a routine will only be
called from within a parallel region, a serial region, or from within
both regions.   Without these directives an implementation might be
required to generate two versions of code for each routine, depending
upon implementation strategies.  The directives simply make the 
generated code size smaller and remove a test.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$  } \> {\tt PARALLEL\_ONLY} \\
{\tt !HPF\$  } \> {\tt SERIAL\_ONLY} \\
{\tt !HPF\$  } \> {\tt PARALLEL\_AND\_SERIAL} \\
\end{tabbing}
The default is {\tt PARALLEL\_ONLY}.

\subsection{{\tt SYMMETRIC}}
{\tt SYMMETRIC} variables are private data that are guaranteed to be at the
same storage location on every processor.  The feature is beneficial to
implementations that provide one-way communication functionality.   One
task can either {\em get} or {\em put} data into another task's 
symmetric data location, without involving the other task.  
There is an implicit barrier synchronization after
{\tt SYMMETRIC} data is allocated.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\tt REAL PRIV1(100), PRIV2} \\
{\tt !HPF\$} \> {\tt SYMMETRIC PRIV1, PRIV2}
\end{tabbing}

\subsection{{\tt RESIDENT}}
The {\tt RESIDENT} directive can be specified at the loop level and at the
routine level.  It is an assertion that the references to particular
variables in the routine (or loop) are only references to data that are
local to the task making the assertion.   In the following loop, all
references to arrays {\tt A}, {\tt B}, and {\tt C} are local to the
task executing each iteration.

\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\tt REAL A(100), B(100), C(100)}   \\
\> {\tt INTEGER IX(100)}               \\
{\tt !HPF\$} \> {\tt DISTRIBUTE A(BLOCK), B(BLOCK), C(BLOCK)} \\
{\tt !HPF\$} \> {\tt RESIDENT A}       \\
\\
\> \ldots \\
{\tt !HPF\$} \> {\tt INDEPENDENT (I) ON B(I) RESIDENT(C)} \\
\> {\tt DO I = 1, 100}                \\
\> \> {\tt A(IX(I)) = B(I) + C(IX(I))} \\
\> {\tt END DO}
\end{tabbing}

\subsection{{\tt ATOMIC UPDATE}}
In HPF\_CRAFT, the {\tt ATOMIC UPDATE} directive tells the compiler
that a particular data item or the elements of a particular array for
a specified operation must be updated atomically.  This can be used
within loops or in array syntax and applies to both the elements of
an array with an assignment of a permutation and the elements of 
an array within a loop.

In the following example, all references to
{\tt R(IX(I))} occur atomically, thus eliminating the possibility that
different iterations might try to modify the same element concurently.

\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
\> {\tt REAL R(200), S(1000)}          \\
\> {\tt INTEGER IX(1000)}              \\
{\tt !HPF\$} \> {\tt DISTRIBUTE R(BLOCK), S(BLOCK), IX(BLOCK)} \\
\\
{\tt !HPF\$} \> {\tt INDEPENDENT (I) ON S(I)} \\
\> {\tt DO I = 1, 1000}                \\
{\tt !HPF\$} \> \> {\tt ATOMIC UPDATE} \\
\> \> {\tt R(IX(I)) = R(IX(I)) + S(I)} \\
\> {\tt END DO}
\end{tabbing}

\subsection{{\tt GEOMETRY}}
The {\tt GEOMETRY} directive is simliar to a {\tt typedef}
in C, only it is for data mapping.   It allows the
user to conveniently change the mappings of many arrays at the same
time.   It is similar in many ways to the {\tt TEMPLATE} directive, but
since it is bound to no particular extent it is sometimes easier to apply.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$} \> {\tt GEOMETRY} {\em geom}{\tt(}{\em d_1} {\tt[}{\em , d_2, ..., d_n}{\tt])} \\
{\tt !HPF\$} \> {\tt DISTRIBUTE (} {\em geom} {\tt ) [::]} {\em var_1{\tt[,} var_2{\tt ,} \ldots {\tt ,} var_m}{\tt ]} 
\end{tabbing}
Where {\em d_i} indicates one of the allowable distribution formats.
\begin{tabbing}
---------\=---------\=---\=---\=---\=------------\=\kill
{\tt !HPF\$} \> {\tt GEOMETRY GBB(BLOCK, CYCLIC)} \\
\> {\tt REAL A(300,300), B(400,400)} \\
{\tt !HPF\$} \> {\tt DISTRIBUTE (GBB) :: A, B} \\
{\tt !} \> {\em if }{\tt GBB }{\em changes then both }{\tt A }{\em and }{\tt B }{\em change}
\end{tabbing}
---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------