Subject: hpff-doc: overview.tex

From: P Sadayappan <[email-masked]>
Date: Sun, 8 Sep 1996 19:57:30 -0500 (EDT)

---------------------------------------------------------------------------
[email-masked] is a mailing list for HPF 2.0 language specification
authors and editors.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------
% File: overview.tex

% Contents:
% Overview for HPF 2.0 document, including
%	Document/language structure
%	Brief descriptions of new features & extensions
%	Fortran language base
%	Notation and Syntax

% Revision history:
% May-10-96	Created by Charles Koelbel, Rice University
%		(copied from HPF 1.1 document)
% June 26-96	
% August 20-96  Edited by P. Sadayappan, Ohio State University
% September 8-96  Edited by P. Sadayappan, Ohio State University

\chapter{Overview}
\label{ch-overview}

{\em
Comments on this chapter should be directed to 
P. Sadayappan ({\tt [email-masked]}),
and {\tt [email-masked]}.
Please use ``{\tt Comments on Overview}'' as the {\tt Subject:}
line.
\par
}

%Please be sure to add this material to some file that glombnf
%will read--- overview.tex  would be fine:
%Check these rule numbers versus the Fortran 95 specification!!!

\Fortranrule{R710}{add-op}
\Fortranrule{R706}{add-operand}
\Fortranrule{R625}{allocate-object}
\Fortranrule{R622}{allocate-stmt}
\Fortranrule{R431}{array-constructor}
\Fortranrule{R512}{array-spec}
\Fortranrule{R735}{assignment-stmt}
\Fortranrule{R838}{assign-stmt}
\Fortranrule{R1210}{call-stmt}
\Fortranrule{R529}{data-stmt}
\Fortranrule{R631}{deallocate-stmt}
\Fortranrule{R1221}{dummy-arg}
\Fortranrule{R1218}{end-function-stmt}
\Fortranrule{R1222}{end-subroutine-stmt}
\Fortranrule{R504}{entity-decl}
\Fortranrule{R215}{executable-construct}
\Fortranrule{R208}{execution-part}
\Fortranrule{R513}{explicit-shape-spec}
\Fortranrule{R723}{expr}
\Fortranrule{R1209}{function-reference}
\Fortranrule{R1215}{function-subprogram}
\Fortranrule{R914}{input-item}
\Fortranrule{R728}{int-expr}
\Fortranrule{R607}{int-variable}
\Fortranrule{R1204}{interface-body}
\Fortranrule{R210}{internal-subprogram-part}
\Fortranrule{R707}{level-2-expr}
\Fortranrule{R741}{mask-expr}
\Fortranrule{R705}{mult-operand}
\Fortranrule{R737}{namelist-group-object}
\Fortranrule{R543}{namelist-stmt}
\Fortranrule{R629}{nullify-stmt}
\Fortranrule{R915}{output-item}
\Fortranrule{R844}{pause-stmt}
\Fortranrule{R736}{pointer-assignment-stmt}
\Fortranrule{R630}{pointer-object}
\Fortranrule{R737}{read-stmt}
\Fortranrule{R734}{specification-expr}
\Fortranrule{R204}{specification-part}
\Fortranrule{R618}{section-subscript}
\Fortranrule{R623}{stat-variable}
\Fortranrule{R842}{stop-stmt}
\Fortranrule{R620}{stride}
\Fortranrule{R617}{subscript}
\Fortranrule{R619}{subscript-triplet}
\Fortranrule{R737}{target}
\Fortranrule{R501}{type-declaration-stmt}
\Fortranrule{R502}{type-spec}
\Fortranrule{R601}{variable}
\Fortranrule{R739}{where-construct}
\Fortranrule{R738}{where-stmt}
\Fortranrule{R737}{write-stmt}

This document specifies the form and establishes the interpretation of
programs expressed in the High Performance Fortran (HPF) language.  It
is designed as a set of extensions and modifications to the established
International Standard for Fortran. At the time of publication of this document,
the version of the standard used as a base is informally referred to as 
``Fortran~95'' (revision of ISO/IEC 1539:1991(E) and ANSIX3.198-1992 
(\cite{F9091}, to be published in October 1996).  In this overview 
chapter of the document, we outline the goals and scope of the language,
introduce the HPF language model, highlight the main features of the language,
and provide a guide to the rest of this document.

\section{Goals and Scope of High Performance Fortran}
\label{overview-goals}

The primary goals behind the development of the HPF language included:
\begin{itemize}

\item Portability across different architectures;

\item Data parallel programming (defined as single threaded, global name
space, and loosely synchronous parallel computation);

\item High performance on parallel computers with non-uniform memory
access costs (while not impeding performance on other machines); 

\item Use of Standard Fortran (currently Fortran 95) as a base; and

\item Open interfaces and interoperability with other languages (e.g. C)
and other programming paradigms (e.g. message passing using MPI).
\end{itemize}

Secondary goals included:
\begin{itemize}

\item Keeping the language understandable and implementable in a short
time span;

\item Provision of input to future standards activities for Fortran and C;

\item Provision of an evolutionary path for adding advanced features to
the base language in a consistent manner.
\end{itemize}

The first version of the language definition, HPF 1.0 was released in
May 1993. A number of language features that were defined in HPF 1.0
have now been absorbed into the current Fortran (95) language standard
(e.g. the {\tt FORALL} construct, {\tt PURE} and {\tt ELEMENTAL} procedures). 
These features
are therefore no longer detailed in the definition of HPF 2.0.
Information about the evolution of the HPF language (through versions 1.0,
1.1, and 2.0) and an enumeration of the differences between HPF 2.0 from
HPF 1.1 may be found in Annex~\ref{HPF-EVOLUTION}.

\section{HPF Language Model}

An important goal of HPF is to achieve code portability across a
variety of parallel machines.  This requires not only that HPF programs
compile on all target machines, but also that a highly-efficient HPF
program on one parallel machine be able to achieve reasonably high
efficiency on another parallel machine with a comparable number of
processors.  Otherwise, the effort spent by a programmer to achieve
high performance on one machine would be wasted when the HPF code is
ported to another machine.  Although 
shared-memory machines and distributed-memory machines may use 
different low-level primitives, there is broad
similarity with respect to the fundamental factors that affect the
performance of parallel programs on these machines. Thus, achieving high
efficiency across different parallel machines with the same high level
HPF program is a feasible goal. Some of the fundamental factors affecting
the performance of a parallel program are the degree of available parallelism,
exploitation of data locality, and choice of appropriate task granularity.
HPF provides mechanisms for the programmer to guide the compiler with respect
to these factors.

The first versions of HPF were defined as an extension of Fortran~90.
HPF 2.0 is defined as an extension to the current Fortran Standard (Fortran~95).
HPF will include and be consistent with advances in the Fortran standards, 
as they are approved by ISO.

Building on Fortran, HPF language features fall into four categories:

\begin{itemize}

\item HPF directives;

\item New language syntax;

\item New library routines; and

\item Language changes and restrictions.

\end{itemize}

HPF directives appear as structured comments that suggest implementation
strategies or assert facts about a program to the compiler.  They may
affect the efficiency of the computation performed, but generally do not change
the value computed by the program.  The form of the HPF directives has
been chosen so that a future Fortran standard may choose to include
these features as full statements in the language by deleting the
initial comment header.

A few new language features have been defined as direct extensions to 
Fortran syntax and interpretation.
The new HPF language features differ from HPF directives in that they 
are first-class 
language constructs rather than comments. They can directly affect the
result computed by a program. 

The HPF library of computational functions defines a standard interface
to routines that have proven valuable for high performance computing,
including additional reduction functions, combining scatter functions,
prefix and suffix functions, and sorting functions.

A small number of changes and restrictions to standard Fortran have also
been defined. The most significant restrictions are those imposed on the use of 
sequence and storage association, since they are not compatible with the use of
data distribution features of HPF. It is however possible to retain sequence
and storage association semantics in a program by use of certain explicit 
HPF directives.

The fundamental model of parallelism in HPF is that of single-threaded
data-parallel execution with a globally shared address space. Fortran 
array statements and the {\tt FORALL} statement are natural ways of
specifying data parallel computation. In addition, HPF provides the
{\tt INDEPENDENT} directive. It can be used to assert that certain loops
do not carry any dependences and therefore may be executed in parallel.

Exploitation of data locality is critical to achieving good performance on
high-performance computers, whether it be a uniprocessor workstation or a
parallel computer. On a Non-Uniform-Memory-Access (NUMA) parallel computer, 
the effective distribution of data among processor memories is very important
in reducing data movement overheads. One of the key features of HPF is the
facility for user specification of distribution of data. HPF provides a 
logical view of the parallel machine as a rectilinear arrangement of abstract
processors in one or more dimensions. The programmer can specify the relative
alignment of elements of different program arrays, and the distribution of 
arrays over the logical processor grid. Data mapping is specified using HPF
directives that can aid the compiler in optimizing parallel performance, 
but have no effect on the semantics of the program.

While HPF's single-threaded data-parallel model with a global name space 
is very convenient in many application contexts, other programming
languages (e.g. C) and other parallel programming paradigms (e.g. explicit
message-passing using MPI) may be more appropriate in certain contexts. In
recognition of this need, HPF formally defines the {\tt Extrinsic} mechanism 
to facilitate interoperability with other programming languages 
and/or paradigms.

The key concepts of HPF are illustrated below using simple examples.

                                                                \CODE
      REAL A(1000,1000)
!HPF$ PROCESSORS procs(4,4)
!HPF$ DISTRIBUTE (BLOCK,BLOCK) ONTO procs :: A
      DO k = 1, num_iter
         FORALL (i=2:999, j=2,999)
           A(i,j) = (A(i,j-1)+A(i-1,j)+A(i,j+1)+A(i+1,j))/4
         END FORALL
      END DO
                                                                \EDOC

The code fragment describes a simple Jacobi relaxation computation on
a two-dimensional floating-point array {\tt A}. The HPF directives
appear as structured comments. The {\tt PROCESSORS} directive specifies a
logical \( 4 \times 4 \) grid of processors {\tt proc}. The {\tt DISTRIBUTE} 
directive recommends that the compiler partition the array {\tt A} into
equal-sized blocks along each of its dimensions. This will result in a 
\( 4 \times 4 \)
configuration of blocks each containing \( 250 \times 250 \) elements, 
one block per processor. 
The {\tt PROCESSORS} and {\tt DISTRIBUTE} directive are described in detail
later in  Chapter~\ref{ch-mapping-base}.

The outer {\tt Do k} loop iterates over {\tt num\_iter}
Jacobi relaxation steps. The inner loop uses the Fortran~95 {\tt FORALL}
construct. It specifies the execution of the loop body for all values of
{\tt i} and {\tt j} in the range {\tt 2:99}. The semantics of the {\tt FORALL}
require that the right-hand-side expressions for all iterations
(i.e. for all values of {\tt i} and {\tt j} between {\tt 2}
and {\tt 99}) be evaluated before the assignments to the 
left-hand-side variables are performed.

When targeted for execution on a distributed-memory machine with {\tt 16}
processors, the HPF compiler generates SPMD code, with each processor 
locally containing a part of the global array {\tt A}. The outer {\tt k} loop
is executed sequentially while the inner {\tt FORALL} loop
is executed in parallel. Each processor will require some ``boundary'' elements
of {\tt A} that reside in partitions mapped to the local memories of other
processors. Message-passing primitives to achieve the necessary inter-processor
communication are inserted by the HPF compiler into the generated SPMD code.
The single-threaded data-parallel model with a global name-space makes it
convenient for the programmer to specify the strategy for parallelization 
and data partitioning at a higher level of abstraction. The tedious low-level
details of translating from an abstract global name space to the local
memories of individual processors and the management of explicit 
inter-processor communication through message-passing are left to the compiler.

The following example illustrates some of the communication implications of
scalar assignment statements.  The purpose is to illustrate the
implications of data distribution specifications on communication
requirements for parallel execution.  The explanations given do not
necessarily reflect the actual compilation process.

Consider the following statements:
								\CODE
      REAL a(1000), b(1000), c(1000), x(500), y(0:501)
      INTEGER inx(1000)
!HPF$ PROCESSORS procs(10)
!HPF$ DISTRIBUTE (BLOCK) ONTO procs :: a, b, inx
!HPF$ DISTRIBUTE (CYCLIC) ONTO procs :: c
!HPF$ ALIGN x(i) WITH y(i+1)
      ...
      a(i) = b(i)                    ! Assignment 1
      x(i) = y(i+1)                  ! Assignment 2
      a(i) = c(i)                    ! Assignment 3
      a(i) = a(i-1) + a(i) + a(i+1)  ! Assignment 4
      c(i) = c(i-1) + c(i) + c(i+1)  ! Assignment 5
      x(i) = y(i)                    ! Assignment 6
      a(i) = a(inx(i)) + b(inx(i))   ! Assignment 7
								\EDOC

In this example, the {\tt PROCESSORS} directive specifies a linear
arrangement of 10 processors.  The {\tt DISTRIBUTE} directives
recommend to the compiler that the arrays {\tt a}, {\tt b}, and {\tt
inx} should be distributed among the 10 processors with blocks of 100
contiguous elements per processor. The array {\tt c} is to be
cyclically distributed among the processors with {\tt c(1)}, {\tt
c(11)}, \ldots , {\tt c(991)} mapped onto processor {\tt procs(1)};
{\tt c(2)}, {\tt c(12)}, \ldots , {\tt c(992)} mapped onto processor
{\tt procs(2)}; and so on.  The complete mapping of arrays {\tt x} and
{\tt y} onto the processors is not specified, but their relative
alignment is indicated by the {\tt ALIGN} directive.  The {\tt ALIGN}
statement causes {\tt x(i)} and {\tt y(i+1)} to be stored on the same
processor for all values of {\tt i}, regardless of the actual
distribution chosen by the compiler for {\tt x} and {\tt y} ({\tt y(0)}
and {\tt y(1)} are not aligned with any element of {\tt x}).  The {\tt
PROCESSORS}, {\tt DISTRIBUTE}, and {\tt ALIGN} directives are discussed
in detail in Chapter~\ref{ch-mapping-base}.

In Assignment 1 ({\tt a(i) = b(i)}), the identical distribution of {\tt
a} and {\tt b} ensures that for all {\tt i}, {\tt a(i)} and {\tt b(i)}
are mapped to the same processor.  Therefore, the statement requires no
communication.

In Assignment 2 ({\tt x(i) = y(i+1)}), there is no inherent
communication.  In this case, the relative alignment of the two arrays
matches the assignment statement for any actual distribution of the
arrays.

Although Assignment 3 ({\tt a(i) = c(i)}) looks very similar to the
first assignment, the communication requirements are very different due
to the different distributions of {\tt a} and {\tt c}.  Array elements
{\tt a(i)} and {\tt c(i)} are mapped to the same processor for only
10\% of the possible values of {\tt i}.  (This can be seen by
inspecting the definitions of {\tt BLOCK} and {\tt CYCLIC} in
Chapter~\ref{ch-mapping-base}.) The elements are located on the same
processor if and only if \( \lfloor (i-1) / 100 \rfloor = (i-1) \bmod
10 \).  For example, the assignment involves no inherent communication
(i.e.,\ both {\tt a(i)} and {\tt c(i)} are on the same processor) if \(
i = 1 \) or \( i = 102 \), but does require communication if \( i = 2
\).

In Assignment 4 ({\tt a(i) = a(i-1) + a(i) + a(i+1)}), the references
to array {\tt a} are all on the same processor for about 98\% of the
possible values of {\tt i}.  The exceptions to this are \( i = 100k \)
for any \( k = 1, 2, \ldots, 9 \), (when {\tt a(i)} and {\tt a(i-1)}
are on {\tt procs(k)} and {\tt a(i+1)} is on {\tt procs(k+1)}) and  \(
i = 100k + 1 \) for any \( k = 1, 2, \ldots, 9 \) (when {\tt a(i)} and
{\tt a(i+1)} are on {\tt procs(k+1)} and {\tt a(i-1)} is on {\tt
procs(k)}).  Thus, except for ``boundary" elements on each processor,
this statement requires no inherent communication.

Assignment~5, {\tt c(i) = c(i-1) + c(i) + c(i+1)}, while superficially
similar to Assignment 4, has very different communication behavior.
Because the distribution of {\tt c} is {\tt CYCLIC} rather than {\tt
BLOCK}, the three references {\tt c(i)}, {\tt c(i-1)}, and {\tt c(i+1)}
are mapped to three distinct processors for any value of {\tt i}.
Therefore, this statement requires communication for at least two of
the right-hand side references, regardless of the implementation
strategy.

The final two assignments have very limited information regarding the
communication requirements.  In Assignment~6 ({\tt x(i) = y(i)}) the
only information available is that {\tt x(i)} and {\tt y(i+1)} are on
the same processor; this has no logical consequences for the
relationship between {\tt x(i)} and {\tt y(i)}.  Thus, nothing can be
said regarding communication in the statement without further
information.  In Assignment 7 ({\tt a(i) = a(inx(i)) + b(inx(i))}), it
can be proved that {\tt a(inx(i))} and {\tt b(inx(i))} are always
mapped to the same processor.  Similarly, it is easy to deduce that
{\tt a(i)} and {\tt inx(i)} are mapped together.  Without knowledge of
the values stored in {\tt inx}, however, the relation between {\tt
a(i)} and {\tt a(inx(i))} is unknown, as is the relationship between
{\tt a(i)} and {\tt b(inx(i))}.

\section{Overview of HPF~2.0 Language Features}
\label{overview-hpf2}

The definition of the HPF 2.0 language comprises of two main parts:
\begin{itemize}
\item The Base HPF~2.0 Language 
\item HPF~2.0 Approved Extensions
\end{itemize}
The base language features include those features that are expected of every
HPF~2.0 implementation within a few months of release of the 
language specification. This includes basic data distribution features,
data parallel features, a number of intrinsic and library routines, and
the extrinsic mechanism. The base language features are first described,
followed by the approved language extensions. 

\subsection{Base HPF~2.0 Language Features}

\subsubsection*{Data Distribution Features}

Most parallel and sequential architectures attain their highest speed
when the data accessed exhibits locality of reference. The sequential
storage order implied by Fortran standards often conflicts with
the locality demanded by the architecture. To avoid this, HPF includes
features which describe the collocation of data ({\tt ALIGN}) and the
partitioning of data among memory regions or abstract processors ({\tt
DISTRIBUTE}). Compilers may interpret these annotations to improve
storage allocation for data, subject to the constraint that
semantically every data object has a single value at any point in
the program.  In all cases, users should expect the compiler to arrange
the computation to minimize communication while retaining parallelism.
Chapter~\ref{ch-mapping-base} describes the distribution features and 
Chapter~\ref{ch-mapping-subr} defines how the mapping features
interact across subprogram boundaries.

\subsubsection*{Data Parallel Execution Features}

To express parallel computation explicitly, HPF extends 
Fortran with a new statement
and a  directive. The {\tt REDUCE} allows the user to identify
simple incremental operations inside loops as candidates for
parallelism. 
The {\tt INDEPENDENT} directive asserts that the statements in a
particular section of code do not exhibit any sequentializing
dependencies; when properly used, it does not change the semantics of the
construct, but may provide more information to the language processor
to allow optimizations. Chapter~\ref{ch-parallel} describes these
features.  

\subsubsection*{Extended Intrinsic Functions and Standard Library}

Experience with massively parallel machines has identified several
basic operations that are very valuable in parallel algorithm design.
The Fortran array intrinsics address some of these, but not
all.  HPF adds several classes of parallel
operations to the language definition as intrinsic functions and as standard
library functions.  In addition, several system inquiry functions
useful for controlling parallel execution are provided in HPF.
Chapter~\ref{ch-library} describes these functions and subroutines.

\subsubsection*{Extrinsic Procedures}

Because HPF is designed as a high-level, machine-independent language,
there are certain operations that are difficult or impossible to
express directly.  For example, many applications benefit from
finely-tuned systolic communications on certain machines; HPF's global
address space does not express this well.  Extrinsic procedures define
an explicit interface to procedures written in other paradigms, such as
explicit message-passing subroutine libraries.  Chapter~\ref{ch-extrinsics}
describes this interface. Specific extrinsic
interfaces are defined in Chapter~\ref{ch-extrinsic-ext} and 
Appendix~\ref{ch-extrinsic-app}.

\subsubsection*{Sequence and Storage Association}

A goal of HPF is to maintain compatibility with Fortran.  Full
support of Fortran sequence and storage association, however, is not
compatible with the goal of high performance through distribution of
data in HPF.  Some forms of associating subprogram dummy arguments
with  actual values make assumptions about the sequence of values in
physical memory which may be incompatible with data distribution.
{\tt COMMON} and {\tt EQUIVALENCE} statements are recognized as
requiring a modified storage association paradigm. HPF
provides a directive to assert that full sequence and storage
association for designated variables must be maintained.  In the absence
of such explicit directives, reliance on the properties of association
is not allowed.  An optimizing compiler may then choose to distribute
any variables across processor memories in order to improve
performance.  To protect program correctness, a given implementation
should provide a mechanism to ensure that all such default optimization
decisions are consistent across an entire program.
Chapter~\ref{ ch-mapping-subr } describes the restrictions and directives
related to storage and sequence association.  

\subsection{HPF~2.0 Approved Extensions}

\subsubsection*{Extensions for Data Mapping}

The extended mapping features permit greater control over the mapping
of data, including facilities for dynamic realignment and redistribution
of arrays at run-time ({\tt REALIGN, REDISTRIBUTE, DYNAMIC} directives), 
mapping of data among subsets of processors, and
support for irregular distribution of data ({\tt GEN\_BLOCK} and 
{\tt INDIRECT} distributions). In addition, mechanisms are
defined that permit the programmer to provide information to the compiler
about the range of possible distributions an array might take ({\tt RANGE}
directive) and the amount of buffering to be used with arrays involved
in stencil-based nearest-neighbor computations ({\tt SHADOW}).
The approved language extensions pertaining to data mapping are described
in Chapter~\ref{ch-mapping-ext}.

\subsubsection*{Extensions for Data and Task Parallelism}

The extended features facilitate explicit computation partitioning
through use of the {\tt ON} directive. The site of recommended execution
of a loop iteration can either be explicitly specified as a specific
processor or indirectly as the one on which a particular variable or
template element is mapped. In order to assist the compiler in
generating efficient code, the {\tt RESIDENT} directive is defined, to be
used in conjunction with an {\tt ON} directive by the programmer. It can
be used to assert that all accesses to the specified variable within the scope
of the {\tt ON} directive are to be found locally on the executing processor.
Specification of task-level parallelism in HPF is facilitated by the new
{\tt TASKING} construct. The {\tt ON}, {\tt RESIDENT} and {\tt TASKING}
directives are detailed in Chapter~\ref{ch-parallel-ext}.

\subsubsection*{Extensions to Intrinsic and Library Procedures}

The approved extensions to the HPF intrinsics and library routines
relate mostly to mapping inquiry procedures. Some new inquiry routines
are defined and other routines defined by Base HPF~2.0 are extended to
facilitate inquiry about extended mapping features, such as mapping
mapping to processor subsets, {\tt GEN\_BLOCK}, {\tt INDIRECT} and
{\tt DYNAMIC} distributions. A generalization of the Fortran
{\tt TRANSPOSE} intrinsic is also defined. These extensions are defined
in Chapter~\ref{ch-library-ext}.

\subsubsection*{Extensions for Asynchronous I/O}

In order to permit overlap of I/O with computation, an extension 
has been defined for asynchronous {\tt READ} of direct, unformatted
data. This is done through a new statement ({\tt WAIT}) and an
additional I/O control parameter in the Fortran {\tt READ} statement
that specifies non-blocking execution. This extension is described in
Chapter~\ref{ch-io-ext}.

\subsubsection*{Extensions for Extrinsic Interfaces}

A number of specific extrinsic interfaces are defined in
Chapter~\ref{ch-extrinsic-ext} as approved HPF~2.0 extended features.
These include interfaces with different models of parallelism ({\tt LOCAL})
for SPMD parallel, and {\tt SERIAL} for single-process sequential) and
different languages (FORTRAN and C). A set of library routines are also
defined (for FORTRAN) that permit inquiry about the data visible in the
callee's space and its relation to the global view of data in the calling 
HPF space. Chapter~\ref{ch-fortran77-ext} defines an extrinsic interface for
{\tt LOCAL FORTRAN\_77} routines. Additional extrinsic interfaces are
included in Appendix~\ref{ch-extrinsic-app}, that are formally recognized by
HPFF, but not defined and maintained by HPFF. The policy and mechanism
for formal recognition of such extrinsic interfaces is also set out in
Appendix~\ref{ch-extrinsic-app}.

\section{Notation}
\label{overview-notation}

This document uses the same notation as the Fortran standard.  In
particular, the same conventions are used for syntax rules.  BNF
descriptions of language features are given in the style used in the
Fortran standard.  To distinguish HPF syntax rules from Fortran
rules, each HPF rule has an identifying number of the form H\(snn\),
where \(s\) is a one-digit major section number and \(nn\) is a one- or
two-digit sequence number.  The syntax rules are also collected in
Annex~\ref{SYNTAX-ANNEX}.  Nonterminals not defined in this document
are defined in the Fortran standard.  Also note that certain
technical terms such as ``storage unit'' are defined by the Fortran
standard; Annex~\ref{CREF-ANNEX} identifies the Fortran rules 
defining these nonterminals.  References in parentheses in the text
refer to the Fortran (95) standard.

\begin{rationale}
Throughout this document, material explaining the rationale for including 
features, choosing particular feature definitions, and other decisions is 
set off in this format.  Readers interested in the language definition 
only may wish to skip these sections, while readers interested in 
language design may want to read them more carefully.
\end{rationale}

\begin{users}
Throughout this document, material that is primarily commentary for users
(including most examples of syntax and interpretation) 
is set off in this format.  Readers interested in 
technical material only may wish to skip these sections, while readers 
wanting a more basic approach may want to read them more carefully.
\end{users}

\begin{implementors}
Throughout this document, material that is primarily commentary for 
implementors is set off in this format.  Readers interested in the 
language definition only may wish to skip these sections, while readers 
interested in compiler implementation may want to read them more carefully.
\end{implementors}

%where does this really go???
\section{Syntax of Directives}

HPF directives are consistent with Fortran  syntax in the following
sense: if any HPF directive were to be adopted as part of a future
Fortran standard, the only change necessary to convert an HPF program
would be to replace the directive-origin with blanks.

THE FOLLOWING BNF NEEDS TO BE UPDATED FOR ANY NEW DIRECTIVE CATEGORIES
ALONG WITH STRATEGY FOR EXTENDED VS 2.0 DIRECTIVES

                                                      \BNF
hpf-directive-line     \IS   directive-origin hpf-directive

directive-origin       \IS   !HPF$
                       \OR   CHPF$
                       \OR   *HPF$

hpf-directive          \IS   specification-directive
                       \OR   executable-directive

specification-directive  \IS   processors-directive
                       \OR   align-directive
                       \OR   distribute-directive
                       \OR   dynamic-directive
                       \OR   inherit-directive
                       \OR   template-directive
                       \OR   combined-directive
                       \OR   sequence-directive

executable-directive   \IS   realign-directive
                       \OR   redistribute-directive
                       \OR   independent-directive
                                                      \FNB

\begin{constraints}

\item An {\it hpf-directive-line} cannot be commentary
following another statement on the same line.

\item A {\it specification-directive} may appear only
where a {\it declaration-construct} may appear.

\item An {\it executable-directive} may appear only where an {\it
executable-construct} may appear.

\item An {\it hpf-directive-line} follows the rules of either Fortran
 free form (3.3.1.1) or fixed form (3.3.2.1) comment lines, depending
on the source form of the surrounding Fortran  source form in that
program unit. (3.3)

\end{constraints}

An {\it hpf-directive} is case insensitive and
conforms to the rules for blanks in free
source form (3.3.1), even in an HPF program otherwise in fixed source
form.  However an HPF-conforming processor is not required to diagnose
extra or missing blanks in an HPF directive.
Note that, due to Fortran  rules, the {\it directive-origin}
in free source form must be the characters {\tt !HPF$}.
HPF directives may be continued, in which case
each continued line also begins with a
{\it directive-origin}. No  statements may
be interspersed within a continued HPF-directive.  HPF directive lines
must not appear within a continued statement.
HPF directive lines may include trailing commentary.

In either source form, the blanks in the adjacent keywords
{\tt END FORALL} and {\tt NO SEQUENCE} are optional.

An example of an HPF directive continuation in free source form is:

                                                      \CODE
!HPF$ ALIGN ANTIDISESTABLISHMENTARIANISM(I,J,K) &
!HPF$        WITH ORNITHORHYNCHUS_ANATINUS(J,K,I)
                                                      \EDOC

An example of an HPF directive continuation in fixed source form follows.
Observe that column 6 must be blank, except when signifying continuation.

                                                       \CODE
!HPF$ ALIGN ANTIDISESTABLISHMENTARIANISM(I,J,K)
!HPF$*WITH ORNITHORHYNCHUS_ANATINUS(J,K,I)
                                                       \EDOC

This example shows  an HPF directive continuation which is ``universal'' in
that it can be treated as either fixed source form or free source form.
Note that the ``{\tt \&}'' in the first line is in column 73.

                                                       \CODE
!HPF$ ALIGN ANTIDISESTABLISHMENTARIANISM(I,J,K)                         &
!HPF$&WITH ORNITHORHYNCHUS_ANATINUS(J,K,I)
                                                       \EDOC

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------