Installation

This page details, how to install or build all components for SX-Aurora offloading.

Installing the Package

NEC provides LLVM including our OpenMP offloading implementation as package in a YUM repository. The packaged version is build from the hpce/stable branch of the NEC llvm-project repository at the llvm-ve-rv-1.9.0 tag. Thus, the included version of our OpenMP offloading implementation will not be at the latest state of development. However, further functionality and SX-Aurora specific fixes and extensions are included.

Install from RPM

Download llvm-ve-rv-1.9.0-1.9.0-1.el8.x86_64.rpm

wget https://sx-aurora.com/repos/llvm/x86_64/llvm-ve-rv-1.9.0-1.9.0-1.el8.x86_64.rpm

Install RPM

sudo yum install ./llvm-ve-rv-1.9.0-1.9.0-1.el8.x86_64.rpm

Install from YUM Repository

Add the https://sx-aurora.com/repos/llvm/x86_64/ yum repository to your /etc/yum.repo.d (consult How to add a Yum repository by Red Hat)
Install llvm-ve-rv-1.9.0
```
sudo yum install llvm-ve-rv-1.9.0
```

Release Notes

LLVM for NEC SX-Aurora VE (llvm-ve-rv 1.8-dev)

This is a fork of the LLVM repositoy with support for the NEC SX-Aurora TSUBASA Vector Engine (VE).

Features

C, C++ support.
VEL intrinsics for low-level vector programming.
Automatic vectorization through LLVM's loop and SLP vectorizers.
When combined with RV for SX-Aurora, provides user-guided (and some automatic) outer-loop vectorization through the Region Vectorizer.
Two OpenMP offloading modes: VE to VH and VH to VE.

Build instructions

To build llvm-ve from source refer to llvm-dev and Compile.rst.

General Usage

To compile C/C++ code for the VE run Clang/Clang++ with the following command line:

$ clang -target ve-linux -O3 ...

OpenMP offloading for/from SX-Aurora

To compile with OpenMP offloading from VE to VH (VHCall) use:

$ clang -target ve-linux -fopenmp -fopenmp-targets=x86_64-pc-linux -O3 ...

To compile with OpenMP offloading from VH to VE (VEO) use:

$ clang -march=native -fopenmp -fopenmp-targets=ve-linux -O3 ...

Outer-loop Vectorization

LLVM for SX-Aurora provides outer-loop vectorization, provided it is build with the Region Vectorizer. The following usage examples require an RV-enabled build.

To use user-guided outer-loop vectorization with RV annotate the loops to vectorize with #pragma omp simd and use:

$ clang -fopenmp-simd -mllvm -rv -O3 ...

This release comes with a preview feature for automatic outer-loop vectorization with RV. This will work for some loops that use int64_t for their iteration variables (loop counters). To enable automatic outer-loop vectorization with RV use:

$ clang -mllvm -rv -mllvm -rv-autovec -O3 ...

VEL Intrinsics for direct vector programming

See the manual. To use VEL intrinsics, pass the compiler option -mattr=+packed. The resulting LLVM bitcode and objects are compatible with those compiler without this option.

Clang Experimental Options

To enable packed mode support, call Clang with -mve-packed. This sets the machine attribute +packed.

LLVM Experimental Options

Clang and llc accept these flags directly, prefix them with -mllvm to use them with Clang.

-ve-regalloc=0 disable the experimental improvements to the vector register allocator.

Building the Project

This project now comes as a single llvm-project repository, forked from LLVM's upstream repository. All components can be checked out from a single source and built, closely following the Clang build instructions.

To build the project, first check out the llvm-project repository and switch to the aurora-offloading branch before configuring the project with CMake.

Configure with CMake

After checking out the aurora-offloading branch, change the working directory to llvm-project/llvm and run CMake with, at least, the following CMake variables:

Set LLVM_ENABLE_PROJECTS to clang;openmp;libcxx;libcxxabi to enable at least clang, openmp, libcxx and libcxxabi.
Set OPENMP_LIBOMPTARGET to ON to enable libomptarget

Additionally, make sure that:

LIBOMPTARGET_DEP_VEO_LIBRARIES Points to an installation of libveo.so (usually found under /opt/nec/ve/veos/lib64/)
LIBOMPTARGET_DEP_VEOSINFO_LIBRARIES Points to an installation of libveosinfo.so (usually found under /opt/nec/ve/veos/lib64/)
NECAURORA_TARGET_COMPILER_NCC points to your installation of ncc (usually found under /opt/nec/ve/bin).

Static Linking

To be able to statically link the target image, please make sure that, additionally, the following CMake variables are set

NECAURORA_LIBAVEOVE_STATIC, pointing to libaveoVE.a
NECAURORA_LIBURPCVE_STATIC, pointing to liburpcVE_omp.a
NECAURORA_LIBVEIO, pointing to libveio.so

Usually these libraries are installed at /opt/nec/ve/lib.

Enable Testing

To enable the test suite for the source transformation, simply set SOTOC_ENABLE_TESTS to ON.

The test suite uses the llvm-lit and FileCheck tool build together with the project and uses the project's Clang as test compiler.

CMake Options

NECAURORA_LIBELF_INCLUDE_DIR Points to the include directory containing the headers for libelf.so (required for static linking).
NECAURORA_LIBELF_LIBRARIES Points to a libelf.so (required for static linking).
NECAURORA_LIBAVEOVE_STATIC Points to libaveoVE.a (required for static linking).
NECAURORA_LIBURPCVE_STATIC Points to liburpcVE_omp.a (required for static linking).
NECAURORA_LIBVEIO Points to libveio.so (required for static linking).
NECAURORA_TARGET_COMPILER_CLANG The path to the Clang compiler used for target compilations with the option -fopenmp-nec-compiler=clang.
NECAURORA_TARGET_COMPILER_RVCLANG The path to the target compiler used for target compilations with the option -fopenmp-nec-compiler=rclang preset.
NECAURORA_TARGET_COMPILER_NCC The path to the target compiler used for target compilations with the option -fopenmp-nec-compiler=ncc.
NECAURORA_DEFAULT_TARGET_OPTION The default option for -fopenmp-nec-compiler= (defaults to ncc).
SOTOC_DEBUG_OUTPUT Enables debug output for sotoc (when the env-var SOTOC_DEBUG=1 is set).
SOTOC_ENABLE_TESTS Enables the test suite for sotoc (can be called with make check-sotoc).
SOTOC_LLVM_LIT_EXECUTABLE The llvm-lit executable required for testing.
LIBOMPTARGET_DEP_LIBFFI_INCLUDE_DIR The include-directory for a libffi installation (required for Run-On-Host).
LIBOMPTARGET_DEP_LIBFFI_LIBRARIES The path to a libffi.so (required for Run-On-Host).
LIBOMPTARGET_DEP_VEO_INCLUDE_DIR The include-directory for ve_offload.h.
LIBOMPTARGET_DEP_VEO_LIBRARIES Path to libveo.so/libaveo.so.

Building the NEC packaged version

To build the packaged version by NEC (e.g. if debug information is needed) some Makefiles are provided. To get those clone the llvm-dev repository.

git clone https://github.com/sx-aurora-dev/llvm-dev.git -b hpce/develop

The actual LLVM repository and all required submodules can be cloned by doing:

LLVM_BUILD_TYPE=Debug BUILD_TYPE=Debug BRANCH=hpce/develop REPOS=https://github.com/sx-aurora-dev make clone

For the build process cmake, ninja and clang are required.

module purge && module load DEVELOP cmake ninja-build clang

Afterwards LLVM can be build by issuing the following command:

LLVM_BUILD_TYPE=Debug BUILD_TYPE=Debug BRANCH=hpce/develop REPOS=https://github.com/sx-aurora-dev make install

Last update: 2021-11-24