Skip to content

Commit edc9be6

Browse files
cabreraamrovka
cabreraam
authored and
rovka
committed
[Fortran] adding a new test now that CMake supports LLVMFlang
As of [this CMake merge request](https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7246), there is now CMake support for LLVMFlang. This allows for the integration of LLVMFlang into the `llvm-test-suite`. To this end, this request for review takes advantage of this new CMake functionality by adding a proxy application called `SNAP`, which aims "[to model the performance of a modern discrete ordinates neutral particle transport application](https://github.com/cabreraam/SNAP/#description)". This application was picked because of its importance to Flang developers, and because there were folks who were already using it to test Flang's correctness in producing end-to-end binaries. A recipe for building the `llvm-test-suite` is listed below: ``` LLVM_TS_BUILD_DIR="/path/to/where/you/want/llvm-test-suite/to/be/built" FLANG_PATH="/path/to/your/build/of/flang" FC="${FLANG_PATH}/bin/flang-new" CMAKE_FORTRAN_FLAGS="-flang-experimental-exec" CMAKE_OPTIONS="-GNinja \ -DCMAKE_C_COMPILER=clang \ -DCMAKE_CXX_COMPILER=clang++ \ -DCMAKE_Fortran_COMPILER=${FC} \ -DTEST_SUITE_FORTRAN=on \ -DTEST_SUITE_SUBDIRS=Fortran" mkdir -p ${LLVM_TS_BUILD_DIR} cd ${LLVM_TS_BUILD_DIR} cmake ${CMAKE_OPTIONS} -DCMAKE_Fortran_FLAGS="${CMAKE_FORTRAN_FLAGS}" .. cmake --build . -j64 ``` > __NOTE__: You can save yourself some build time by just specifying `-DTEST_SUITE_SUBDIRS=Fortran` so you don't have to build the entire `llvm-test-suite`, but this option can be omitted if you want to build the whole thing. The major components of this review request are: + adding the necessary source files from the `SNAP` project repository + `CMakeLists.txt` for the `SNAP` proxy application I have taken the [SNAP project repo](https://github.com/cabreraam/SNAP/) and placed it in `llvm-test-suite/Fortran`. From the [SNAP project repo](https://github.com/cabreraam/SNAP/), I have taken the files in [src](https://github.com/lanl/SNAP/tree/main/src) and added them to the `llvm-test-suite`. I have also explicitly added a `LICENSE.md` file. Based on the license, we should be able to include the source in `llvm-test-suite`. I have removed some stuff files from the original `SNAP` repo that really shouldn't have been added -- thank you @rovka for the feedback! -- but there are some other files that I have included from the `SNAP` repo that I have kept in the revision (but ultimately probably shouldn't be there): * `docs` * `qasnap` (the input test files that aren't `qasnap/mms_src/2d_mms_st.inp`) * really, there's only one input that we test, which is listed above. It would be possible to test other inputs to act as additional regression tests, but I would agree with the sentiment that the other inputs can be removed and then subsequently added if folks wanted to try more inputs. I've removed already modified some of the CMake stuff that @rovka pointed out wasn't really necessary, but here are the salient bits of the CMake build infrastructure that I've added. There are two files: `Fortran/SNAP/CMakeLists.txt` and `Fortran/SNAP/src/CMakeLists.txt`. There are methods that allow you to build targets that consist of multiple source files. In general, though, using this call does not give enough fine-grained control over the target and how it is compiled/linked. Additionally, this question breaks down into: * Why doesn't `llvm_test_executable_no_test` work? * this CMake function only mutates the global `CFLAGS`, `CPPFLAGS`, and `CXXFLAGS` variables, which doesn't apply to `Fortran`. Not using `llvm_test_executable_no_test` frees me up to use functions like `target_compile_options, `target_link_libraries`, and the like. * Why doesn't `llvm_add_tests_for_target` work? * this function assumes that the name of the target is also the base name for the input file, which is not the case for `SNAP`. Since this assumption doesn't hold, I have to roll my own CMake for this bit, though admittedly there is a lot of overlap since the name is the only issue. For future applications that require more fine-grained control of parameterizing the target, it might make sense to create a `.cmake` file geared for Fortran based on the work I've done here. But I'll call that future work and outside the scope of this PR. The `RUN:` and `VERIFY:` commands that eventually get run are, in part, based on work done by @awarzynski and others. Specifically, I am running the application with the input that they have verified and validated to be correct. As noted above, I am using the `SNAP/qasnap/mms_src/2d_mms_st.inp` file. Currently, using LLVMFlang means that you need to externally link `libpgmath` so that those symbols are defined. Differential Revision: https://reviews.llvm.org/D130734
1 parent 9075928 commit edc9be6

35 files changed

+10328
-0
lines changed

‎Fortran/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@
77
# This file should only contain add_subdirectory(...) one for each test
88
# directory
99
add_subdirectory(UnitTests)
10+
add_subdirectory(SNAP)

‎Fortran/SNAP/CMakeLists.txt

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
################################################################################
2+
#
3+
# CMakeLists.txt
4+
#
5+
# Author: Anthony Cabrera:
6+
# Contact: cabreraam AT ieee DOT org
7+
# Description: CMakeLists.txt replacement for SNAP Makefile
8+
#
9+
################################################################################
10+
11+
cmake_minimum_required(VERSION 3.18)
12+
13+
add_subdirectory(src)

‎Fortran/SNAP/LICENSE.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# License
2+
Los Alamos National Security, LLC owns the copyright to "SNAP: SN (Discrete
3+
Ordinates) Application Proxy, Version 1.x (C13087)". The license is BSD with
4+
standard clauses regarding indicating modifications before future
5+
redistribution:
6+
7+
Copyright (c) 2013, Los Alamos National Security, LLC All rights reserved.
8+
9+
Copyright 2013. Los Alamos National Security, LLC. This software was produced
10+
under U.S. Government contract DE-AC52-06NA25396 for Los Alamos National
11+
Laboratory (LANL), which is operated by Los Alamos National Security, LLC for
12+
the U.S. Department of Energy. The U.S. Government has rights to use,
13+
reproduce, and distribute this software. NEITHER THE GOVERNMENT NOR LOS
14+
ALAMOS NATIONAL SECURITY, LLC MAKES ANY WARRANTY, EXPRESS OR IMPLIED, OR
15+
ASSUMES ANY LIABILITY FOR THE USE OF THIS SOFTWARE. If software is modified
16+
to produce derivative works, such modified software should be clearly marked,
17+
so as not to confuse it with the version available from LANL.
18+
19+
Additionally, redistribution and use in source and binary forms, with or
20+
without modification, are permitted provided that the following conditions
21+
are met:
22+
23+
+ Redistributions of source code must retain the above copyright notice, this
24+
list of conditions and the following disclaimer.
25+
+ Redistributions in binary form must reproduce the above copyright notice,
26+
this list of conditions and the following disclaimer in the documentation
27+
and/or other materials provided with the distribution.
28+
+ Neither the name of Los Alamos National Security, LLC, Los Alamos National
29+
Laboratory, LANL, the U.S. Government, nor the names of its contributors may
30+
be used to endorse or promote products derived from this software without
31+
specific prior written permission.
32+
33+
THIS SOFTWARE IS PROVIDED BY LOS ALAMOS NATIONAL SECURITY, LLC AND
34+
CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
35+
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
36+
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL LOS ALAMOS NATIONAL
37+
SECURITY, LLC OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
38+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
39+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
40+
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
41+
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
42+
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
43+
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎Fortran/SNAP/README.md

Lines changed: 418 additions & 0 deletions
Large diffs are not rendered by default.

‎Fortran/SNAP/qasnap/mms_src/2d_mms_st.inp

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
! Input from namelist
2+
&invar
3+
nthreads=4
4+
npey=1
5+
ndimen=2
6+
nx=20
7+
lx=0.2
8+
ny=20
9+
ly=0.2
10+
nmom=4
11+
nang=12
12+
ng=10
13+
mat_opt=0
14+
src_opt=3
15+
timedep=0
16+
it_det=1
17+
tf=1.0
18+
nsteps=10
19+
iitm=5
20+
oitm=100
21+
epsi=1.E-4
22+
fluxp=2
23+
scatp=1
24+
fixup=0
25+
angcpy=2
26+
/

‎Fortran/SNAP/src/CMakeLists.txt

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
################################################################################
2+
#
3+
# CMakeLists.txt
4+
#
5+
# Author: Anthony Cabrera:
6+
# Contact: cabreraam AT ieee DOT org
7+
# Description: CMakeLists.txt replacement for SNAP Makefile
8+
#
9+
################################################################################
10+
11+
################################################################################
12+
#
13+
# Options and Variables
14+
#
15+
################################################################################
16+
17+
set(SNAP_USER_DEFS "" CACHE STRING "Semi-colon separted preprocessor pound defines")
18+
19+
set(FP_TOLERANCE "0.001" CACHE STRING "FP tolerance value")
20+
21+
set(SNAP_COMPILER_OPTIONS_LLVM_FLANG
22+
""
23+
CACHE
24+
STRING
25+
"Compiler options for LLVMFlang"
26+
)
27+
28+
################################################################################
29+
#
30+
# Setting up build target
31+
#
32+
################################################################################
33+
34+
set(SNAP_SOURCE_FILES
35+
global.f90
36+
snap_main.f90
37+
utils.f90
38+
version.f90
39+
plib.F90
40+
geom.f90
41+
sn.f90
42+
data.f90
43+
control.f90
44+
input.f90
45+
setup.f90
46+
dealloc.f90
47+
translv.f90
48+
solvar.f90
49+
outer.f90
50+
expxs.f90
51+
inner.f90
52+
sweep.f90
53+
octsweep.f90
54+
dim1_sweep.f90
55+
dim3_sweep.f90
56+
output.f90
57+
time.F90
58+
mms.f90
59+
analyze.f90
60+
thrd_comm.f90
61+
mkba_sweep.f90
62+
CACHE
63+
STRING
64+
"List of SNAP source files"
65+
)
66+
67+
add_executable(snap
68+
${SNAP_SOURCE_FILES}
69+
)
70+
71+
target_compile_options(snap
72+
PRIVATE
73+
$<$<Fortran_COMPILER_ID:LLVMFlang>:${SNAP_COMPILER_OPTIONS_LLVM_FLANG}>
74+
)
75+
target_compile_definitions(snap
76+
PRIVATE
77+
${SNAP_USER_DEFS}
78+
)
79+
80+
macro(llvm_test_verify_hash_program_output _file)
81+
llvm_test_verify(%b/HashProgramOutput.sh ${_file})
82+
endmacro()
83+
84+
# This if-body takes care of all of the llvm-test suite stuff. It is modeled
85+
# after the llvm_multisource function in the llvm-test-suite, which takes the
86+
# following form:
87+
#function(llvm_multisource target)
88+
# set(sources ${ARGN})
89+
# if(NOT sources)
90+
# file(GLOB sources *.c *.cpp *.cc *.f *.F *.f90 *.F90)
91+
# endif()
92+
# llvm_test_executable_no_test(${target} ${sources})
93+
# llvm_test_traditional(${target})
94+
# llvm_add_test_for_target(${target})
95+
#endfunction()
96+
# Because
97+
# - I could not directly use those pre-existing functions/macros,
98+
# - I wanted to reference the work I'd already done with the SNAP CMake infra
99+
# I will show how my implementation below corresponds to llvm_multisource. Look
100+
# for comments that take the form # start <function>
101+
if(PROJECT_NAME STREQUAL "test-suite")
102+
set(target "snap")
103+
104+
# start llvm_test_executable_no_test_fortran
105+
# Note: we ignore TEST_SUITE_PROFILE_USE. Will this break LLVM Flang? if we
106+
# aded that flag?
107+
llvm_codesign(snap)
108+
set_property(GLOBAL APPEND PROPERTY TEST_SUITE_TARGETS snap)
109+
test_suite_add_build_dependencies(snap)
110+
111+
if(TEST_SUITE_LLVM_SIZE)
112+
add_custom_command(TARGET ${target} POST_BUILD
113+
COMMAND ${TEST_SUITE_LLVM_SIZE} --format=sysv $<TARGET_FILE:${target}>
114+
> $<TARGET_FILE:${target}>.size)
115+
endif()
116+
117+
# start llvm_test_traditional(${target})
118+
# isolate path for the input file that we use
119+
set(TEST_INPUT_DIR
120+
"${CMAKE_CURRENT_SOURCE_DIR}/../qasnap/mms_src"
121+
)
122+
message(STATUS "TEST_INPUT_DIR - ${TEST_INPUT_DIR}")
123+
set(TEST_INPUT_FILENAME "2d_mms_st.inp")
124+
set(TEST_OUTPUT_FILENAME "snap-output")
125+
llvm_test_data(${target}
126+
MUST_COPY
127+
SOURCE_DIR
128+
${TEST_INPUT_DIR}
129+
${TEST_INPUT_FILENAME}
130+
)
131+
# make run command include the new input we just copied
132+
list(APPEND RUN_OPTIONS
133+
${TEST_INPUT_FILENAME}
134+
${TEST_OUTPUT_FILENAME}
135+
)
136+
list(INSERT RUN_OPTIONS 0 WORKDIR %S)
137+
# here, we take the options for the RUN: command that we want to craft, and
138+
# pass it to the llvm_test_run function
139+
llvm_test_run(${RUN_OPTIONS})
140+
141+
# Hash if we've been asked to.
142+
if(HASH_PROGRAM_OUTPUT)
143+
llvm_test_verify_hash_program_output(%o)
144+
endif()
145+
146+
# Deal with output that we want to compare to
147+
if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/snap.reference_output)
148+
set(REFERENCE_OUTPUT
149+
snap.reference_output
150+
)
151+
set(DIFFPROG_BASE %b/${FPCMP})
152+
set(FP_TOL_SUFFIX "-r ${FP_TOLERANCE}")
153+
set(FP_ABSTOL_SUFFIX "-a ${FP_ABSTOLERANCE}")
154+
set(DIFFPROG
155+
${DIFFPROG_BASE}
156+
"$<$<BOOL:${FP_TOLERANCE}>:${FP_TOL_SUFFIX}>"
157+
"$<$<BOOL:${FP_ABSTOLERANCE}>:${FP_ABSTOL_SUFFIX}>"
158+
)
159+
160+
llvm_test_verify(${DIFFPROG} %o %S/${REFERENCE_OUTPUT})
161+
llvm_test_data(${target} ${REFERENCE_OUTPUT})
162+
else()
163+
message("-- No reference output found for test ${name}")
164+
endif()
165+
set(TESTSCRIPT "${TESTSCRIPT}" PARENT_SCOPE)
166+
167+
# this creates the .test file for you!
168+
# start llvm_add_test_for_target(${target})
169+
# don't need to make any mods to this, woo!
170+
llvm_add_test_for_target(snap)
171+
172+
# lit.local.cfg has since been added to the src directory. Additionally,
173+
# we'll need to copy that file over to the binary directory.
174+
file(COPY lit.local.cfg DESTINATION "${CMAKE_CURRENT_BINARY_DIR}")
175+
endif()
176+

‎Fortran/SNAP/src/analyze.f90

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
!-----------------------------------------------------------------------
2+
!
3+
! MODULE: analyze_module
4+
!> @brief
5+
!> This module contains subroutines for analyzing data.
6+
!
7+
!-----------------------------------------------------------------------
8+
9+
MODULE analyze_module
10+
11+
USE global_module, ONLY: i_knd, r_knd, ounit
12+
13+
USE control_module, ONLY: timedep
14+
15+
USE geom_module, ONLY: dx, dy, dz, nx, ny, nz, ndimen
16+
17+
USE data_module, ONLY: v, ng
18+
19+
USE solvar_module, ONLY: flux0, pop
20+
21+
USE plib_module, ONLY: iproc, root, comm_snap, rtsum
22+
23+
IMPLICIT NONE
24+
25+
PUBLIC
26+
27+
SAVE
28+
29+
30+
CONTAINS
31+
32+
33+
SUBROUTINE analyze_pop_calc ( cy, time )
34+
35+
!-----------------------------------------------------------------------
36+
!
37+
! Compute the neutron spectrum: the population per group integrated over
38+
! volume.
39+
!
40+
!-----------------------------------------------------------------------
41+
42+
INTEGER(i_knd), INTENT(IN) :: cy
43+
44+
REAL(r_knd), INTENT(IN) :: time
45+
!_______________________________________________________________________
46+
!
47+
! Local variables
48+
!_______________________________________________________________________
49+
50+
INTEGER(i_knd) :: g
51+
52+
REAL(r_knd) :: vol
53+
54+
REAL(r_knd), DIMENSION(nx,ny,nz,ng) :: tmp
55+
!_______________________________________________________________________
56+
!
57+
! Integrate flux over volume of local spatial sub-domain
58+
!_______________________________________________________________________
59+
60+
vol = dx
61+
IF ( ndimen > 1 ) THEN
62+
vol = vol*dy
63+
IF ( ndimen > 2 ) vol = vol*dz
64+
END IF
65+
tmp = vol*flux0
66+
67+
DO g = 1, ng
68+
pop(g) = SUM( tmp(:,:,:,g) )
69+
END DO
70+
!_______________________________________________________________________
71+
!
72+
! Integrate over global volume with MPI reduction (to root only, not
73+
! all_reduce)
74+
!_______________________________________________________________________
75+
76+
CALL rtsum ( pop, ng, comm_snap, root )
77+
!_______________________________________________________________________
78+
!
79+
! Divide volume integrated flux by velocity to get particle density
80+
! per unit energy. Print out results. Note, only root process has
81+
! correct value because the reduction is not global. Can easily
82+
! change this though.
83+
!_______________________________________________________________________
84+
85+
IF ( iproc == root ) THEN
86+
87+
IF ( timedep == 1 ) THEN
88+
pop = pop/v
89+
ELSE
90+
WRITE( ounit, 352 )
91+
END IF
92+
93+
WRITE( ounit, 350 ) cy, time
94+
DO g = 1, ng
95+
WRITE( ounit, 351 ) g, pop(g)
96+
END DO
97+
WRITE( ounit, * )
98+
99+
END IF
100+
!_______________________________________________________________________
101+
102+
350 FORMAT( 2X, 'Particle spectrum, cycle (time)', I4, ' ( ', &
103+
ES11.4, ' )' )
104+
351 FORMAT( 8X, I4, ES13.6 )
105+
352 FORMAT( 2X, 'Static calc: v(g) not set. Use dummy value of 1.' )
106+
!_______________________________________________________________________
107+
!_______________________________________________________________________
108+
109+
END SUBROUTINE analyze_pop_calc
110+
111+
112+
END MODULE analyze_module

0 commit comments

Comments
 (0)