question on interaction of compiler options

[grburgess]

Summary:

I'm curious how the build of Cmdstan with a make/local file and the compiling of a model with supplied compiler args interact with one another

Description:

I'm constantly doing a lot of profiling on models where I'm switching at the CXXFlags/ STAN options when building Cmdstan. Since one can specify those flags in the make/local as well as when building a model via cmdstanpy, I was curious how these things interact. E.g., If I build Cmdstan:

make build

and my make/local has

STAN_OPENCL=True

Does that opencl option when building the model model override what is in make/local? So, if I set to false in cmdstanpy, it will be overridden, but if I do not set it, then it will always be tru?

Additional Information:

Provide any additional information here.

Current Version:

1.0.0

[ahartikainen]

Good question. ( I don't have an answer to this)

which one is preferred a cmdline argument or the make/local file.

(Question for anyone needing to debug this, is there a way to print in somewhere while building is going on)

[grburgess]

I have noticed that if I turn on OpenCL in the make/local, but say nothing in the cmdline, I get the following output during the build:

WARNING:cmdstanpy:g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes      -DSTAN_NO_RANGE_CHECKS  -c -MT stan/src/stan/model/model_header_threads_opencl_nochecks.hpp.gch -MT stan/src/stan/model/model_header_threads_opencl_nochecks.d -MM -E -MG -MP -MF stan/src/stan/model/model_header_threads_opencl_nochecks.d stan/src/stan/model/model_header.hpp

--- Translating Stan model to C++ code ---
bin/stanc --include_paths=/home/jburgess/sw/whimstan/whimstan/stan_code --o=/home/jburgess/sw/whimstan/whimstan/stan_code/whim.hpp /home/jburgess/sw/whimstan/whimstan/stan_code/whim.stan
--include_paths is deprecated and Will be removed in Stan 2.32.0. Please use --include-paths.
Warning in '/home/jburgess/sw/whimstan/whimstan/stan_code/whim.stan', line 43, column 2: Declaration
    of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc

--- Compiling, linking C++ code ---
g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes      -DSTAN_NO_RANGE_CHECKS  -c -Wno-ignored-attributes -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -x c++ -o /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o /home/jburgess/sw/whimstan/whimstan/stan_code/whim.hpp
g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes      -DSTAN_NO_RANGE_CHECKS        -Wl,-L,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb"   -flto -fuse-linker-plugin -fdevirtualize-at-ltrans   /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o src/cmdstan/main_threads_opencl_nochecks.o -lblas -llapack -llapacke      -lOpenCL  -Wl,-L,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb"   stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a  stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /home/jburgess/sw/whimstan/whimstan/stan_code/whim
rm -f /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o

I do see the OpenCL stuff popping up.

[WardBrian]

CmdStanPy does do something special related to opencl, which is if you pass the use-opencl flag to stanc, it will also turn on STAN_OPENCL in the C++ options.

There are two things I'm not sure about -

1. How command line arguments and make/local compose
2. What STAN_OPENCL does if the model is not compiled with use-opencl in stanc.

@rok-cesnovar - do you have a better answer for either of those?

[mitzimorris]

command line arguments to make override make/local argument.
https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_9.html#SEC90

here's what's in the CmdStan docs w/r/t OpenCL - @rok-cesnovar are these correct? https://mc-stan.org/docs/cmdstan-guide/parallelization.html#opencl

[rok-cesnovar]

What STAN_OPENCL does if the model is not compiled with use-opencl in stanc.

Nothing really as none of the OpenCL-related stuff will be used, so this just adds a bit more code in C++ so a bit longer compile times. You need the use-opencl flag in stanc3 in order to generate the model with the OpenCL-related constructs (matrix_cl class, etc) and setting STAN_OPENCL then enables the OpenCL code in Stan Math so that the generated C++ actually compiles.

To sum up:

• no use-opencl stanc3 flag & STAN_OPENCL=False -> normal sequential case
• no use-opencl stanc3 flag & STAN_OPENCL=True -> same as sequential code, a bit longer compile times excepted
• use-opencl stanc3 flag & STAN_OPENCL= False -> if any OpenCL code was generated in stanc3, the C++ will not compile
• use-opencl stanc3 flag & STAN_OPENCL= True -> OpenCL model

are these correct?

They are.

@grburgess if you are ever in doubt about what gets passed to the compiler, I suggest using ˙make print-variable` command. Its precisely used for debugging these kind of stuff.

I set STAN_OPENCL=false in make/local and run:

rok@rok-Z97X-UD5H:~/Desktop/cmdstan$ make print-STAN_OPENCL
STAN_OPENCL = false
rok@rok-Z97X-UD5H:~/Desktop/cmdstan$ make STAN_OPENCL=true print-STAN_OPENCL
STAN_OPENCL = true

[rok-cesnovar]

And make is its own beast, I never assume I know anything without triple-checking and even then I don't trust it haha

[grburgess]

Awesome. I think I understand the workflow now. Thanks!