If you're new to Mesos

See the getting started page for more information about downloading, building, and deploying Mesos.

If you'd like to get involved or you're looking for support

See our community page for more details.

Adding a new library or executable

When adding a new library or executable, prefer using the name directly as the target. E.g. libprocess is add_library(process), and mesos-agent is add_executable(mesos-agent). Note that, on platforms where it is conventional, add_library will prepend lib when writing the library to disk.

Do not introduce a variable simply to hold the name of the target; if the name on disk needs to be a specific value, set the target property OUTPUT_NAME.

Adding a third-party dependency

When adding a third-party dependency, keep the principle of locality in mind. All necessary data for building with and linking to the library should be defined where the library is imported. A consumer of the dependency should only have to add target_link_libraries(consumer dependency), with every other build property coming from the graph (library location, include directories, compiler definitions, etc.).

The steps to add a new third-party dependency are:

  1. Add the version and SHA256 hash to Versions.cmake.
  2. Add the URL/tarball file to the top of 3rdparty/CMakeLists.txt.
  3. Find an appropriate location in 3rdparty/CMakeLists.txt to declare the library. Add a nice header with the name, description, and home page.
  4. Use add_library(IMPORTED) to declare an imported target. A header-only library is imported with add_library(INTERFACE).
  5. Use ExternalProject_Add to obtain, configure, and build the library.
  6. Link the consumer to the dependency.

INTERFACE libraries

Using header-only libraries in CMake is a breeze. The special INTERFACE library lets you declare a header-only library as a proper CMake target, and then use it like any other library. Let’s look at Boost for an example.

First, we add two lines to Versions.cmake:

set(BOOST_VERSION "1.53.0")
set(BOOST_HASH    "SHA256=CED7CE2ED8D7D34815AC9DB1D18D28FCD386FFBB3DE6DA45303E1CF193717038")

This lets us keep the versions (and the SHA256 hash of the tarball) of all our third-party dependencies in one location.

Second, we add one line to the top of 3rdparty/CMakeLists.txt to declare the location of the tarball:

set(BOOST_URL ${FETCH_URL}/boost-${BOOST_VERSION}.tar.gz)

The FETCH_URL variable lets the REBUNDLED option switch between offline and online versions. The use of BOOST_VERSION shows why this variable is declared early; it’s used a few times.

Third, we find a location in 3rdparty/CMakeLists.txt to declare the Boost library.

# Boost: C++ Libraries.
# http://www.boost.org
#######################
...

We start with a proper header naming and describing the library, complete with its home page URL. This is for other developers to easily identify why this third-party dependency exists.

...
EXTERNAL(boost ${BOOST_VERSION} ${CMAKE_CURRENT_BINARY_DIR})
add_library(boost INTERFACE)
add_dependencies(boost ${BOOST_TARGET})
target_include_directories(boost INTERFACE ${BOOST_ROOT})
...

Fourth, we declare the Boost target.

To make things easier, we invoke our custom CMake function EXTERNAL to setup some variables for us: BOOST_TARGET, BOOST_ROOT, and BOOST_CMAKE_ROOT. See the docs for more explanation of EXTERNAL.

Then we call add_library(boost INTERFACE). This creates a header-only CMake target, usable like any other library. We use add_dependencies(boost ${BOOST_TARGET}) to add a manual dependency on the ExternalProject_Add step; this is necessary as CMake is lazy and won’t execute code unless it must (say, because of a dependency). The final part of creating this header-only library in our build system is target_include_directories(boost INTERFACE ${BOOST_ROOT}), which sets the BOOST_ROOT folder (the destination of the extracted headers) as the include interface for the boost target. All dependencies on Boost will now automatically include this folder during compilation.

Fifth, we setup the ExternalProject_Add step. This CMake module is incredibly flexible, but we’re using it in the simplest case.

...
ExternalProject_Add(
  ${BOOST_TARGET}
  PREFIX            ${BOOST_CMAKE_ROOT}
  CONFIGURE_COMMAND ${CMAKE_NOOP}
  BUILD_COMMAND     ${CMAKE_NOOP}
  INSTALL_COMMAND   ${CMAKE_NOOP}
  URL               ${BOOST_URL}
  URL_HASH          ${BOOST_HASH})

The name of the custom target this creates is BOOST_TARGET, and the prefix directory for all the subsequent steps is BOOST_CMAKE_ROOT. Because this is a header-only library, and ExternalProject_Add defaults to invoking cmake, we use CMAKE_NOOP to disable the configure, build, and install commands. See the docs for more explanation of CMAKE_NOOP. Thus this code will simply verify the tarball with BOOST_HASH, and then extract it from BOOST_URL to BOOST_ROOT (a sub-folder of BOOST_CMAKE_ROOT).

Sixth, and finally, we link stout to boost. This is the only change necessary to 3rdparty/stout/CMakeLists.txt, as the include directory information is embedded in the CMake graph.

target_link_libraries(
  stout INTERFACE
  ...
  boost
  ...)

This dependency need not be specified again, as libprocess and libmesos link to stout, and so boost is picked up transitively.

Stout

Stout is a header-only library. Like Boost, it is a real CMake target, declared in 3rdparty/stout/CMakeLists.txt, just without the external bits.

add_library(stout INTERFACE)
target_include_directories(stout INTERFACE include)
target_link_libraries(
  stout INTERFACE
  apr
  boost
  curl
  elfio
  glog
  ...)

It is added as an INTERFACE library. Its include directory is specified as an INTERFACE (the PUBLIC property cannot be used as the library itself is just an interface). Its “link” dependencies (despite not being a real, linkable library) are specified as an INTERFACE.

This notion of an interface in the CMake dependency graph is what makes the build system reasonable. The Mesos library and executables, and libprocess, do not have to repeat these lower level dependencies that come from stout.

IMPORTED libraries

Third-party dependencies that we build are only more complicated because we have to encode their build steps too. We’ll examine glog, and go over the differences from the interface library boost.

Notably, when we declare the library, we use:

add_library(glog ${LIBRARY_LINKAGE} IMPORTED GLOBAL)

Instead of INTERFACE we specify IMPORTED as it is an actual library. We add GLOBAL to enable our pre-compiled header module cotire to find the targets (as they would otherwise be scoped only to 3rdparty and below). And most oddly, we use ${LIBRARY_LINKAGE} to set it as SHARED or STATIC based on BUILD_SHARED_LIBS, as we can build this dependency in both manners. See the docs for more information.

We must patch our bundled version of glog so we call:

PATCH_CMD(GLOG_PATCH_CMD glog-${GLOG_VERSION}.patch)

This generates a patch command. See the docs for more information.

This library is an example of where we differ on Windows and other platforms. On Windows, we build glog with CMake, and have several properties we must set:

set_target_properties(
  glog PROPERTIES
  IMPORTED_LOCATION_DEBUG ${GLOG_ROOT}-build/Debug/glog${LIBRARY_SUFFIX}
  IMPORTED_LOCATION_RELEASE ${GLOG_ROOT}-build/Release/glog${LIBRARY_SUFFIX}
  IMPORTED_IMPLIB_DEBUG ${GLOG_ROOT}-build/Debug/glog${CMAKE_IMPORT_LIBRARY_SUFFIX}
  IMPORTED_IMPLIB_RELEASE ${GLOG_ROOT}-build/Release/glog${CMAKE_IMPORT_LIBRARY_SUFFIX}
  INTERFACE_INCLUDE_DIRECTORIES ${GLOG_ROOT}/src/windows
  # TODO(andschwa): Remove this when glog is updated.
  IMPORTED_LINK_INTERFACE_LIBRARIES DbgHelp
  INTERFACE_COMPILE_DEFINITIONS "${GLOG_COMPILE_DEFINITIONS}")

The location of an imported library must be set for the build system to link to it. There is no notion of search through link directories for imported libraries.

Windows requires both the DEBUG and RELEASE locations of the library specified, and since we have (experimental) support to build glog as a shared library on Windows, we also have to declare the IMPLIB location. Fortunately, these locations are programmatic based of GLOG_ROOT, set from our call to EXTERNAL.

Note that we cannot use target_include_directories with an imported target. We have to set INTERFACE_INCLUDE_DIRECTORIES manually instead.

This version of glog on Windows depends on DbgHelp but does not use a #pragma to include it, so we set it as an interface library that must also be linked, using the IMPORTED_LINK_INTERFACE_LIBRARIES property.

For Windows there are multiple compile definitions that must be set when building with the glog headers, these are specified with the INTERFACE_COMPILE_DEFINITIONS property.

For non-Windows platforms, we just set the Autotools commands to configure, make, and install glog. These commands depend on the project requirements. We also set the IMPORTED_LOCATION and INTERFACE_INCLUDE_DIRECTORIES.

set(GLOG_CONFIG_CMD  ${GLOG_ROOT}/src/../configure --with-pic GTEST_CONFIG=no --prefix=${GLOG_ROOT}-build)
set(GLOG_BUILD_CMD   make)
set(GLOG_INSTALL_CMD make install)

set_target_properties(
  glog PROPERTIES
  IMPORTED_LOCATION ${GLOG_ROOT}-build/lib/libglog${LIBRARY_SUFFIX}
  INTERFACE_INCLUDE_DIRECTORIES ${GLOG_ROOT}-build/include)

To work around some issues, we have to call MAKE_INCLUDE_DIR(glog) to create the include directory immediately so as to satisfy CMake’s requirement that it exists (it will be populated by ExternalProject_Add during the build, but must exist first). See the docs for more information.

Then call GET_BYPRODUCTS(glog) to create the GLOG_BYPRODUCTS variable, which is sent to ExternalProject_Add to make the Ninja build generator happy. See the docs for more information.

MAKE_INCLUDE_DIR(glog)
GET_BYPRODUCTS(glog)

Like with Boost, we call ExternalProject_Add:

ExternalProject_Add(
  ${GLOG_TARGET}
  PREFIX            ${GLOG_CMAKE_ROOT}
  BUILD_BYPRODUCTS  ${GLOG_BYPRODUCTS}
  PATCH_COMMAND     ${GLOG_PATCH_CMD}
  CMAKE_ARGS        ${CMAKE_FORWARD_ARGS};-DBUILD_TESTING=OFF
  CONFIGURE_COMMAND ${GLOG_CONFIG_CMD}
  BUILD_COMMAND     ${GLOG_BUILD_CMD}
  INSTALL_COMMAND   ${GLOG_INSTALL_CMD}
  URL               ${GLOG_URL}
  URL_HASH          ${GLOG_HASH})

In contrast to an interface library, we need to send all the build information, which we set in variables prior. This includes the BUILD_BYPRODUCTS, and the PATCH_COMMAND as we have to patch glog.

Since we build glog with CMake on Windows, we have to set CMAKE_ARGS with the CMAKE_FORWARD_ARGS, and particular to glog, we disable its tests with -DBUILD_TESTING=OFF, though this is not a canonical CMake option.

On Linux, we set the config, build, and install commands, and send them too. These are empty on Windows, so ExternalProject_Add will fallback to using CMake, as we needed.

Finally, we add glog to as a link library to stout:

target_link_libraries(
  stout INTERFACE
  ...
  glog
  ...)

No other code is necessary, we have completed adding, building, and linking to glog. The same patterns can be adapted for any other third-party dependency.

Building debug or release configurations

The default configuration is always Debug, which means with debug symbols and without (many) optimizations. Of course, when deploying Mesos an optimized Release build is desired. This is one of the few inconsistencies in CMake, and it’s due to the difference between so-called “single-configuration generators” (such as GNU Make) and “multi-configuration generators” (such as Visual Studio).

Configuration-time configurations

In single-configuration generators, the configuration (debug or release) is chosen at configuration time (that is, when initially calling cmake to configure the build), and it is not changeable without re-configuring. So building a Release configuration on Linux (with GNU Make) is done via:

cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build .

Build-time configurations

However, the Visual Studio generator on Windows allows the developer to change the release at build-time, making it a multi-configuration generator. CMake generates a configuration-agnostic solution (and so CMAKE_BUILD_TYPE is ignored), and the user switches the configuration when building. This can be done with the familiar configuration menu in the Visual Studio IDE, or with CMake via:

cmake ..
cmake --build . --config Release

In the same build folder, a Debug build can also be built, with the binaries stored in Debug and Release folders respectively. Unfortunately, the current CMake build explicitly sets the final binary destination directories, and so the final libraries and executables will overwrite each other when building different configurations.

Note that Visual Studio is not the only IDE that uses a multi-configuration generator, Xcode on Mac OS X does as well. See MESOS-7943 for more information.

Building with shared or static libraries

On Linux, the configuration option -DBUILD_SHARED_LIBS=FALSE can be used to switch to static libraries where possible. Otherwise Linux builds shared libraries by default.

On Windows, static libraries are the default. Building with shared libraries on Windows is not yet supported, as it requires code change to import symbols properly.

© 2012-2022 The Apache Software Foundation. Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.