Embed resources (eg, shader code; images) into executable/library with CMake

Question

I am writing an application in C++ which relies on various resources in my project. Right now, I have the relative path from the produced executable to each resource hard-coded in my sources, and that allows my program to open the files and read in the data in each resource. This works ok, but it requires that I start the executable from a specific path relative to the resources. So if I try to start my executable from anywhere else, it fails to open the files and cannot proceed.

Is there a portable way to have CMake embed my resources into the executables (or libraries) such that I can simply access them in memory at runtime instead of opening files whose paths are brittle? I have found a related question, and it looks like embedding resources can be done well enough with some ld magic. So my question is how do I do this in a portable, cross platform manner using CMake? I actually need my application run on both x86 and ARM. I am ok with supporting only Linux (Embedded), but bonus points if anyone can suggest how to do this for Windows (Embedded) as well.

EDIT: I forgot to mention a desired property of the solution. I would like to be able to use CMake to cross-compile the application when I am building for ARM rather than have to compile it natively on my ARM target.

Answer 1

Perhaps one non-CMake approach should be provided. Use the program xxd (included with Linux and Mac, may need to download it for Windows) with the -i option on the resource file, and #include the resulting file in your C++ code.

Let's say you just want to get a small SFML project running. Your program is "sfmlgame.cpp", and you want to use the picture "image1.png". Before compiling your code, navigate to the directory with the files in question and enter

xxd -i image1.png > image1.hpp

This will create "image1.hpp" in the same directory, which will contain an array of unsigned char called image1_png, and an unsigned int called image1_png_len. Any files input to xxd will behave analogously -- swapping the dot for an underscore in the original filename -- so that "Monaco.ttf" will produce the identifiers Monaco_ttf and Monaco_ttf_len etc. In your "sfmlgame.cpp", #include "image1.hpp" at the top. The resource is now embedded. Now, instead of writing

texture.loadFromFile("image1.png")

you write

texture.loadFromMemory(image1_png, image1_png_len); 

In SFML you can load images, fonts, and sounds this way.

Answer 2

I'm using this super easy library in my projects: embed

It can be used with C++20 and C++14.

Example of CMakeLists.txt:

cmake_minimum_required(VERSION 3.21)
project(Test)

include(FetchContent)
FetchContent_Declare(
  battery-embed
  GIT_REPOSITORY https://github.com/batterycenter/embed.git
  GIT_TAG        <latest-git-tag>
)
FetchContent_MakeAvailable(battery-embed)

add_executable(Test src/main.cpp)

b_embed(Test resources/message.txt)

And an example of src/main.cpp:

#include <iostream>
#include "battery/embed.hpp"

int main() {
    std::cout << b::embed<"resources/message.txt">() << std::endl;
    return 0;
}

Answer 3

I used this CMake function to embed files into .lib files. It depends on nothing but cmake itself, and the data is re-generated every time the corresponding file is updated.

function(embed_resources target)
    set(script_path "${CMAKE_CURRENT_BINARY_DIR}/anything_to_c.cmake")
    file(WRITE  "${script_path}" "file(READ \${CMAKE_ARGV3} buf HEX)\n")
    file(APPEND "${script_path}" "string(REGEX REPLACE \"([0-9a-f][0-9a-f])\" \"0x\\\\1, \" buf \${buf})\n")
    file(APPEND "${script_path}" "file(WRITE \${CMAKE_ARGV4} \"const unsigned char \${CMAKE_ARGV5}[] = { \${buf}0x00 };\\n\")\n")
    file(APPEND "${script_path}" "file(APPEND \${CMAKE_ARGV4} \"const unsigned \${CMAKE_ARGV6} = sizeof(\${CMAKE_ARGV5}) - 1;\\n\")\n")
    foreach(res_path ${ARGN})
        string(MAKE_C_IDENTIFIER ${res_path} identifier)
        set(src_path "${CMAKE_CURRENT_SOURCE_DIR}/${res_path}")
        set(dst_path "${CMAKE_CURRENT_BINARY_DIR}/${identifier}.c")
        set(anything_to_c ${CMAKE_COMMAND} -P ${script_path} ${src_path} ${dst_path} ${identifier} ${identifier}_size)
        add_custom_command(OUTPUT ${dst_path} COMMAND ${anything_to_c} DEPENDS ${src_path} VERBATIM)
        target_sources(${target} PRIVATE ${src_path} ${dst_path} )
    endforeach()
endfunction()

For example, in CMakeLists.txt, copy and paste the function above and write:

embed_resources(${PROJECT_NAME}
    res/test.vert
    res/test.frag
)

And in C/C++ code:

extern "C" const char res_test_vert[];
extern "C" const unsigned res_test_vert_size;

https://github.com/shir0areed/non-invasive-embed-resources.cmake

Answer 4

This is an improved version of @Itay Grudev's solution by adding null-terminated character to the file. So that it is possible to just use symbol as a null-terminated string directly: extern const char file[] asm("_binary_your_file_start"); cout << strlen(file);

This work nice with text files. For binary files, it is better to stay with the original solution :)

set(RC_DEPENDS "")

# If you want to make the symbol name look prettier, just use relative path as the input
function(add_resource input)
  string(MAKE_C_IDENTIFIER ${input} input_identifier)
  set(res_intermediate_dir ${CMAKE_CURRENT_BINARY_DIR}/resources)
  set(res_with_null_output "${res_intermediate_dir}/${input}")
  set(output "${res_intermediate_dir}/${input_identifier}.o")

  # Add null-terminated character to the file
  add_custom_command(
      DEPENDS ${input}
      OUTPUT ${res_with_null_output}
      COMMAND ${CMAKE_COMMAND} -E copy ${input} ${res_with_null_output};
      COMMAND echo -n '\\0' >> ${res_with_null_output}
      WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
  )

  add_custom_command(
      DEPENDS ${res_with_null_output}
      OUTPUT ${output}
      COMMAND ${CMAKE_LINKER} --relocatable --format binary --output ${output} ${input}
      WORKING_DIRECTORY ${res_intermediate_dir}
  )

  set(RC_DEPENDS ${RC_DEPENDS} ${output} PARENT_SCOPE)
endfunction()

# Resource file list
add_resource( "src/html/index.html" )

add_custom_target( rc ALL DEPENDS ${RC_DEPENDS} )

Answer 5

One of the easiest ways to do this is to include a small, portable C program in your build that reads the resource and generates a C file that contains the length of the resource data and the actual resource data as an array of constant character literals. This will be entirely platform independent, but should only be used for resources that are reasonably small. For larger resources, you probably don't want to embed the files in your program.

For resource "foo", the generated C file "foo.c" would contain:

const char foo[] = { /* bytes of resource foo */ };
const size_t foo_len = sizeof(foo);

To access the resource from C++, you declare the following two symbols in either a header or the cpp file where they're used:

extern "C" const char foo[];
extern "C" const size_t foo_len;

To generate foo.c in the build, you need a target for the C program (call it embedfile.c), and you need to use the add_custom_command command to call this program:

add_executable(embedfile embedfile.c)

add_custom_command(
  OUTPUT foo.c
  COMMAND embedfile foo foo.rsrc
  DEPENDS foo.rsrc)

Then, include foo.c on the source list of a target that requires the "foo" resource. You now have access to the bytes of "foo".

The program embedfile.c is:

#include <stdlib.h>
#include <stdio.h>

FILE* open_or_exit(const char* fname, const char* mode)
{
  FILE* f = fopen(fname, mode);
  if (f == NULL) {
    perror(fname);
    exit(EXIT_FAILURE);
  }
  return f;
}

int main(int argc, char** argv)
{
  if (argc < 3) {
    fprintf(stderr, "USAGE: %s {sym} {rsrc}\n\n"
        "  Creates {sym}.c from the contents of {rsrc}\n",
        argv[0]);
    return EXIT_FAILURE;
  }

  const char* sym = argv[1];
  FILE* in = open_or_exit(argv[2], "r");

  char symfile[256];
  snprintf(symfile, sizeof(symfile), "%s.c", sym);

  FILE* out = open_or_exit(symfile,"w");
  fprintf(out, "#include <stdlib.h>\n");
  fprintf(out, "const char %s[] = {\n", sym);

  unsigned char buf[256];
  size_t nread = 0;
  size_t linecount = 0;
  do {
    nread = fread(buf, 1, sizeof(buf), in);
    size_t i;
    for (i=0; i < nread; i++) {
      fprintf(out, "0x%02x, ", buf[i]);
      if (++linecount == 10) { fprintf(out, "\n"); linecount = 0; }
    }
  } while (nread > 0);
  if (linecount > 0) fprintf(out, "\n");
  fprintf(out, "};\n");
  fprintf(out, "const size_t %s_len = sizeof(%s);\n\n",sym,sym);

  fclose(in);
  fclose(out);

  return EXIT_SUCCESS;
}

Answer 6

There is a single-file CMake script that allows you to embed data easily that's called cmrc.

Example usage:

include(CMakeRC.cmake)
cmrc_add_resource_library(foo-resources
        ALIAS foo::rc
        NAMESPACE foo
        shaders/trig.vert
        shaders/trig.frag)

target_link_libraries(foo foo::rc)

#include <cmrc/cmrc.hpp>

CMRC_DECLARE(foo); // It should be the NAMESPACE property you specified
                   // in your CMakeLists.txt

int main() {
    auto fs = cmrc::foo::get_filesystem();
    auto vert_shader = fs.open("shaders/trig.vert");
    auto frag_shader = fs.open("shaders/trig.frag");
    ...
    glShaderSource(vertexShader, 1, &vert_shader.begin(), nullptr);
}

It's probably the easiest library to setup and use.

Answer 7

As an alternative to the answer of sfstewman, here's a small cmake (2.8) function to convert all files in a specific folder to C data and write them in wished output file:

# Creates C resources file from files in given directory
function(create_resources dir output)
    # Create empty output file
    file(WRITE ${output} "")
    # Collect input files
    file(GLOB bins ${dir}/*)
    # Iterate through input files
    foreach(bin ${bins})
        # Get short filename
        string(REGEX MATCH "([^/]+)$" filename ${bin})
        # Replace filename spaces & extension separator for C compatibility
        string(REGEX REPLACE "\\.| |-" "_" filename ${filename})
        # Read hex data from file
        file(READ ${bin} filedata HEX)
        # Convert hex data for C compatibility
        string(REGEX REPLACE "([0-9a-f][0-9a-f])" "0x\\1," filedata ${filedata})
        # Append data to output file
        file(APPEND ${output} "const unsigned char ${filename}[] = {${filedata}};\nconst unsigned ${filename}_size = sizeof(${filename});\n")
    endforeach()
endfunction()

Answer 8

Pure CMake function to convert any file into C/C++ source code, implemented with only CMake commands:

####################################################################################################
# This function converts any file into C/C++ source code.
# Example:
# - input file: data.dat
# - output file: data.h
# - variable name declared in output file: DATA
# - data length: sizeof(DATA)
# embed_resource("data.dat" "data.h" "DATA")
####################################################################################################

function(embed_resource resource_file_name source_file_name variable_name)

    file(READ ${resource_file_name} hex_content HEX)

    string(REPEAT "[0-9a-f]" 32 column_pattern)
    string(REGEX REPLACE "(${column_pattern})" "\\1\n" content "${hex_content}")

    string(REGEX REPLACE "([0-9a-f][0-9a-f])" "0x\\1, " content "${content}")

    string(REGEX REPLACE ", $" "" content "${content}")

    set(array_definition "static const unsigned char ${variable_name}[] =\n{\n${content}\n};")

    set(source "// Auto generated file.\n${array_definition}\n")

    file(WRITE "${source_file_name}" "${source}")

endfunction()

https://gist.github.com/amir-saniyan/de99cee82fa9d8d615bb69f3f53b6004

Answer 9

I would like to propose another alternative. It uses the GCC linker to directly embed a binary file into the executable, with no intermediary source file. Which in my opinion is simpler and more efficient.

set( RC_DEPENDS "" )

function( add_resource input )
    string( MAKE_C_IDENTIFIER ${input} input_identifier )
    set( output "${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}/${input_identifier}.o" )
    target_link_libraries( ${PROJECT_NAME} ${output} )

    add_custom_command(
        OUTPUT ${output}
        COMMAND ${CMAKE_LINKER} --relocatable --format binary --output ${output} ${input}
        DEPENDS ${input}
    )

    set( RC_DEPENDS ${RC_DEPENDS} ${output} PARENT_SCOPE )
endfunction()

# Resource file list
add_resource( "src/html/index.html" )

add_custom_target( rc ALL DEPENDS ${RC_DEPENDS} )

Then in your C/C++ files all you need is:

extern char index_html_start[] asm( "_binary_src_html_index_html_start" );
extern char index_html_end[]   asm( "_binary_src_html_index_html_end" );
extern size_t index_html_size  asm( "_binary_src_html_index_html_size" );

Answer 10

I'd say the most elegant way to have embedded resources in C++ is simply to use the Qt Resource System which is portable across different platforms, compatible with CMake, and essentially wraps up everything done in the answer above, besides providing compression, being fully tested and fool-proof, everything else.

Create a Qt resource file - an XML listing the files to be embedded:

<RCC>
    <qresource prefix="/">
        <file>uptriangle.png</file>
        <file>downtriangle.png</file>
    </qresource>
</RCC>

Call the file qtres.qrc. The resource file above will have the two png files (located in the same directory as qtres.qrc) embedded in the final executable. You can easily add/remove monitor resources to a qrc file using QtCreator (the Qt IDE).

Now in your CMakeLists.txt file add:

set(CMAKE_AUTOMOC ON)
find_package(Qt5Core)
qt5_add_resources(QT_RESOURCE qtres.qrc)

In your main.cpp, before you need to access the resource, add the following line:

Q_INIT_RESOURCE(qtres);

Now you can access any of the resources above using Qt classes compatible with Qt Resource System, such as QPixmap, QImage ... and mosty importantly maybe in general cases the QResource wrapper class which wraps an embedded Qt resource and enables access to it through a friendly interface. As an example, to access data within downtriangle.png in the above resources, the following lines will do the trick:

#include <QtCore>
#include <QtGui>

// ...

int main(int argc, char **argv)
{

    // ...

    Q_INIT_RESOURCE(qtres);

    // ...
    QResource res("://downtriangle.png"); // Here's your data, anyway you like
    // OR
    QPixmap pm("://downtriangle.png");  // Use it with Qt classes already

    // ...

}

Here, res can be used to directly access the data using res.data(), res.size() ... To parse the image content of the file use pm. Use pm.size(), pm.width() ...

And you're good to go. I hope it helped.