How to configure Conan to not link in everything possible?

Question

I'm trying out the Conan package manager, and started writing a test C++ project that uses the Poco libraries. I produced a simple program that just decrypts a string using AES-256-CBC. I was extremely surprised to find the generated binary was almost 4 MB after building with Conan and Cmake. I tried tweaking the conanfile.txt and CmakeLists.txt files to only link the necessary libraries, but I either couldn't get the project to compile, or couldn't reduce the size of the compiled binary.

I'm pretty sure that PCRE, bzip2, SQLlite and more are getting linked in to my binary, as Poco depends on them. I'm just very confused as to why gcc isn't smart enough to figure out that the Poco code I'm calling is only using a small bit of OpenSSL code.

How can I only compile in/link what I need to, and keep my binary to a reasonable size?

conanfile.txt:

[requires]
poco/1.10.1

[generators]
cmake

CmakeLists.txt:

cmake_minimum_required(VERSION 3.7...3.18)

if(${CMAKE_VERSION} VERSION_LESS 3.12)
    cmake_policy(VERSION ${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION})
endif()

project(main)

add_definitions("-std=c++17")

include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()

add_executable(${PROJECT_NAME} main.cpp)
target_link_libraries(${PROJECT_NAME} ${CONAN_LIBS})

main.cpp:

#include <cstdlib>
#include <iostream>
#include <sstream>

#include "Poco/Base64Decoder.h"
#include "Poco/Crypto/Cipher.h"
#include "Poco/Crypto/CipherFactory.h"
#include "Poco/Crypto/CipherKey.h"
#include "Poco/DigestStream.h"
#include "Poco/SHA2Engine.h"


std::string sha512(std::string value);
std::string aesDecrypt(const std::string ciphertext, const std::string key, const std::string iv);
std::string getEnvVar(const std::string key);
std::string base64Decode(const std::string encoded);

int main(int argc, char** argv) {
    std::string enc = "Ug7R5BQIosmn1yPeawSUIzY8N9wzASmI/w0Wz/xX7Yw=";
    std::cout << aesDecrypt(enc, "admin", "7K/OkQIrl4rqUk8/1h+uuQ==") << "\n";

    std::cout << sha512("Hello there") << "\n";
    std::cout << getEnvVar("USER") << "\n";

    return 0;
}

std::string aesDecrypt(const std::string ciphertext, const std::string key, const std::string iv) {
    auto keyHash = sha512(key);
    Poco::Crypto::Cipher::ByteVec keyBytes{keyHash.begin(), keyHash.end()};
    auto rawIV = base64Decode(iv);
    Poco::Crypto::Cipher::ByteVec ivBytes{rawIV.begin(), rawIV.end()};

    auto &factory = Poco::Crypto::CipherFactory::defaultFactory();
    auto pCipher = factory.createCipher(Poco::Crypto::CipherKey("aes-256-cbc", keyBytes, ivBytes));

    return pCipher->decryptString(ciphertext, Poco::Crypto::Cipher::ENC_BASE64);
}

std::string sha512(const std::string value) {
    Poco::SHA2Engine sha256(Poco::SHA2Engine::SHA_512);
    Poco::DigestOutputStream ds(sha256);
    ds << value;
    ds.close();

    return Poco::DigestEngine::digestToHex(sha256.digest());
}

std::string getEnvVar(const std::string key) {
    char * val = getenv(key.c_str());
    return val == NULL ? std::string("") : std::string(val);
}

std::string base64Decode(const std::string encoded) {
    std::istringstream istr(encoded);
    std::ostringstream ostr;
    Poco::Base64Decoder b64in(istr);

    copy(std::istreambuf_iterator<char>(b64in),
    std::istreambuf_iterator<char>(),
    std::ostreambuf_iterator<char>(ostr));

    return ostr.str();
}

How I build the code:

#!/bin/bash

set -e
set -x

rm -rf build
mkdir build
pushd build

conan install ..
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .

ls -lah bin/main
bin/main

Answer 1

Take a careful look at Poco/Config.h as there a several macros that allow you to disable certain parts of Poco. These macros exist to help you easily strip down your binaries if you don't want them (ex: XML, JSON, INI config files, also POCO_NO_AUTOMATIC_LIBS). I would expect these and others to reduce your object file size.

I know this was quite a while ago, but Poco has been used to run a web server on a very small "board". See https://pocoproject.org/blog/?p=193.

Answer 2

There isn't anything wrong with what you are doing. It might just be that poco has a lot of dependencies and functionality. You can add message(STATUS "Linkd libraries: " ${CONAN_LIBS}) in CMakeLists.txt and run cmake again to view the libraries that you are currently linking with when using ${CONAN_LIBS}.

You could also try using conan_basic_setup(TARGETS) in CMakeLists.txt instead of just conan_basic_setup(). If you do this, then you need to change target_link_libraries(${PROJECT_NAME} ${CONAN_LIBS}) to target_link_libraries(${PROJECT_NAME} CONAN_PKG::poco). This allows you to have finer control of which libraries in conanfile.txt you link with each target in CMaksLists.txt. But since your conanfile.txt only has poco as a dependency, this shouldn't change anything.

Another thing that you can try is checking if the poco recipe in conan has any options you can set to include/exclude parts of the poco library. Run the command below (assuming poco/1.10.1 is already installed in your conan cache)

conan get poco/1.10.1

This will show you the full recipe for poco that conan is using. Here I got

from conans import ConanFile, CMake, tools
from conans.errors import ConanException, ConanInvalidConfiguration
from collections import namedtuple, OrderedDict
import os


class PocoConan(ConanFile):
    name = "poco"
    url = "https://github.com/conan-io/conan-center-index"
    homepage = "https://pocoproject.org"
    topics = ("conan", "poco", "building", "networking", "server", "mobile", "embedded")
    exports_sources = "CMakeLists.txt", "patches/**"
    generators = "cmake", "cmake_find_package"
    settings = "os", "arch", "compiler", "build_type"
    license = "BSL-1.0"
    description = "Modern, powerful open source C++ class libraries for building network- and internet-based " \
                  "applications that run on desktop, server, mobile and embedded systems."
    options = {
        "shared": [True, False],
        "fPIC": [True, False],
    }
    default_options = {
        "shared": False,
        "fPIC": True,
    }

    _PocoComponent = namedtuple("_PocoComponent", ("option", "default_option", "dependencies", "is_lib"))
    _poco_component_tree = {
        "mod_poco": _PocoComponent("enable_apacheconnector", False, ("PocoUtil", "PocoNet", ), False),  # also external apr and apr-util
        "PocoCppParser": _PocoComponent("enable_cppparser", False, ("PocoFoundation", ), False),
        # "PocoCppUnit": _PocoComponent("enable_cppunit", False, ("PocoFoundation", ), False)),
        "PocoCrypto": _PocoComponent("enable_crypto", True, ("PocoFoundation", ), True),    # also external openssl
        "PocoData": _PocoComponent("enable_data", True, ("PocoFoundation", ), True),
        "PocoDataMySQL": _PocoComponent("enable_data_mysql", False, ("PocoData", ), True),
        "PocoDataODBC": _PocoComponent("enable_data_odbc", False, ("PocoData", ), True),
        "PocoDataPostgreSQL": _PocoComponent("enable_data_postgresql", False, ("PocoData", ), True),    # also external postgresql
        "PocoDataSQLite": _PocoComponent("enable_data_sqlite", True, ("PocoData", ), True),  # also external sqlite3
        "PocoEncodings": _PocoComponent("enable_encodings", True, ("PocoFoundation", ), True),
        # "PocoEncodingsCompiler": _PocoComponent("enable_encodingscompiler", False, ("PocoNet", "PocoUtil", ), False),
        "PocoFoundation": _PocoComponent(None, "PocoFoundation", (), True),
        "PocoJSON": _PocoComponent("enable_json", True, ("PocoFoundation", ), True),
        "PocoJWT": _PocoComponent("enable_jwt", True, ("PocoJSON", "PocoCrypto", ), True),
        "PocoMongoDB": _PocoComponent("enable_mongodb", True, ("PocoNet", ), True),
        "PocoNet": _PocoComponent("enable_net", True, ("PocoFoundation", ), True),
        "PocoNetSSL": _PocoComponent("enable_netssl", True, ("PocoCrypto", "PocoUtil", "PocoNet", ), True),    # also external openssl
        "PocoNetSSLWin": _PocoComponent("enable_netssl_win", True, ("PocoNet", "PocoUtil", ), True),
        "PocoPDF": _PocoComponent("enable_pdf", False, ("PocoXML", "PocoUtil", ), True),
        "PocoPageCompiler": _PocoComponent("enable_pagecompiler", False, ("PocoNet", "PocoUtil", ), False),
        "PocoFile2Page": _PocoComponent("enable_pagecompiler_file2page", False, ("PocoNet", "PocoUtil", "PocoXML", "PocoJSON", ), False),
        "PocoPocoDoc": _PocoComponent("enable_pocodoc", False, ("PocoUtil", "PocoXML", "PocoCppParser", ), False),
        "PocoRedis": _PocoComponent("enable_redis", True, ("PocoNet", ), True),
        "PocoSevenZip": _PocoComponent("enable_sevenzip", False, ("PocoUtil", "PocoXML", ), True),
        "PocoUtil": _PocoComponent("enable_util", True, ("PocoFoundation", "PocoXML", "PocoJSON", ), True),
        "PocoXML": _PocoComponent("enable_xml", True, ("PocoFoundation", ), True),
        "PocoZip": _PocoComponent("enable_zip", True, ("PocoUtil", "PocoXML", ), True),
    }
    
    for comp in _poco_component_tree.values():
        if comp.option:
            options[comp.option] = [True, False]
            default_options[comp.option] = comp.default_option
    del comp

    @property
    def _poco_ordered_components(self):
        remaining_components = dict((compname, set(compopts.dependencies)) for compname, compopts in self._poco_component_tree.items())
        ordered_components = []
        while remaining_components:
            components_no_deps = set(compname for compname, compopts in remaining_components.items() if not compopts)
            if not components_no_deps:
                raise ConanException("The poco dependency tree is invalid and contains a cycle")
            for c in components_no_deps:
                remaining_components.pop(c)
            ordered_components.extend(components_no_deps)
            for rname in remaining_components.keys():
                remaining_components[rname] = remaining_components[rname].difference(components_no_deps)
        ordered_components.reverse()
        return ordered_components

    _cmake = None

    @property
    def _source_subfolder(self):
        return "source_subfolder"

    @property
    def _build_subfolder(self):
        return "build_subfolder"

    def source(self):
        tools.get(**self.conan_data["sources"][self.version])
        extracted_folder = "poco-poco-{}-release".format(self.version)
        os.rename(extracted_folder, self._source_subfolder)

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC
        else:
            del self.options.enable_netssl_win
        if tools.Version(self.version) < "1.9":
            del self.options.enable_encodings
        if tools.Version(self.version) < "1.10":
            del self.options.enable_data_postgresql
            del self.options.enable_jwt

    def configure(self):
        if self.options.enable_apacheconnector:
            raise ConanInvalidConfiguration("Apache connector not supported: https://github.com/pocoproject/poco/issues/1764")
        if self.options.enable_data_mysql:
            raise ConanInvalidConfiguration("MySQL not supported yet, open an issue here please: %s" % self.url)
        if self.options.get_safe("enable_data_postgresql", False):
            raise ConanInvalidConfiguration("PostgreSQL not supported yet, open an issue here please: %s" % self.url)
        for compopt in self._poco_component_tree.values():
            if not compopt.option:
                continue
            if self.options.get_safe(compopt.option, False):
                for compdep in compopt.dependencies:
                    if not self._poco_component_tree[compdep].option:
                        continue
                    if not self.options.get_safe(self._poco_component_tree[compdep].option, False):
                        raise ConanInvalidConfiguration("option {} requires also option {}".format(compopt.option, self._poco_component_tree[compdep].option))

    def requirements(self):
        self.requires("pcre/8.41")
        self.requires("zlib/1.2.11")
        if self.options.enable_xml:
            self.requires("expat/2.2.9")
        if self.options.enable_data_sqlite:
            self.requires("sqlite3/3.31.1")
        if self.options.enable_apacheconnector:
            self.requires("apr/1.7.0")
            self.requires("apr-util/1.6.1")
            raise ConanInvalidConfiguration("apache2 is not (yet) available on CCI")
            self.requires("apache2/x.y.z")
        if self.options.enable_netssl or \
                self.options.enable_crypto or \
                self.options.get_safe("enable_jwt", False):
            self.requires("openssl/1.1.1g")

    def _patch_sources(self):
        for patch in self.conan_data.get("patches", {}).get(self.version, []):
            tools.patch(**patch)

    def _configure_cmake(self):
        if self._cmake:
            return self._cmake
        self._cmake = CMake(self)
        if tools.Version(self.version) < "1.10.1":
            self._cmake.definitions["POCO_STATIC"] = not self.options.shared
        for comp in self._poco_component_tree.values():
            if not comp.option:
                continue
            self._cmake.definitions[comp.option.upper()] = self.options.get_safe(comp.option, False)
        self._cmake.definitions["POCO_UNBUNDLED"] = True
        self._cmake.definitions["CMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP"] = True
        if self.settings.os == "Windows" and self.settings.compiler == "Visual Studio":  # MT or MTd
            self._cmake.definitions["POCO_MT"] = "ON" if "MT" in str(self.settings.compiler.runtime) else "OFF"
        self.output.info(self._cmake.definitions)
        # On Windows, Poco needs a message (MC) compiler.
        with tools.vcvars(self.settings) if self.settings.compiler == "Visual Studio" else tools.no_op():
            self._cmake.configure(build_dir=self._build_subfolder)
        return self._cmake

    def build(self):
        if self.options.enable_data_sqlite:
            if self.options["sqlite3"].threadsafe == 0:
                raise ConanInvalidConfiguration("sqlite3 must be built with threadsafe enabled")
        self._patch_sources()
        cmake = self._configure_cmake()
        cmake.build()

    def package(self):
        self.copy("LICENSE", dst="licenses", src=self._source_subfolder)
        cmake = self._configure_cmake()
        cmake.install()
        tools.rmdir(os.path.join(self.package_folder, "lib", "cmake"))
        tools.rmdir(os.path.join(self.package_folder, "cmake"))

    @property
    def _ordered_libs(self):
        libs = []
        for compname in self._poco_ordered_components:
            comp_options = self._poco_component_tree[compname]
            if comp_options.is_lib:
                if not comp_options.option:
                    libs.append(compname)
                elif self.options.get_safe(comp_options.option, False):
                    libs.append(compname)
        return libs

    def package_info(self):
        suffix = str(self.settings.compiler.runtime).lower()  \
                 if self.settings.compiler == "Visual Studio" and not self.options.shared \
                 else ("d" if self.settings.build_type == "Debug" else "")

        self.cpp_info.libs = list("{}{}".format(lib, suffix) for lib in self._ordered_libs)
        
        if self.settings.os == "Linux":
            self.cpp_info.system_libs.extend(["pthread", "dl", "rt"])

        if self.settings.compiler == "Visual Studio":
            self.cpp_info.defines.append("POCO_NO_AUTOMATIC_LIBS")
        if not self.options.shared:
            self.cpp_info.defines.append("POCO_STATIC=ON")
            if self.settings.compiler == "Visual Studio":
                self.cpp_info.system_libs.extend(["ws2_32", "iphlpapi", "crypt32"])
        self.cpp_info.names["cmake_find_package"] = "Poco"
        self.cpp_info.names["cmake_find_package_multi"] = "Poco"

What you want to see is what the recipe has in options and default_options. As far as I know, there is no way to query which options a recipe provides and what they do besides looking at the actual recipe source code like this.

It seems that the poco recipe adds a lot of options from this _poco_component_tree dictionary. What you want to check is the options with names enable_something with value True. Since these are added as options, it means that the client (you running conan) can control these when running conan install. For instance, try the command below (you can add multiple -o poco:something to set multiple options)

conan install .. -o poco:enable_data_sqlite=False

We can see in the requirements method in the recipe that only when enable_data_sqlite is True conan will add "sqlite3/3.31.1" is a poco dependency. That means if you set enable_data_sqlite to False then it should not be included at all and your binary should get smaller.

Since conan (and the poco developers or whoever created the recipe for poco) wants to make installing poco using conan as easy as possible it makes sense to include the most common parts of poco by default. Using conan options to disable parts of it is the way you can control this. You will have to try a few of these options to see what you get. In case you disable something you actually need you will get errors when compiling and/or linking your actual code.