Advanced Dependencies Model in Conan 2.0 C, C++ Package Manager @ ACCU 2022

October 24, 2022

2 min read

Advanced Dependencies Model in Conan 2.0 C, C++ Package Manager – Diego Rodriguez-Losada – ACCU 2022

Slides & Materials:… Website: Conan C, C++ package manager has been running the 1.X stable version, up to recent 1.43, for the last 4 years. One of its limitations was the relatively simple dependencies model, with “requires” to model “library” dependencies and “build_requires” to model “build time tools” dependencies. While this model works well in some of the most typical scenarios, like applications linking all static libraries, it can be limited in some advanced scenarios, like mixing shared and static libraries in different topologies, or the (dreaded, but still happening in real life projects) case of depending on different versions of the same static library in the same project, via isolation of shared libraries. Conan 2.0 introduces a new dependencies model with requirements “traits” like visibility, definition and propagation of headers and libraries independently, and more that allow modeling all these advanced use cases. This talk will present this new model, and apply it to solve different advanced use cases, with real life examples. It will also describe the ABI compatibility implications, and how it can affect and be leveraged while building large C and C++ projects at scale in CI.

View Slides Here


Diego Rodriguez-Losada Gonzalez

    Diego Rodriguez-Losada Gonzalez

    Diego Rodriguez

    Diego Rodriguez-Losada Gonzalez 's passions are robotics and SW engineering and development. He has developed many years in C and C ++ in the Industrial, Robotics and AI fields. Diego was also a University (tenure track) professor and robotics researcher for 8 years, till 2012, when he quit academia to try to build a C / C ++ dependency manager and co-founded a startup. Since then he mostly develops in Python. Diego is a C / C ++ package manager co-creator and maintainer, now working at JFrog as senior SW engineer and C / C ++ advocate.

    Video Transcript

    good afternoon uh i am diego i am one of the two conan co-founders
    and today i’m going to talk about about conan first hey what is conan conan is a free and
    open source package manager mit license it is universal it can target any operating system and it can integrate
    with any build system it is very efficient it can manage binaries for managing huge projects and
    is stable very actively developed and very supported by both the maintainers
    the company and the community what i’m going to talk today is about
    the graph about the graph model in tuxedo all i but i’m going to start first with how c and c plus plus
    binaries are built into into artifacts then i will talk about how conan one x
    represents the dependencies in the package measure the limitations of the of this model and
    how conan 2 can improve over these limitations i will finish with a demonstration and conclusions
    so first let’s start with this this simple example we want to build a small video game
    and the game application depends on an engine library and this engine library in turn depends
    on a math library that contains typical math operations and both engine and math are static
    libraries by default and game is unexecutable so the interesting thing here is when i
    build some something in the math static library something like the add function
    that is having two integers then i will be getting an assembly that looks like this by the way this is visual studio
    in debug mode but here we can see the add operation and the symbols
    and then when we build the engine cppn that contains a move3d function that
    calls twice the add function then i will get an engine static library which assembly looks like this
    we can see that there are two code to the add function which is basically a pointer this is not
    resolved in the engine static library this will be resolved later when we call
    the linker but the add function is not inside the engine static library
    then when we build the game executable uh linking against these two flaggers we
    will get something like this we get disassembly for the game executable we see the main the main function here that
    is calling move3d from the engine static library and in the assembly of
    the executable we will see a call that is a jump basically to other players of the assembly because
    and in turn this the move 3d assembly code will be doing two calls to the add
    function that is also in depth in that assembly because the linker would it did it took two copies
    the copies of the static libraries assembly code and it put it into the into the game executable of course if we
    build in in the release mode this this will be optimized and it will be in line
    everything but the concept is still the same we are getting copies of the assembly code of the static libraries
    into our executable but if now we decide that the engine
    library we want to build us a shared library then we will get something like this
    the math static library will still still be the same same assembly nothing changes but when we are building the
    engine certainly the dll in this case we will get an assembly that looks like
    this again debug mode non-optimized but the the concept is still the same we see that the assembling engine dll
    contains a call to the and function and it also contains a copy of the add function itself in the assembly of the
    server driver and now the game executable that is built
    against this engine the ln cert library now is something like this it contains a
    call but this call is just a pointer it doesn’t exist in the in the assembly anymore because that will be resolved at
    runtime only when the cert library is loaded by the system this pointer will be actually pointing to the sending code
    in the self library space but again the important thing is it doesn’t
    contain a copy it’s very important to know because everyone is so excited and
    so am i this is great models are going to be great but it’s a build system thing it’s not it’s not a package
    management thing it’s not an artifact thing executables search libraries study
    libraries will still be the same and exactly the same thing that we are seeing like assembly code from an
    artifact that is being embedded in the in another you know native artifact it still happens with models it’s exactly
    the same by the way if someone wants to learn more about models i like this talk from daniel anger
    so i suggest having having a look okay let’s see how this affects
    package management and in this case we will start describing the economy one model and how it manages the packages
    and the binaries first this is the basics we have the math
    library and for creating a package of the math library we have a column file
    py it’s what we call a recipe a recipe is a python script and that contains the description how
    these package artifacts are built from sources okay so is this in a simplified way
    this is a recipe that contains a source method how to get the sources the build method how to build from sources and the
    package method how to actually extract the artifacts and create the final batch
    okay so when when with this package recipe we can do something like this we we can
    clone the the math repository and we can issue a coin and create command this concrete command will take the column
    file recipe and it will start executing the the methods and we will end
    with a math library binary for for this for this source code
    when we want to represent uh dependency in conan this is what we have uh in our
    small example of the game we have the math quantum files exactly what we have seen i’ve removed the methods for
    simplicity but the methods would be there and then in the same it will contain also methods for building from
    source i’ve removed them the important bit here is the requires to represent a
    dependency from engine to math we have a class attribute that is called requires
    that will point to the to the package and the version of the packet that we want to use or if we need some conditional logic we
    need something more powerful we can use the same in the requirements method and
    we can apply some some ifs there you want to conditionally depend on on the math on the math library but the idea is
    is the same we declare a requires from engine to mass
    so when when someone clones the engine library repository and it contains the
    content file and they issue a current install command what happens is conan will go and check the
    the content file from engine it will find the requires to math and then say okay i need math 1.0 it will go to the
    common cache it will go to the servers fetch that math100 package and it will
    find there the content file for the math package
    it will check again okay no requires stop this is my dependency graph
    and once it has the dependency graph it can prepare the thing to build the
    engine package in this case as the as the engine package is declaring a generator to symmeteps is basically 10 a
    engine is built with cna please conan generate for me some cma files so i can use them in my build and what conan will
    do is okay you depend on math so i will create for you a math config.cma file
    that contains a math target that you can use in your bit this is a bit simplified
    but that’s the concept of how kong works so when the developer that is working on the engine source code they can start
    running cmake calling the simic list of engine and it will find the the math config she make
    and it will find the math target it will be able to link with the with the math library and everything will will work
    exactly the same process is is done when we have transitive dependencies in this case if we the developer is going to be
    building from the game the game executable it will be cloning the game repo it will be issue icon and install
    and it will proceed in the same manner it will go to the gamecon and file final requires to engine okay i need to fetch
    the engine package it will find it will look for engine one serial inside the current cast or in the server fetch it find the
    required mass then it will fetch the math package it will find no more requirements there it
    will stop this will be the dependency graph with these three packages and
    now conan will generate two files it will generate a cmak file for engine with the
    engine target and it will generate also a similar file for math so game can now
    link with engine and the math static libraries because it needs them for for being linked
    okay this was a bit simplified model let’s see now how can i manage the binaries so when we’re talking about a
    package like the math package we are actually talking about this we have seen the recipe this conan file
    script that describes how these binaries are built but every time
    we we call account and create we create a package then a binary is produced
    okay and we can have as many as these binaries as we want every time we build
    with a different configuration we are going to get a different binary those binaries are identified by what we
    call the package id the package is is basically a hash it’s a z1 checksum of this
    okay this is the configuration a bit simplified but the idea is there
    if we take the settings some things like the operating system or the architecture the compiler compiler version build type
    okay and we change any of those things we are going to get a different binary our our assembly code will be different
    okay for example if we change from from um x 86 64 architecture to rc7 architecture
    we are going to get a different battery that’s that’s obvious and also there is another thing with
    which what we call the options that is something per package that is for example a package can be a
    static library or a certain library and in some situations can be one or the other if we change that the math static
    library is going to be obviously a different binary than the than the math and search library so if we change the
    option share from false to true we are going to get a different configuration this configuration is going to be hashed
    into a different package id that represents the unique binary that this configuration produces
    this was a bit simplified view of the configuration that is put into the package id computation
    because as we have seen in the first part of the presentation today
    the also the dependencies also matter so if we are building a binary here for
    math and let’s say that math depends on set different boosts and if we are building a a minor against sadly one two
    eight and boost one 68 we are most likely going to get a different binary for math than if we
    build from sadly one to 11 and boost 175 okay so what the package computation the
    package id computation does it also takes into account the versions of the dependencies that i’m using to produce
    my banner okay let’s finalize describe it in
    column one model talking about about two requires so something that a very early in the in
    the process uh when we we were releasing conan users asked for a can i put you’re
    talking about libraries only right and executables can i put a tool itself into a content package and then
    use it for building my other packages like the canonical example ecma can i put cmake a
    mother version of cmake into a content package and then use it to build my other packages and then we created the
    the concept of build requires or tool requires specific name two requires is the modern naming of build requires
    they are declared exactly the same in as a class attribute as we can see here in the engine a recipe two requires equals
    something and that will be a tool required okay in this case the semi one in blue
    the first important thing that we we have to know in column one is that dependency graph for two requires
    changes is not always the same and it changes depending if you need to build from sources or you don’t
    in this case if you are installing the game uh package for example and it is able to
    find a pre-compiled binary for engine it doesn’t need to build engine from source and as it doesn’t need to build engine
    from source then the tool required for engine doesn’t exist at all in the graph so we have the what we have in the left
    side here that’s the full dependency graph cmake doesn’t exist at all in this
    instance of the dependency graph but if in other situation we are building engine from source because
    maybe it doesn’t exist that pre-compiled binary or something and we are somehow
    engine is being rebuilt from source then the tool requires is used and the scale dependency graph will contain the cmake
    tool requires of engine so engine can be built from front source it will require that cma version here
    applied to engine a very interesting thing that two requires introduced in conan one is the
    concept of the context this is very evident in a cross-building context let’s say that now our little
    game we want to target a raspberry pi we want it to build to run in our raspberry
    pi but we are cross-building we are building this thing with our cross-compiler in windows
    so the thing is that our packages the game engine and math should be built from the raspberry pi
    and the configuration that we should be using is the open system is linux and the architecture is rp7 and so on this
    is what we call the host context that’s the the naming that gnu is using and
    that refers to the machine where the server will run in this case the raspberry pi
    but we are cross building and that means that the cma that we are using cannot be the cmake
    built for for linux it has to be a cmake for windows 64 bits
    and that means the configuration for the build context in blue is a different one
    than the one that we are using for the for the host context so basically tool requires
    always live in the build context that is the current build machine that you are using that can be different from the
    host context this was very evident for cross-building
    but it happens the same in other cases a very typical one is this one you you
    want to to build your thing your game in debug mode because you want to debug it right so your configuration will contain
    build type equal debug but that doesn’t mean that cmake you want cmake
    using a cmake that was built in debug mode because it might be very slow or you can have a tons of traces that you
    don’t want you still want the release executable of cmake that means that the that the configuration context for the
    build context will be built like equal release so even in in this case that we are not cross building
    tool requires leaving the build context which is different from the host context
    okay so let’s talk a bit about limitations of the conon 1x model and before that i want to
    say introduce a a number of figures and metrics about
    the project and for example the iso c survey of the last year is showing that uh scotland
    has the as the most user language specific package manager and we also have some data about the for
    example the community in this case this is the number of pull requests that conan repos in github are getting per
    year you can see here in the last year 2021 we got more than four thousand pull
    requests from the committee in our different repos this is the the
    number of visits this is our web traffic most of these and we see that it’s in the in the millions and thousands of of
    unique users most of it like i’m talking 80 to 90 is in our documentation okay
    which is great it’s not they’re not they’re not doing or marketing blog posts or whatever they are reading our documentation
    and another good metric this is the this is the telemetry data we have for the artifactory server gave artifactory
    servers uh that means that there are at least out there 8 000
    teams companies running running in production okay every team this each one
    of these servers can serve teams from a few developers to hundreds of other developers
    they range typically they range from startups and we know that many of these companies are fortune 100 companies we know they
    are they are using us why am i saying this i want to highlight
    um that’s we this is not something that we are inventing this is not something that we want this is the result of
    thousands of of uh support tickets thousands of conversations to customers saying hey
    team we need this this is important for us you need to improve this this is limiting for us this is painful
    whatever so this is what i’m presenting here and this is the reason conan 2 is
    going to evolve in this direction because we we have been listening to to all this much feedback
    so let’s start with the limitations first about the transitive headers visibility
    so this is our small example here we can see a
    standard thing we have our simic list we have our game conan with our request to engine and we have the game
    cpp source code okay so this works in conan one okay it’s fine you do the conan install you get your files you can
    build the thing and everything it works okay but then one day
    some user came to us and said hey but one of my developers added this include math.h in-game
    and it it compiled and said hey great it worked right say no no it really needs
    to fail if this is a bad thing this shouldn’t be possible at all and we say oh
    wow yeah let’s stop for what is happening here okay yes
    the content install is creating the engine cmake and the math cmake files for us
    both of them locate the engine libraries and engine headers and the math libraries and math headers so
    the math headers are by default exposed to game okay and there is little that we
    can do because actually we have not enough information in common one to deal with us but
    we were thinking about is this is a toy example of course in these three packages graph it doesn’t doesn’t it’s
    not a big deal but many of our users they are having graphs of many doses i’m talking 150
    packages dependency graph there and when they have up to hundreds of developers working on the same dependency graph and
    in the same project this is a problem because engine can evolve and the team that is developing engine
    might decide to drop math dependency or to upgrade it or whatever
    and game only depends on engine okay and then suddenly
    something game is not compiling anymore it is hey but why game only dimension engine engine the api of engine is still
    great it doesn’t it didn’t change the api of engine at all why is game breaking
    and the the the answer is a because some developer introduced this indirect
    dependency via this include okay and we it to be catch very early in the process
    so it’s not a problem later when something changes in the in the dependency graph now we say yes you are
    totally right the default the default for for headers is that they shouldn’t be propagating downstream
    and packages should explicitly define what they require in order to use those headers in the default case
    something very similar happens to to the libraries and the linkage requirements so so far we have been seeing and we
    have been talking static libraries but what happens if engine has an option
    a shared equal true or false it can be a static library or a shared library and
    now suddenly now is someone is doing good uninstall engine search equal true
    okay and now we have this this this thing that happened in the assembly code that we are seeing the math assembly is
    being linked into the engine dll and now game shouldn’t receive any more the linkage
    of math at all but the problem is okay our files don’t know that
    they they don’t model that at all so the math config and the engine config cement
    files they are still propagating the linkage requirements to game again in most cases it doesn’t matter
    the linkers are smart they can they can strip away all the symbols that are not used so if game doesn’t have any any is
    not using any function directly from from math it’s not going to be a problem the math
    assembly will not be copied into game but yeah that’s simple works but if you have
    a 150 depend uh packages dependency graph maybe your lean time is minutes
    many minutes okay just because it’s doing a lot of job and you want it to be efficient or you have a complicated link
    step and sometimes it will really embed a copy of math into game because yeah
    but like and then your game size starts to grow it won’t break but it’s just annoying to see the the size of your
    assembly executable to grow without the reason and then yes game shouldn’t be
    linking directly with math because it has already been embedded into the into the engine executable dll
    okay about conflicts and dependencies visibility this is the typical diamond
    structure that we can have in dependency graphs in this case we have an artificial intelligence package that also depends
    on math it defines its requires to math 1 0 besides the engine packet that already
    depends on math and if game now depends on both the artificial intelligence package and the engine
    package we will have this diamond structure okay so far this is good this is work this works
    but this also opens the door to the possibility that we have a conflict because if for example the engine team
    decides to use a modern version of math and they start to use math 2.0 then we
    have this dependency graph and this dependency graph means that game is going to try to link
    against two different binaries of math and then you get the duplicated symbols error all the other way it’s basically
    not possible to have this column one is fine it will detect this version conflict and will tell you hey you have
    a conflict solve it it is also interesting to know that this
    conflict also happens and this is particular for native builds doesn’t happen in in python for example or in
    java but in native netflix it happens when the your binary can be different
    let’s say that the math package has a configuration an option that allows you to define the accuracy
    of the all the mathematics there and it can be either double or float and then the artificial intelligence
    package decides that a float accuracy is good enough and then they define it in their options and in the engine they
    need double accuracy and then they’re they are defining default options so even if now the exactly the same version
    of math is used then the banner is still different and we have two different binaries and if
    game is going to trying to link with these two different math libraries you will get exactly the same the same
    link errors the same duplicate the symbols errors and it basically won’t link
    so conan current one so far did this right it is able to detect the conflicts
    and is able to allow users to resolve them the problem comes when
    this has to be allowed and yeah this happens so some teams
    came to us and say hey but we have this we have the artificial intelligence package it has millions of lines of code
    that depends on math one this is not going to be upgraded anytime soon
    and we also have the ng team that decided to upgrade to math to zero okay we know it’s not ideal but i
    guarantee you that nothing is propagated by the aps so
    artificial intelligence and engine are both certain libraries okay i know it i control them i build
    them they are set libraries anything from math is propagated via the public api
    of ai engine packages nothing nothing is propagated it’s also very important api
    is very evident that is not provided you can check the headers public headers analyze the includes and you can be sure
    but also i know i’ve done a lot of testing and whatever i know that nothing escapes the api
    of the packages no object is constructed in memory and pass along from engine to to ar whatever i know that and no data
    is also is also visible propagated via via the public interfaces of these packages so i know i’m fine this would
    work this should be allowed to work again this is not great we always prefer
    to everything to be upgraded and everything but this is real life and one of our premises in conan is we we want
    to support the real use cases of the users so this should be possible conan one was kind of possible but really
    challenging to get it to get this right oh this this one we were so naive when
    we decided to to model the the package id how we compute the package id regarding the versions of
    your dependencies and this is what we decided okay december is almost a standard right it
    should be right for us as well why we wouldn’t uh well okay so we decided december and
    this means that that if we have the engine package that contains a version range
    that allows math between one and two because we know my package version is api compatible
    we can write this we can have these two dependency graphs in two different points in time first
    it resolves to math one zero and then some time later in time then
    they release the math 1.3 version and then our dependency graph for game contains math 1.3
    so this is what we are baking into the configuration file for taking second dependencies is
    december representation of the dependency and the same representation means major doesn’t break
    so we say okay so the mapping of math 1.0 to assemble compatible
    representation is math 1 dot y dot set and math 1.3
    exactly the same is one dot y dot set that means that exactly the same binary
    for engine should be able to be binary compatible with math 1 and math 1.3
    and that might be okay or it might not it depends in the specifically in the binary embedding case it is not right
    and also it’s okay but it’s fine we you can bump
    the major okay if you bump the major in this case we resolve to math one on the left and on to math two on the right you
    get two different chamber representations in the configuration you get math one dot y dot set on the left
    and math two dot y dot set and those result in two different
    package ids and two different binaries now we have a binary for engine that is binary compatible with math one
    dot whatever versions and we have another binary for engine that is binary
    compatible with all math 2.0 versions okay really bad things project in c plus plus
    specifically open source but also company projects they don’t use sember this way they use
    basically the the major version for simplex class we use it as a generation thing it’s a
    basically never changed in years like boost boost is one dot something forever
    right and we use the minor as the major and we do breaking changes
    we remove things we change the apis in the minor okay so just expecting projects to bound
    the major to for this to make sense simply simply doesn’t work
    okay and finally the last limitation that i’m going to talk is that two requires
    cannot be factored in into the package id computation so even if i know that using different versions of c make will
    affect my binary i cannot do this i cannot ask conan to the a if you’re using this
    c make version then i want a different binary because maybe c make introduces some some compiler flags here
    okay this is not possible and i’m going to explain why i’m going to explain how conan is actually computing the binaries
    that are to be used the first stage in the dependency graph resolution is computing the graph of the
    host context so we resolve the game that depends on engine and depends on math
    in the host context then we take the configuration we take the settings the options and the
    versions of the dependencies and we feed them into the graph and we compute the package ids
    for for this for every package in the in the graph then we check
    if this package ids are found in somewhere yeah do we have a
    pre-compiled binary for this package id in the cash or in the remote we are able to find in this example
    math and game binaries free compartments okay everything good but for some reason the engine binary
    is not there then only in that case conan decides to expand the dependency
    graph and add a new node to the graph expanding the tool requires
    that basically means that this is a chicken and egg problem for knowing when we want to expand the
    dependency graph to what the tool requires we first had to compute the package id
    and then that means that all the tool requires cannot be part of the package id
    okay so let’s see how content2 is going to improve overall all of these problems
    i wanted to to also briefly introduce this because i’m i’m
    really excited about cornell industrial currency is going to be a huge release and it is
    improving many many things in conan conan one is already five years old we
    have been doing a great job trying to maintain compatibility in all this time for all the conan one series
    but there is already too many legacy too many bad defaults to make things that have changed in time
    so now quantum 2 is going to change all those things we are going to have a a new user commands layer that you can
    customize we are going to have a public python api something that our users are really looking forward we have a new
    binary compatibility system new business integrations many many things the log files for example and the ci is going to
    improve a lot as well so many things are going to change today i wanted i only
    have time to talk about the dependency graph but i’m i’m so excited about what is coming coming with conan and tusito
    so this is the new killer proposal the new amazing proposal of conan antucido is this
    is we will handle the requires with ourselves requires math 1.0 this is the the major
    advance in the graph mode of course the truth is there is a lot more there but as we are going to
    propose also a smooth syntax compatibility to move forward from column one to conan
    to zero we also have to be very careful with being able to write this this common
    syntax between the two the two versions so this was important this really looks like exactly the same
    but it is not the same the major difference here is now that the requires they have to introduce a concept that we
    call requirement traits and a regular requires self requires math 1.0 by default it contains many
    trades i’m going to start with the headers and the leaves trades the default requires assumes you are
    requiring a library okay and by default when you are requiring a library directly you will use the headers of the
    library and you will use the libraries of that line okay so by default header trade is true
    and the trade is true and now we have the information
    when when we are creating the the cma files to build engine we can use this and we say okay you
    depend on the headers fine i’m going to add include directories property to the math
    target so engine will find the headers in math and you
    are using the ellipse of math fine i will add an interface link libraries property to the math target so engine
    can link with the libraries in math if for some reason we go to the
    we write explicitly in the recipe header sql false then the cmake file that is
    being generated for us will omit will completely omit the include directories property or the math
    target so the math headers will not be available while building engine it’s not something that you typically want to do
    if you are of course linking with using math you will want the headers but this is how conan handles this trade this
    header trades while you are building engine and exactly the same if the leaps
    trade is false now the the cmec fat that is generated doesn’t include the link libraries so it will only provide you
    the headers of map but it will not provide you the link the link libraries of of the math
    libraries okay and then the very interesting
    thing is how the transitive requirements are propagated so we
    we define two different types of requirements of course first the direct explicit requirements are those that are
    defined in recipes so we have the game that contains a direct explicit requires
    to engine and we have engine that defines a requires explicit requires to
    math okay those are direct requires and represented here but the thing is in the dependency graph
    this is creating like kind of a virtual a virtual computer requires and is still
    in a game as any depend on math game also depends on math somehow we need to we need to
    understand how but game depends on math this is a transitive requires is equivalent if as
    if we wrote self dot requires math one in the game recipe that we don’t but
    it’s more or less the idea so
    let’s now the the the key point of the new 2-0 model
    is this how the trades are propagated from the in the graph from upstream to
    downstream so i told you by default requires define true
    and and sorry headers and leaves equal true okay and the question here is okay if
    the explicit requires define this what are the traits how game is going to
    depend on math and what are the values of the headers trade and the leaps trade that are
    defined in this virtual virtual transitive requires from game to math
    this is basically a function this is something the inputs that we have for this function
    is the explicit requires under traits from from game to engine and engine to
    math and also also engine and math packages themselves they have other
    information there we will see later that this useful so we take these four things the two packages information and the two
    required information and with that we can define this f function that will tell us hey the trade that you need to
    to use here downstream is this
    in this case we know it we learn from our users we know that by default if we have these input trades and we have this
    these input packages we want the header straights to be false and we want the
    library trades to be true because game needs to still need to link with the math methodic library
    okay so that means hey i told you when you are we are using
    this information when we are generating the files in this case that we are getting the math config cement file for
    building game as the header trade from the transitive requirements from game to math is false
    then we are not no longer generating the input directories for math and now we can build game and game will not have in
    this include paths it will not have the math headers at all because the header sql falls straight remove it
    okay say okay this is great it’s looking great right until someone tells you hey but hi i have this
    i have in my engine package i have some classes there that uses matrix and i have a
    include a directive here that is using things from math
    that means you cannot remove the the headers from math in game they should be visible because the preprocessor is
    going to need them there okay then you with this you have just
    broken this say okay yeah don’t worry now we have this trade framework to define how how
    requirements work so we introduce explicitly transitive headers trait and we now have a mechanism to
    tell a engine if engine the engine package creator knows that he’s using the math headers
    in the public api and those headers are are visible will be necessary in the
    include paths in game it can defines a me engine i requires math
    i request the headers i require the libraries myself but also my consumers are going to
    report the headers because they are visible over my apis then if we get this trade
    over the f function then we will it will resolve it will result in a in a transitive requires
    from game to math that now contains the headers trade equal true
    and the opposite if another uh scenario and some some developer there say okay
    engine i’m a cert library i’m always bitter library i know
    that the static library that i’m linking math doesn’t need to be propagated down to
    the game so i can specify in my requires i can specify a trade transitive leaves equal
    false okay and that is taken by the f function and it’s propagating down to the
    consumer so headers it will be equal false that was our default that keeps
    being the the right one and now leaps is false true that means a
    in this scenario game doesn’t depend doesn’t need the math headers at all
    and it doesn’t need the math libraries at all because it has in the middle this engine insert library that is completely
    the decoupling decoupling the the thing and it will happen it will it will
    basically those traits will result in a math config that is empty it doesn’t define the libraries or the headers from
    game to math okay and actually as once the information is
    there it can be even smarter it can say okay that’s that’s perfect i don’t need
    the headrest of math i don’t need the libraries of math why fetch in the the math binary at all
    i can now build game application and it will only need the binary of
    engine it doesn’t it does no longer need the mathematical library in my computer at
    all and this can result in huge time savings because if you have a 150 package
    dependency graph and you are building an application that depends on just three cert libraries to build an application
    doing an economy store whatever will fetch you just three binary packages out of the 150 and that is really saving you
    a lot of time in ci and also in your developers machine and developers and developers time
    more trades now we are loving these trades they are really powerful and yes they allow us more one of the things
    that conan one didn’t manage at all was the runtime it didn’t have information
    it was everything was built time it was only considering what you do with libraries at build time but
    some of these libraries might be needed at runtime when you are building your application this was this functionality
    was was improved adding the run the runtime trade in this case if we know for example that
    math is always a certain library and we know it engine no knows know that or math a very common use case even if math
    is static library but that package contains some xml files
    something that is going to be loaded at runtime it happens i don’t know in computer vision pattern recognition many things
    that depends on on a file there you actually need the math package to be
    to be installed in your machine at runtime to get those files from the from the package
    that information can be provided with the run with the run credit and this is just just another trade that
    it doesn’t affect the bill at all so the math config that you make is not affected by run it’s still driven by the
    headers and lips and lip straights but if you start defining run equal true trait then conan
    2 will start generating files for you like this conan brand shell or bad depends on your system
    which contains variables that you can activate and it will put things in the path so for example if math is a certain
    library i’m going to know that because someone defined run equal true it will generate for you a script that contains
    you can activate and it will contain the path to all the search libraries so every everything works without messing
    without copying around the the certain libraries from one place to another or we have without having to manage
    yourself this run trade is by default propagated
    if someone defines engine defines a math is going to be needed at runtime then game is going to need math at
    random it doesn’t it doesn’t depend how game depends on engine the the mapping is very very direct
    someone requires something at runtime downstream you are going to need data and runtime as well
    okay things have started to get a bit complicated right i’m i’ve explained a few traits so far
    and headers sleeps run transitive headers and leaves and now i have my requirements
    that i can start assigning values if i know it’s a static library i will do this if i know it’s a cert library maybe
    i want to do this other thing if my headers are visible i want to to define those right so it seems that
    my recipes can get complicated very quickly
    because the truth is when we are dealing with different different different types of libraries
    my trades will be different if i depend something depends on on an application then you want headers it is going to be
    false you are not going to use the headers of an application libraries will be false as well but run will be true
    because you want to use that at runtime okay but if you are depending on a cert library then your headers will be true
    you want still want to link with the library your libraries will be true you want to link them and also you need the
    artifacts at runtime so it will have the three traits to true and if it is static it will be the same but the run will be
    equal false because you no longer need a static gallery at runtime and similar with a header library
    so now we have this we have our requires and we need to start defining different
    value for the trades if we are changing the the different the different types of library that we are
    managing and this is where we welcome another addition to the model in current zero is
    the package type okay the package type is just a class attribute that we can define
    in our recipes so we can add in math package type equal static library and we can also add an engine if we want
    package type equal static library okay something something like that the truth is that it’s not even
    necessary always because we have other modeling conan like the options and third option in this case has has become
    a standard at the factor standard every package out there is is using
    option so if some package is defined in the share option the package type can be automatically reduced if served equal
    true it will be a shared library it’s your equal fault will be a static library also header only option is
    common out there for for libraries that can be built in in both header only mode and and compile mode
    so if we detect that the header only option is there and header only option is true of course
    then the packet style will be a header library so basically the values that that
    package type can take at this moment are these applications library static library header library
    library is like a kind of an abstract one it is telling a i want you to really
    force that an option share will be will be defined otherwise i will error okay and then i will deduce based on the
    option i will deduce if it’s a cert library or a static library but i want to make sure that conan knows that that
    option should be defined that should be used to deduce the package stack then we can have uh build scripts to
    model just a packet that contains some cmake stuff for example and python requires
    so now how can we use this to to define the the traits
    so basically something like that in practice this is looking this is the typical use case
    uh how the package type helps so for many cases for example if math is
    a cert library or a static library i only need the options to be defined and the requires
    has to be like sorry about this yeah requires is as simple as this because
    all the all the correct traits from these requires will be obtained will be automatically
    deduced from the package type of math okay and only only if i want to change
    something of course i always have the power i can go to the requires add the trade value that i want and that will be
    respected but otherwise uh defining the auctions is good enough and only in those cases that conan
    doesn’t have enough information then it’s good to define explicitly the package type for example for an application
    it’s not that that easy to know that something is an application for for conan or for example if a package is
    always a certain library always but it doesn’t contain the option because you cannot opt into certain library or
    static library then you can explicitly help conor until a package type equals cert library but only in those cases you
    need to specify that and once that is defined then in most cases the requires doesn’t
    need to explicitly define all the trades values
    okay good let’s continue with a couple of more trades
    and now the bill trade okay we learned before in conan one about about two
    requires and uh now i’m going to explain the bill the bill trade
    so basically the tool required in current zero is defined with the build equal to trade okay adding that you know
    requires that equal truth means a this requires leaves in the build context
    okay this is the meaning of the build trade this is build context it will get the configuration from the build context
    okay and um being in the build context
    typically by default we’ll also get a run equal through this is typically a tool required it’s something you want to
    run it’s an executable that you want to run in your build and as such it will not generate any cement file for
    you that’s ridiculous actually it’s a package so it doesn’t make sense and but it will
    put this this this trade run equal true and build equal true will put
    the the path of cmake into a script that is called conan bill.
    cell or back depending on the system i i told earlier we have now the two
    things we have the conan run for runtime and we have the conan bill scripts for build time so this is something that
    will be activated only when we are building but when we are running the final application cmake doesn’t maybe
    doesn’t need to be in the path at all so that’s a different script
    and the combination of trades run and build run equal false and build equal true
    might make sense in other cases for example let’s say that we want to put a bunch of build scripts uh utilities
    maybe maybe data some image files that we we have there that we only need them
    at build time because they are going to be embedded in my binary for example
    okay so then we don’t need them on runtime because we know that we are they are going to use at build time
    and that means run equal false and then build equal true in that case that combination of trades
    uh conan doesn’t generate any file it doesn’t know exactly what to do is because this is very specific to your
    build and in this case the engine package can can get the access to my
    script artifacts and files via dependencies mysql’s packet folder and if you are getting there the assets like
    images you copy them you build them then you have the path and you do whatever you want in your build to use to use
    those scripts or those assets and now the other problem that we were
    seeing is fixed because now this is the dependency graph that we get in the phase one of the
    graph evaluation okay if you recall that because basically two requires and requires are
    not a different thing anymore they are exactly the same two requires is just an
    alias it’s just uh some defaults it’s identical to define to define itself that requires clause with headers
    false library false run through built through and other things so the thing is
    when you are expanding the dependency graph you always get the full graph including the host context and the build
    context is not an issue you get the graph of the of the recipes it doesn’t have an impact on on time okay but you always
    have the full dependency graph always of all the recipes that are involved in building something in both host context
    and build context and now if you have the whole dependency graph before computing the packages the package ids
    for every package you can factor the bill requires into your package id and you and this is
    now possible with 2-0 basically you can use the same the same rules and put cmake december
    representation of the cma version into the configuration that is going to be hashed into the package id
    okay and finally the visible trait the visible trait is basically
    removing what we have learned so far in in by default visible is true that
    means that a game has a virtual transitive requirements to
    math always even if it removes if even if if this requires has headers equal
    false uh libraries equal false even if it’s not using the artifacts game still contains
    a transitive requires to math but
    this is the mechanism that forces the complete removal of the transitive
    requires if we put visible equal false that means a
    nobody below me nobody below engine will even know if they will not
    see that math package is there is introducing a full a full break
    in the dependency graph so now the transitive requirements from game to map doesn’t exist anymore and of
    course engine is fully responsible to know what they’re doing and know that they are cert library
    embedding whatever but at least this is possible so in our
    version conflict that we had before and we had this situation okay now it is possible to handle that
    let’s see because the actual origin of the conflict of course is this in the previous case
    when visibility is propagated we have a transitive requires from game to math 1
    over the artificial intelligence left branch of the graph and we have a transitive requires from game to math to
    zero that is created by the engine package in the middle and now we have these two requires of math that goes to
    the game executable and in this point is where the conflict happens
    okay so basically if we remove this transitive we make these transitive headers to disappear
    then problem is gone the conflict doesn’t exist anymore and this is basically what is achieved
    with a visual trade if both ai and engine packages they know that nothing
    will conflict downstream they manage the embedding of the binaries they can declare the visible equal false trade
    and then the the graph will not conflict anymore
    looking good on time of course that doesn’t mean that in most situations we want to resolve the
    conflict it’s a good thing to resolve the conflicts a good thing that both ai and engine use the same
    upstream version and we actually close the diamond this might remove many runtime issues and everything so if we
    want to force this into our dependency graph now we have two other traits
    that we can use for this the first one is the override equal true that means a
    over radical true means game doesn’t directly depend on math it will not introduce a direct dependency between
    game and math what is telling the override is that a if someone upstream depends on math i
    want them to depend on math to zero okay so this is basically forcing in this case the ai package that depends on
    math one one zero is forcing ai package two depends on math to zero instead and closing the
    clause in the graph of course that this this might create compilation problems in the artificial intelligence
    package but that’s another issue developers should go there improve the api but at least from the conan point of
    view you can create the graph that you want this is the graph that you want and you can define it
    specifying the overall equal true and we have a different trait for when
    you have explicit you have a different game for some reason directly depends on math it declare it declares oh something
    wrong there so i no it’s fine this means game explicitly declares directly on math
    but is to say hey i depend on math 2.0 and if some of my dependencies in this
    case ai is dependent on math i want them to be forced into to see it as well
    okay because the thing here is that we we are according to zero is more strict
    it will it will throw more conflicts your way if there are actual russian
    conflicts conan one was doing overrided overriding in most situations that you wouldn’t
    expect because it was not explicit but it was still overriding now it will always conflict and you need to
    explicitly force or overwrite if you want the dependency graph to be the way that you want
    and finally the new package id this is the video that is going to save a ton of co2
    emissions out there i really hope so so we we saw that the the corner one
    package id mode is is getting the chamber representation and it depends on math one three two
    and this is mapping as math one one dot y dot set
    okay the the conan two trade model introduce a package id trade that allows you to
    explicitly change that so engine can explicitly opt in to depend on math in patch mode
    patch mode means a everything down to the patch will be put in the configuration file
    and that will be factored in into the package id okay this is good conan one has something similar this is not a
    really really a killer feature the really clear feature is this is that we have changed the default package at the
    computation so even if recipes they don’t need to change anything in their package id this will happen
    okay first we there are two modes there are two major modes the first one is the
    non-embedding mode okay which is the case that happens when an application is dependent on a cert
    library or a shared library depends on another certain library or a static library depends on another static
    library the the assembly code will not be embedded in the consumer
    assembly code okay in this case we are going to a minor to a minor mode that means that if
    engine depends on math one three two by default we are we are going to put in the in the same representation we are
    going to put 1.3.7 that means that by default every time
    the miner is bumped you will need a different binary of engine but
    if only the patch of math is bumped you will not need a new a new binary of engine that allows
    if your developers know what they are doing they can do patches to math and this will not require rebuilding
    engine downstream engine or other dependency dependencies in the
    graph of course and this can save a lot of time if your team is disciplined and they are really managing versions
    correctly patch releases of a package of a static library will not rebuild the whole world
    okay and the other mode is the embed mode for all the other cases for example an
    application linking a static library or a header library or a cert library
    linking a static library or a header library or a static library link in a header library by default this is an
    embedded mode the assembly of the of the dependency will be will be copied into
    the assembly of the consumer okay in this case we have gone full mode that
    means hey you take the whole reference of the binary dependency which is the the version one three
    two full the revision which is an automatic version that canon is adding which is
    the hash of the contents you change a comma in your source code and even if you you’re not bumping the
    version conan will compute a new revision for you so that means that changing something in the source code of
    math okay will be will result in a new binary of engine which is exactly what you want hey
    something change in math you need to rebuild engine because the binary will always be different
    okay and even the package id of math is factored in into the into the equation
    why because in the case we have the math packets with float accuracy and double
    accuracy okay if you change that that change the package id of math you also need to rebuild you
    need a different binary of the of the downstream consumer application so absolutely everything of the binary of
    the dependency is factored in into the into the downstream consumer
    and of course one of the things that we are trying to be very careful with column 2 is being flexible if you want to change any
    of these defaults now you have full control you can individually change the embed mode the
    non-embed node the python mode you have full control over the over these modes
    that you can use okay we are good on time i will run a very quick demo
    let’s let’s try it here is the phone good enough
    okay so let’s go to the game uh first thing i’m going
    to do is i’m going to run conan graph info
    just to to show you the dependency graph format by the way this is conan 2 0
    alpha 6 which is already released okay you can go there and you can start trying yourself
    so i’m going to write the html format into my graph html
    and i’m going to show you this this graph html and you will see that this is basically
    the example that we have been seeing today okay we have the game application that depends on engine that in turn
    depends on artificial intelligence and graphics well it’s a bit different but it’s more or less the the same idea that
    in turns depend on mathly here i’m using matlib because when i play the demo uh
    math.h is a different thing and i was getting system conflict so i renamed it to map to to avoid this
    so this is my dependency graph i can create my game package with this syntax for
    example version 1.0 and it will be basically fetching my
    dependencies here we can see that we depend on on math leap
    okay this is dependency to matlab when the my cmake code will be sorry
    about this one second okay my c make will be telling a this it
    created the matlab target for us so it’s actually getting the target from the math
    transitive dependency and it is executing the game which is printing
    one message for every every library there in this case it is executing this code it is written in the math library
    okay and it’s been actually been called twice because it’s been called by the diamond uh dependencies so i’m
    actually using the mat library in my game right and i know this is a very
    small step uh but i’m really excited about this now i
    can go to my game source code here i can introduce
    the matlab.h that actually exists there
    and if i do a conan creed again then my mechanism that the trades propagation
    doesn’t make the headers for math available to game at all and this is what developers get they
    are they can no longer introduce a hidden hidden header inclusion in the
    game from math if they don’t explicitly define a requires from game to math
    okay so let’s remove it
    let’s make sure that this this works and the other thing that i’m going to demo now
    is also a very very simple thing but it’s the thing that i’m really very excited
    here i created my game package right so i can now let’s install it let’s install
    requires equal game 1.0
    and actually the thing that i’m most excited about is this one you see here the requirements
    artificial intelligence skip i no longer need this binary my computer
    to it’s all game engine package skip graphics package skip
    package skip so when i’m installing game in this case i only need to fetch
    the game executable binary and i will not even download all the other packages
    from the servers if they are not needed at all okay this is the thing that will result in saving tons of time
    for things not only this but this will also be applied for building
    so conan 20 will be building way less things if you use versioning correctly
    while not skipping changes if something changes that needs to be rebuilt it will guarantee that things get rebuilt
    cool we are on time perfect about to finish
    okay let’s jump to the conclusions so conan 2 has defined a very powerful
    mechanism its base is built on three things first the requirement traits we
    have seen a bunch of them headers library transitivity visibility build run and so there are there are others by
    the way there is the test one that i haven’t talked about it there is the options for the first time you can
    explicitly tell a this package here has these options okay
    um that that wasn’t possible in column one to cope with the with the complexities
    that could arise from only the required trades the package types
    solve the 90 something percent of the cases that
    will not add extra complexity to your recipes your requires will still be plain requires and just managing the
    package types knowing that something is a static library or a cert library or something really brings the power the knowledge of
    what conan should be doing in order to to do the right things and the mechanism of how how these
    requirement traits are propagated from upstream to downstream uh taking into
    account the package types is is the proposal here and with this proposal we
    we can do amazing things we have seen on demo that we can hide the headers
    visibility we can really avoid the linkage propagating the leak catch through shared libraries if
    it’s not needed we can implement private dependencies very simple we can do
    things like bootstrapping now we cannot have a cmake 320 package to require
    cma319 for example if we want okay the much better package id will be
    saving tons of time of developers and nci the tool requirements and executables is
    managed in a better way conflicts and resolution is is also necessary and the smart skipping of of
    dependencies but the real power of all of these is that for most cases nothing changes
    for for developers so we have something that in many cases it will look like this
    just very simple standard content recipes with options define it in most of the
    cases when it is not necessary it will it will use the package tab explicitly just one attribute to help conan to
    identify what is the package type and it will be this simple in most
    situations but for really weird things that happens out there and i can tell you they happen with with c and c plus
    plus building they happen out there you will be able to implement them you will be able to handle them because now
    the the controls that you have over the dependency graph with the trades is much much bigger
    okay so i wanted to uh end today saying thank you first for for
    being there and um some pointers if you want more information about
    anything the first entry point is of course that’s very simple
    for uh you can go from everywhere from there to the documentation there is already
    2.0 beta documentation out there okay you just need to go to the docs and go
    to the versions and click on the two zero alpha actually and
    if you have we have a training for conan one okay it’s good if you don’t know anything about conan it’s good go there
    for training is a free training video format self-paced is good to get to know about conan one we will duplicate that
    for conan two once conan two is is ga if you have any questions the best resource resources github you can submit
    a github issue for a question it doesn’t need to be a bad report so please if you have any question any issue any problem
    open a github and follow us in in social by the way i wanted to say what have
    been showing to today is conan 2-0 alpha okay the plan is the following we plan to go
    uh to zero alpha seven the next month that will be the last alpha and then we will
    move to beta we plan to do at least four betas is the expected team and then
    we’ll move to ga but beta is being there very very very close um i will be later in in gather town so
    for the online people if you want to to try to reach me i try to be to be there
    if not you can ping me you can reach me in twitter dlr los ala i’m open you have
    any question not only today other day and you want to talk this about and
    discuss anything about what we haven’t seen today please ping me and i’d love to to connect with you and talk to you
    and and everything and now i guess uh time for some questions thank you