Advanced Dependencies Model in Conan 2.0 C, C++ Package Manager @ ACCU 2022

good afternoon uh i am diego i am one of the two conan co-founders
and today i’m going to talk about about conan first hey what is conan conan is a free and
open source package manager mit license it is universal it can target any operating system and it can integrate
with any build system it is very efficient it can manage binaries for managing huge projects and
is stable very actively developed and very supported by both the maintainers
the company and the community what i’m going to talk today is about
the graph about the graph model in tuxedo all i but i’m going to start first with how c and c plus plus
binaries are built into into artifacts then i will talk about how conan one x
represents the dependencies in the package measure the limitations of the of this model and
how conan 2 can improve over these limitations i will finish with a demonstration and conclusions
so first let’s start with this this simple example we want to build a small video game
and the game application depends on an engine library and this engine library in turn depends
on a math library that contains typical math operations and both engine and math are static
libraries by default and game is unexecutable so the interesting thing here is when i
build some something in the math static library something like the add function
that is having two integers then i will be getting an assembly that looks like this by the way this is visual studio
in debug mode but here we can see the add operation and the symbols
and then when we build the engine cppn that contains a move3d function that
calls twice the add function then i will get an engine static library which assembly looks like this
we can see that there are two code to the add function which is basically a pointer this is not
resolved in the engine static library this will be resolved later when we call
the linker but the add function is not inside the engine static library
then when we build the game executable uh linking against these two flaggers we
will get something like this we get disassembly for the game executable we see the main the main function here that
is calling move3d from the engine static library and in the assembly of
the executable we will see a call that is a jump basically to other players of the assembly because
and in turn this the move 3d assembly code will be doing two calls to the add
function that is also in depth in that assembly because the linker would it did it took two copies
the copies of the static libraries assembly code and it put it into the into the game executable of course if we
build in in the release mode this this will be optimized and it will be in line
everything but the concept is still the same we are getting copies of the assembly code of the static libraries
into our executable but if now we decide that the engine
library we want to build us a shared library then we will get something like this
the math static library will still still be the same same assembly nothing changes but when we are building the
engine certainly the dll in this case we will get an assembly that looks like
this again debug mode non-optimized but the the concept is still the same we see that the assembling engine dll
contains a call to the and function and it also contains a copy of the add function itself in the assembly of the
server driver and now the game executable that is built
against this engine the ln cert library now is something like this it contains a
call but this call is just a pointer it doesn’t exist in the in the assembly anymore because that will be resolved at
runtime only when the cert library is loaded by the system this pointer will be actually pointing to the sending code
in the self library space but again the important thing is it doesn’t
contain a copy it’s very important to know because everyone is so excited and
so am i this is great models are going to be great but it’s a build system thing it’s not it’s not a package
management thing it’s not an artifact thing executables search libraries study
libraries will still be the same and exactly the same thing that we are seeing like assembly code from an
artifact that is being embedded in the in another you know native artifact it still happens with models it’s exactly
the same by the way if someone wants to learn more about models i like this talk from daniel anger
so i suggest having having a look okay let’s see how this affects
package management and in this case we will start describing the economy one model and how it manages the packages
and the binaries first this is the basics we have the math
library and for creating a package of the math library we have a column file
py it’s what we call a recipe a recipe is a python script and that contains the description how
these package artifacts are built from sources okay so is this in a simplified way
this is a recipe that contains a source method how to get the sources the build method how to build from sources and the
package method how to actually extract the artifacts and create the final batch
okay so when when with this package recipe we can do something like this we we can
clone the the math repository and we can issue a coin and create command this concrete command will take the column
file recipe and it will start executing the the methods and we will end
with a math library binary for for this for this source code
when we want to represent uh dependency in conan this is what we have uh in our
small example of the game we have the math quantum files exactly what we have seen i’ve removed the methods for
simplicity but the methods would be there and then in the same it will contain also methods for building from
source i’ve removed them the important bit here is the requires to represent a
dependency from engine to math we have a class attribute that is called requires
that will point to the to the package and the version of the packet that we want to use or if we need some conditional logic we
need something more powerful we can use the same in the requirements method and
we can apply some some ifs there you want to conditionally depend on on the math on the math library but the idea is
is the same we declare a requires from engine to mass
so when when someone clones the engine library repository and it contains the
content file and they issue a current install command what happens is conan will go and check the
the content file from engine it will find the requires to math and then say okay i need math 1.0 it will go to the
common cache it will go to the servers fetch that math100 package and it will
find there the content file for the math package
it will check again okay no requires stop this is my dependency graph
and once it has the dependency graph it can prepare the thing to build the
engine package in this case as the as the engine package is declaring a generator to symmeteps is basically 10 a
engine is built with cna please conan generate for me some cma files so i can use them in my build and what conan will
do is okay you depend on math so i will create for you a math config.cma file
that contains a math target that you can use in your bit this is a bit simplified
but that’s the concept of how kong works so when the developer that is working on the engine source code they can start
running cmake calling the simic list of engine and it will find the the math config she make
and it will find the math target it will be able to link with the with the math library and everything will will work
exactly the same process is is done when we have transitive dependencies in this case if we the developer is going to be
building from the game the game executable it will be cloning the game repo it will be issue icon and install
and it will proceed in the same manner it will go to the gamecon and file final requires to engine okay i need to fetch
the engine package it will find it will look for engine one serial inside the current cast or in the server fetch it find the
required mass then it will fetch the math package it will find no more requirements there it
will stop this will be the dependency graph with these three packages and
now conan will generate two files it will generate a cmak file for engine with the
engine target and it will generate also a similar file for math so game can now
link with engine and the math static libraries because it needs them for for being linked
okay this was a bit simplified model let’s see now how can i manage the binaries so when we’re talking about a
package like the math package we are actually talking about this we have seen the recipe this conan file
script that describes how these binaries are built but every time
we we call account and create we create a package then a binary is produced
okay and we can have as many as these binaries as we want every time we build
with a different configuration we are going to get a different binary those binaries are identified by what we
call the package id the package is is basically a hash it’s a z1 checksum of this
okay this is the configuration a bit simplified but the idea is there
if we take the settings some things like the operating system or the architecture the compiler compiler version build type
okay and we change any of those things we are going to get a different binary our our assembly code will be different
okay for example if we change from from um x 86 64 architecture to rc7 architecture
we are going to get a different battery that’s that’s obvious and also there is another thing with
which what we call the options that is something per package that is for example a package can be a
static library or a certain library and in some situations can be one or the other if we change that the math static
library is going to be obviously a different binary than the than the math and search library so if we change the
option share from false to true we are going to get a different configuration this configuration is going to be hashed
into a different package id that represents the unique binary that this configuration produces
this was a bit simplified view of the configuration that is put into the package id computation
because as we have seen in the first part of the presentation today
the also the dependencies also matter so if we are building a binary here for
math and let’s say that math depends on set different boosts and if we are building a a minor against sadly one two
eight and boost one 68 we are most likely going to get a different binary for math than if we
build from sadly one to 11 and boost 175 okay so what the package computation the
package id computation does it also takes into account the versions of the dependencies that i’m using to produce
my banner okay let’s finalize describe it in
column one model talking about about two requires so something that a very early in the in
the process uh when we we were releasing conan users asked for a can i put you’re
talking about libraries only right and executables can i put a tool itself into a content package and then
use it for building my other packages like the canonical example ecma can i put cmake a
mother version of cmake into a content package and then use it to build my other packages and then we created the
the concept of build requires or tool requires specific name two requires is the modern naming of build requires
they are declared exactly the same in as a class attribute as we can see here in the engine a recipe two requires equals
something and that will be a tool required okay in this case the semi one in blue
the first important thing that we we have to know in column one is that dependency graph for two requires
changes is not always the same and it changes depending if you need to build from sources or you don’t
in this case if you are installing the game uh package for example and it is able to
find a pre-compiled binary for engine it doesn’t need to build engine from source and as it doesn’t need to build engine
from source then the tool required for engine doesn’t exist at all in the graph so we have the what we have in the left
side here that’s the full dependency graph cmake doesn’t exist at all in this
instance of the dependency graph but if in other situation we are building engine from source because
maybe it doesn’t exist that pre-compiled binary or something and we are somehow
engine is being rebuilt from source then the tool requires is used and the scale dependency graph will contain the cmake
tool requires of engine so engine can be built from front source it will require that cma version here
applied to engine a very interesting thing that two requires introduced in conan one is the
concept of the context this is very evident in a cross-building context let’s say that now our little
game we want to target a raspberry pi we want it to build to run in our raspberry
pi but we are cross-building we are building this thing with our cross-compiler in windows
so the thing is that our packages the game engine and math should be built from the raspberry pi
and the configuration that we should be using is the open system is linux and the architecture is rp7 and so on this
is what we call the host context that’s the the naming that gnu is using and
that refers to the machine where the server will run in this case the raspberry pi
but we are cross building and that means that the cma that we are using cannot be the cmake
built for for linux it has to be a cmake for windows 64 bits
and that means the configuration for the build context in blue is a different one
than the one that we are using for the for the host context so basically tool requires
always live in the build context that is the current build machine that you are using that can be different from the
host context this was very evident for cross-building
but it happens the same in other cases a very typical one is this one you you
want to to build your thing your game in debug mode because you want to debug it right so your configuration will contain
build type equal debug but that doesn’t mean that cmake you want cmake
using a cmake that was built in debug mode because it might be very slow or you can have a tons of traces that you
don’t want you still want the release executable of cmake that means that the that the configuration context for the
build context will be built like equal release so even in in this case that we are not cross building
tool requires leaving the build context which is different from the host context
okay so let’s talk a bit about limitations of the conon 1x model and before that i want to
say introduce a a number of figures and metrics about
the project and for example the iso c survey of the last year is showing that uh scotland
has the as the most user language specific package manager and we also have some data about the for
example the community in this case this is the number of pull requests that conan repos in github are getting per
year you can see here in the last year 2021 we got more than four thousand pull
requests from the committee in our different repos this is the the
number of visits this is our web traffic most of these and we see that it’s in the in the millions and thousands of of
unique users most of it like i’m talking 80 to 90 is in our documentation okay
which is great it’s not they’re not they’re not doing or marketing blog posts or whatever they are reading our documentation
and another good metric this is the this is the telemetry data we have for the artifactory server gave artifactory
servers uh that means that there are at least out there 8 000
teams companies running running in production okay every team this each one
of these servers can serve teams from a few developers to hundreds of other developers
they range typically they range from startups and we know that many of these companies are fortune 100 companies we know they
are they are using us why am i saying this i want to highlight
um that’s we this is not something that we are inventing this is not something that we want this is the result of
thousands of of uh support tickets thousands of conversations to customers saying hey
team we need this this is important for us you need to improve this this is limiting for us this is painful
whatever so this is what i’m presenting here and this is the reason conan 2 is
going to evolve in this direction because we we have been listening to to all this much feedback
so let’s start with the limitations first about the transitive headers visibility
so this is our small example here we can see a
standard thing we have our simic list we have our game conan file.py with our request to engine and we have the game
cpp source code okay so this works in conan one okay it’s fine you do the conan install you get your files you can
build the thing and everything it works okay but then one day
some user came to us and said hey but one of my developers added this include math.h in-game
and it it compiled and said hey great it worked right say no no it really needs
to fail if this is a bad thing this shouldn’t be possible at all and we say oh
wow yeah let’s stop for what is happening here okay yes
the content install is creating the engine cmake and the math cmake files for us
both of them locate the engine libraries and engine headers and the math libraries and math headers so
the math headers are by default exposed to game okay and there is little that we
can do because actually we have not enough information in common one to deal with us but
we were thinking about is this is a toy example of course in these three packages graph it doesn’t doesn’t it’s
not a big deal but many of our users they are having graphs of many doses i’m talking 150
packages dependency graph there and when they have up to hundreds of developers working on the same dependency graph and
in the same project this is a problem because engine can evolve and the team that is developing engine
might decide to drop math dependency or to upgrade it or whatever
and game only depends on engine okay and then suddenly
something game is not compiling anymore it is hey but why game only dimension engine engine the api of engine is still
great it doesn’t it didn’t change the api of engine at all why is game breaking
and the the the answer is a because some developer introduced this indirect
dependency via this include okay and we it to be catch very early in the process
so it’s not a problem later when something changes in the in the dependency graph now we say yes you are
totally right the default the default for for headers is that they shouldn’t be propagating downstream
and packages should explicitly define what they require in order to use those headers in the default case
something very similar happens to to the libraries and the linkage requirements so so far we have been seeing and we
have been talking static libraries but what happens if engine has an option
a shared equal true or false it can be a static library or a shared library and
now suddenly now is someone is doing good uninstall engine search equal true
okay and now we have this this this thing that happened in the assembly code that we are seeing the math assembly is
being linked into the engine dll and now game shouldn’t receive any more the linkage
of math at all but the problem is okay our files don’t know that
they they don’t model that at all so the math config and the engine config cement
files they are still propagating the linkage requirements to game again in most cases it doesn’t matter
the linkers are smart they can they can strip away all the symbols that are not used so if game doesn’t have any any is
not using any function directly from from math it’s not going to be a problem the math
assembly will not be copied into game but yeah that’s simple works but if you have
a 150 depend uh packages dependency graph maybe your lean time is minutes
many minutes okay just because it’s doing a lot of job and you want it to be efficient or you have a complicated link
step and sometimes it will really embed a copy of math into game because yeah
but like and then your game size starts to grow it won’t break but it’s just annoying to see the the size of your
assembly executable to grow without the reason and then yes game shouldn’t be
linking directly with math because it has already been embedded into the into the engine executable dll
okay about conflicts and dependencies visibility this is the typical diamond
structure that we can have in dependency graphs in this case we have an artificial intelligence package that also depends
on math it defines its requires to math 1 0 besides the engine packet that already
depends on math and if game now depends on both the artificial intelligence package and the engine
package we will have this diamond structure okay so far this is good this is work this works
but this also opens the door to the possibility that we have a conflict because if for example the engine team
decides to use a modern version of math and they start to use math 2.0 then we
have this dependency graph and this dependency graph means that game is going to try to link
against two different binaries of math and then you get the duplicated symbols error all the other way it’s basically
not possible to have this column one is fine it will detect this version conflict and will tell you hey you have
a conflict solve it it is also interesting to know that this
conflict also happens and this is particular for native builds doesn’t happen in in python for example or in
java but in native netflix it happens when the your binary can be different
let’s say that the math package has a configuration an option that allows you to define the accuracy
of the all the mathematics there and it can be either double or float and then the artificial intelligence
package decides that a float accuracy is good enough and then they define it in their options and in the engine they
need double accuracy and then they’re they are defining default options so even if now the exactly the same version
of math is used then the banner is still different and we have two different binaries and if
game is going to trying to link with these two different math libraries you will get exactly the same the same
link errors the same duplicate the symbols errors and it basically won’t link
so conan current one so far did this right it is able to detect the conflicts
and is able to allow users to resolve them the problem comes when
this has to be allowed and yeah this happens so some teams
came to us and say hey but we have this we have the artificial intelligence package it has millions of lines of code
that depends on math one this is not going to be upgraded anytime soon
and we also have the ng team that decided to upgrade to math to zero okay we know it’s not ideal but i
guarantee you that nothing is propagated by the aps so
artificial intelligence and engine are both certain libraries okay i know it i control them i build
them they are set libraries anything from math is propagated via the public api
of ai engine packages nothing nothing is propagated it’s also very important api
is very evident that is not provided you can check the headers public headers analyze the includes and you can be sure
but also i know i’ve done a lot of testing and whatever i know that nothing escapes the api
of the packages no object is constructed in memory and pass along from engine to to ar whatever i know that and no data
is also is also visible propagated via via the public interfaces of these packages so i know i’m fine this would
work this should be allowed to work again this is not great we always prefer
to everything to be upgraded and everything but this is real life and one of our premises in conan is we we want
to support the real use cases of the users so this should be possible conan one was kind of possible but really
challenging to get it to get this right oh this this one we were so naive when
we decided to to model the the package id how we compute the package id regarding the versions of
your dependencies and this is what we decided okay december is almost a standard right it
should be right for us as well why we wouldn’t uh well okay so we decided december and
this means that that if we have the engine package that contains a version range
that allows math between one and two because we know my package version is api compatible
we can write this we can have these two dependency graphs in two different points in time first
it resolves to math one zero and then some time later in time then
they release the math 1.3 version and then our dependency graph for game contains math 1.3
so this is what we are baking into the configuration file for taking second dependencies is
december representation of the dependency and the same representation means major doesn’t break
so we say okay so the mapping of math 1.0 to assemble compatible
representation is math 1 dot y dot set and math 1.3
exactly the same is one dot y dot set that means that exactly the same binary
for engine should be able to be binary compatible with math 1 and math 1.3
and that might be okay or it might not it depends in the specifically in the binary embedding case it is not right
and also it’s okay but it’s fine we you can bump
the major okay if you bump the major in this case we resolve to math one on the left and on to math two on the right you
get two different chamber representations in the configuration you get math one dot y dot set on the left
and math two dot y dot set and those result in two different
package ids and two different binaries now we have a binary for engine that is binary compatible with math one
dot whatever versions and we have another binary for engine that is binary
compatible with all math 2.0 versions okay really bad things project in c plus plus
specifically open source but also company projects they don’t use sember this way they use
basically the the major version for simplex class we use it as a generation thing it’s a
basically never changed in years like boost boost is one dot something forever
right and we use the minor as the major and we do breaking changes
we remove things we change the apis in the minor okay so just expecting projects to bound
the major to for this to make sense simply simply doesn’t work
okay and finally the last limitation that i’m going to talk is that two requires
cannot be factored in into the package id computation so even if i know that using different versions of c make will
affect my binary i cannot do this i cannot ask conan to the a if you’re using this
c make version then i want a different binary because maybe c make introduces some some compiler flags here
okay this is not possible and i’m going to explain why i’m going to explain how conan is actually computing the binaries
that are to be used the first stage in the dependency graph resolution is computing the graph of the
host context so we resolve the game that depends on engine and depends on math
in the host context then we take the configuration we take the settings the options and the
versions of the dependencies and we feed them into the graph and we compute the package ids
for for this for every package in the in the graph then we check
if this package ids are found in somewhere yeah do we have a
pre-compiled binary for this package id in the cash or in the remote we are able to find in this example
math and game binaries free compartments okay everything good but for some reason the engine binary
is not there then only in that case conan decides to expand the dependency
graph and add a new node to the graph expanding the tool requires
that basically means that this is a chicken and egg problem for knowing when we want to expand the
dependency graph to what the tool requires we first had to compute the package id
and then that means that all the tool requires cannot be part of the package id
okay so let’s see how content2 is going to improve overall all of these problems
i wanted to to also briefly introduce this because i’m i’m
really excited about cornell industrial currency is going to be a huge release and it is
improving many many things in conan conan one is already five years old we
have been doing a great job trying to maintain compatibility in all this time for all the conan one series
but there is already too many legacy too many bad defaults to make things that have changed in time
so now quantum 2 is going to change all those things we are going to have a a new user commands layer that you can
customize we are going to have a public python api something that our users are really looking forward we have a new
binary compatibility system new business integrations many many things the log files for example and the ci is going to
improve a lot as well so many things are going to change today i wanted i only
have time to talk about the dependency graph but i’m i’m so excited about what is coming coming with conan and tusito
so this is the new killer proposal the new amazing proposal of conan antucido is this
is we will handle the requires with ourselves requires math 1.0 this is the the major
advance in the graph mode of course the truth is there is a lot more there but as we are going to
propose also a smooth syntax compatibility to move forward from column one to conan
to zero we also have to be very careful with being able to write this this common
syntax between the two the two versions so this was important this really looks like exactly the same
but it is not the same the major difference here is now that the requires they have to introduce a concept that we
call requirement traits and a regular requires self requires math 1.0 by default it contains many
trades i’m going to start with the headers and the leaves trades the default requires assumes you are
requiring a library okay and by default when you are requiring a library directly you will use the headers of the
library and you will use the libraries of that line okay so by default header trade is true
and the trade is true and now we have the information
when when we are creating the the cma files to build engine we can use this and we say okay you
depend on the headers fine i’m going to add include directories property to the math
target so engine will find the headers in math and you
are using the ellipse of math fine i will add an interface link libraries property to the math target so engine
can link with the libraries in math if for some reason we go to the
we write explicitly in the recipe header sql false then the cmake file that is
being generated for us will omit will completely omit the include directories property or the math
target so the math headers will not be available while building engine it’s not something that you typically want to do
if you are of course linking with using math you will want the headers but this is how conan handles this trade this
header trades while you are building engine and exactly the same if the leaps
trade is false now the the cmec fat that is generated doesn’t include the link libraries so it will only provide you
the headers of map but it will not provide you the link the link libraries of of the math
libraries okay and then the very interesting
thing is how the transitive requirements are propagated so we
we define two different types of requirements of course first the direct explicit requirements are those that are
defined in recipes so we have the game that contains a direct explicit requires
to engine and we have engine that defines a requires explicit requires to
math okay those are direct requires and represented here but the thing is in the dependency graph
this is creating like kind of a virtual a virtual computer requires and is still
in a game as any depend on math game also depends on math somehow we need to we need to
understand how but game depends on math this is a transitive requires is equivalent if as
if we wrote self dot requires math one in the game recipe that we don’t but
it’s more or less the idea so
let’s now the the the key point of the new 2-0 model
is this how the trades are propagated from the in the graph from upstream to
downstream so i told you by default requires define true
and and sorry headers and leaves equal true okay and the question here is okay if
the explicit requires define this what are the traits how game is going to
depend on math and what are the values of the headers trade and the leaps trade that are
defined in this virtual virtual transitive requires from game to math
this is basically a function this is something the inputs that we have for this function
is the explicit requires under traits from from game to engine and engine to
math and also also engine and math packages themselves they have other
information there we will see later that this useful so we take these four things the two packages information and the two
required information and with that we can define this f function that will tell us hey the trade that you need to
to use here downstream is this
in this case we know it we learn from our users we know that by default if we have these input trades and we have this
these input packages we want the header straights to be false and we want the
library trades to be true because game needs to still need to link with the math methodic library
okay so that means hey i told you when you are we are using
this information when we are generating the files in this case that we are getting the math config cement file for
building game as the header trade from the transitive requirements from game to math is false
then we are not no longer generating the input directories for math and now we can build game and game will not have in
this include paths it will not have the math headers at all because the header sql falls straight remove it
okay say okay this is great it’s looking great right until someone tells you hey but hi i have this
i have in my engine package i have some classes there that uses matrix and i have a
include a directive here that is using things from math
that means you cannot remove the the headers from math in game they should be visible because the preprocessor is
going to need them there okay then you with this you have just
broken this say okay yeah don’t worry now we have this trade framework to define how how
requirements work so we introduce explicitly transitive headers trait and we now have a mechanism to
tell a engine if engine the engine package creator knows that he’s using the math headers
in the public api and those headers are are visible will be necessary in the
include paths in game it can defines a me engine i requires math
i request the headers i require the libraries myself but also my consumers are going to
report the headers because they are visible over my apis then if we get this trade
over the f function then we will it will resolve it will result in a in a transitive requires
from game to math that now contains the headers trade equal true
and the opposite if another uh scenario and some some developer there say okay
engine i’m a cert library i’m always bitter library i know
that the static library that i’m linking math doesn’t need to be propagated down to
the game so i can specify in my requires i can specify a trade transitive leaves equal
false okay and that is taken by the f function and it’s propagating down to the
consumer so headers it will be equal false that was our default that keeps
being the the right one and now leaps is false true that means a
in this scenario game doesn’t depend doesn’t need the math headers at all
and it doesn’t need the math libraries at all because it has in the middle this engine insert library that is completely
the decoupling decoupling the the thing and it will happen it will it will
basically those traits will result in a math config that is empty it doesn’t define the libraries or the headers from
game to math okay and actually as once the information is
there it can be even smarter it can say okay that’s that’s perfect i don’t need
the headrest of math i don’t need the libraries of math why fetch in the the math binary at all
i can now build game application and it will only need the binary of
engine it doesn’t it does no longer need the mathematical library in my computer at
all and this can result in huge time savings because if you have a 150 package
dependency graph and you are building an application that depends on just three cert libraries to build an application
doing an economy store whatever will fetch you just three binary packages out of the 150 and that is really saving you
a lot of time in ci and also in your developers machine and developers and developers time
more trades now we are loving these trades they are really powerful and yes they allow us more one of the things
that conan one didn’t manage at all was the runtime it didn’t have information
it was everything was built time it was only considering what you do with libraries at build time but
some of these libraries might be needed at runtime when you are building your application this was this functionality
was was improved adding the run the runtime trade in this case if we know for example that
math is always a certain library and we know it engine no knows know that or math a very common use case even if math
is static library but that package contains some xml files
something that is going to be loaded at runtime it happens i don’t know in computer vision pattern recognition many things
that depends on on a file there you actually need the math package to be
to be installed in your machine at runtime to get those files from the from the package
that information can be provided with the run with the run credit and this is just just another trade that
it doesn’t affect the bill at all so the math config that you make is not affected by run it’s still driven by the
headers and lips and lip straights but if you start defining run equal true trait then conan
2 will start generating files for you like this conan brand shell or bad depends on your system
which contains variables that you can activate and it will put things in the path so for example if math is a certain
library i’m going to know that because someone defined run equal true it will generate for you a script that contains
you can activate and it will contain the path to all the search libraries so every everything works without messing
without copying around the the certain libraries from one place to another or we have without having to manage
yourself this run trade is by default propagated
if someone defines engine defines a math is going to be needed at runtime then game is going to need math at
random it doesn’t it doesn’t depend how game depends on engine the the mapping is very very direct
someone requires something at runtime downstream you are going to need data and runtime as well
okay things have started to get a bit complicated right i’m i’ve explained a few traits so far
and headers sleeps run transitive headers and leaves and now i have my requirements
that i can start assigning values if i know it’s a static library i will do this if i know it’s a cert library maybe
i want to do this other thing if my headers are visible i want to to define those right so it seems that
my recipes can get complicated very quickly
because the truth is when we are dealing with different different different types of libraries
my trades will be different if i depend something depends on on an application then you want headers it is going to be
false you are not going to use the headers of an application libraries will be false as well but run will be true
because you want to use that at runtime okay but if you are depending on a cert library then your headers will be true
you want still want to link with the library your libraries will be true you want to link them and also you need the
artifacts at runtime so it will have the three traits to true and if it is static it will be the same but the run will be
equal false because you no longer need a static gallery at runtime and similar with a header library
so now we have this we have our requires and we need to start defining different
value for the trades if we are changing the the different the different types of library that we are
managing and this is where we welcome another addition to the model in current zero is
the package type okay the package type is just a class attribute that we can define
in our recipes so we can add in math package type equal static library and we can also add an engine if we want
package type equal static library okay something something like that the truth is that it’s not even
necessary always because we have other modeling conan like the options and third option in this case has has become
a standard at the factor standard every package out there is is using
option so if some package is defined in the share option the package type can be automatically reduced if served equal
true it will be a shared library it’s your equal fault will be a static library also header only option is
common out there for for libraries that can be built in in both header only mode and and compile mode
so if we detect that the header only option is there and header only option is true of course
then the packet style will be a header library so basically the values that that
package type can take at this moment are these applications library static library header library
library is like a kind of an abstract one it is telling a i want you to really
force that an option share will be will be defined otherwise i will error okay and then i will deduce based on the
option i will deduce if it’s a cert library or a static library but i want to make sure that conan knows that that
option should be defined that should be used to deduce the package stack then we can have uh build scripts to
model just a packet that contains some cmake stuff for example and python requires
so now how can we use this to to define the the traits
so basically something like that in practice this is looking this is the typical use case
uh how the package type helps so for many cases for example if math is
a cert library or a static library i only need the options to be defined and the requires
has to be like sorry about this yeah requires is as simple as this because
all the all the correct traits from these requires will be obtained will be automatically
deduced from the package type of math okay and only only if i want to change
something of course i always have the power i can go to the requires add the trade value that i want and that will be
respected but otherwise uh defining the auctions is good enough and only in those cases that conan
doesn’t have enough information then it’s good to define explicitly the package type for example for an application
it’s not that that easy to know that something is an application for for conan or for example if a package is
always a certain library always but it doesn’t contain the option because you cannot opt into certain library or
static library then you can explicitly help conor until a package type equals cert library but only in those cases you
need to specify that and once that is defined then in most cases the requires doesn’t
need to explicitly define all the trades values
okay good let’s continue with a couple of more trades
and now the bill trade okay we learned before in conan one about about two
requires and uh now i’m going to explain the bill the bill trade
so basically the tool required in current zero is defined with the build equal to trade okay adding that you know
requires that equal truth means a this requires leaves in the build context
okay this is the meaning of the build trade this is build context it will get the configuration from the build context
okay and um being in the build context
typically by default we’ll also get a run equal through this is typically a tool required it’s something you want to
run it’s an executable that you want to run in your build and as such it will not generate any cement file for
you that’s ridiculous actually it’s a package so it doesn’t make sense and but it will
put this this this trade run equal true and build equal true will put
the the path of cmake into a script that is called conan bill.
cell or back depending on the system i i told earlier we have now the two
things we have the conan run for runtime and we have the conan bill scripts for build time so this is something that
will be activated only when we are building but when we are running the final application cmake doesn’t maybe
doesn’t need to be in the path at all so that’s a different script
and the combination of trades run and build run equal false and build equal true
might make sense in other cases for example let’s say that we want to put a bunch of build scripts uh utilities
maybe maybe data some image files that we we have there that we only need them
at build time because they are going to be embedded in my binary for example
okay so then we don’t need them on runtime because we know that we are they are going to use at build time
and that means run equal false and then build equal true in that case that combination of trades
uh conan doesn’t generate any file it doesn’t know exactly what to do is because this is very specific to your
build and in this case the engine package can can get the access to my
script artifacts and files via dependencies mysql’s packet folder and if you are getting there the assets like
images you copy them you build them then you have the path and you do whatever you want in your build to use to use
those scripts or those assets and now the other problem that we were
seeing is fixed because now this is the dependency graph that we get in the phase one of the
graph evaluation okay if you recall that because basically two requires and requires are
not a different thing anymore they are exactly the same two requires is just an
alias it’s just uh some defaults it’s identical to define to define itself that requires clause with headers
false library false run through built through and other things so the thing is
when you are expanding the dependency graph you always get the full graph including the host context and the build
context is not an issue you get the graph of the of the recipes it doesn’t have an impact on on time okay but you always
have the full dependency graph always of all the recipes that are involved in building something in both host context
and build context and now if you have the whole dependency graph before computing the packages the package ids
for every package you can factor the bill requires into your package id and you and this is
now possible with 2-0 basically you can use the same the same rules and put cmake december
representation of the cma version into the configuration that is going to be hashed into the package id
okay and finally the visible trait the visible trait is basically
removing what we have learned so far in in by default visible is true that
means that a game has a virtual transitive requirements to
math always even if it removes if even if if this requires has headers equal
false uh libraries equal false even if it’s not using the artifacts game still contains
a transitive requires to math but
this is the mechanism that forces the complete removal of the transitive
requires if we put visible equal false that means a
nobody below me nobody below engine will even know if they will not
see that math package is there is introducing a full a full break
in the dependency graph so now the transitive requirements from game to map doesn’t exist anymore and of
course engine is fully responsible to know what they’re doing and know that they are cert library
embedding whatever but at least this is possible so in our
version conflict that we had before and we had this situation okay now it is possible to handle that
let’s see because the actual origin of the conflict of course is this in the previous case
when visibility is propagated we have a transitive requires from game to math 1
over the artificial intelligence left branch of the graph and we have a transitive requires from game to math to
zero that is created by the engine package in the middle and now we have these two requires of math that goes to
the game executable and in this point is where the conflict happens
okay so basically if we remove this transitive we make these transitive headers to disappear
then problem is gone the conflict doesn’t exist anymore and this is basically what is achieved
with a visual trade if both ai and engine packages they know that nothing
will conflict downstream they manage the embedding of the binaries they can declare the visible equal false trade
and then the the graph will not conflict anymore
looking good on time of course that doesn’t mean that in most situations we want to resolve the
conflict it’s a good thing to resolve the conflicts a good thing that both ai and engine use the same
upstream version and we actually close the diamond this might remove many runtime issues and everything so if we
want to force this into our dependency graph now we have two other traits
that we can use for this the first one is the override equal true that means a
over radical true means game doesn’t directly depend on math it will not introduce a direct dependency between
game and math what is telling the override is that a if someone upstream depends on math i
want them to depend on math to zero okay so this is basically forcing in this case the ai package that depends on
math one one zero is forcing ai package two depends on math to zero instead and closing the
clause in the graph of course that this this might create compilation problems in the artificial intelligence
package but that’s another issue developers should go there improve the api but at least from the conan point of
view you can create the graph that you want this is the graph that you want and you can define it
specifying the overall equal true and we have a different trait for when
you have explicit you have a different game for some reason directly depends on math it declare it declares oh something
wrong there so i no it’s fine this means game explicitly declares directly on math
but is to say hey i depend on math 2.0 and if some of my dependencies in this
case ai is dependent on math i want them to be forced into to see it as well
okay because the thing here is that we we are according to zero is more strict
it will it will throw more conflicts your way if there are actual russian
conflicts conan one was doing overrided overriding in most situations that you wouldn’t
expect because it was not explicit but it was still overriding now it will always conflict and you need to
explicitly force or overwrite if you want the dependency graph to be the way that you want
and finally the new package id this is the video that is going to save a ton of co2
emissions out there i really hope so so we we saw that the the corner one
package id mode is is getting the chamber representation and it depends on math one three two
and this is mapping as math one one dot y dot set
okay the the conan two trade model introduce a package id trade that allows you to
explicitly change that so engine can explicitly opt in to depend on math in patch mode
patch mode means a everything down to the patch will be put in the configuration file
and that will be factored in into the package id okay this is good conan one has something similar this is not a
really really a killer feature the really clear feature is this is that we have changed the default package at the
computation so even if recipes they don’t need to change anything in their package id this will happen
okay first we there are two modes there are two major modes the first one is the
non-embedding mode okay which is the case that happens when an application is dependent on a cert
library or a shared library depends on another certain library or a static library depends on another static
library the the assembly code will not be embedded in the consumer
assembly code okay in this case we are going to a minor to a minor mode that means that if
engine depends on math one three two by default we are we are going to put in the in the same representation we are
going to put 1.3.7 that means that by default every time
the miner is bumped you will need a different binary of engine but
if only the patch of math is bumped you will not need a new a new binary of engine that allows
if your developers know what they are doing they can do patches to math and this will not require rebuilding
engine downstream engine or other dependency dependencies in the
graph of course and this can save a lot of time if your team is disciplined and they are really managing versions
correctly patch releases of a package of a static library will not rebuild the whole world
okay and the other mode is the embed mode for all the other cases for example an
application linking a static library or a header library or a cert library
linking a static library or a header library or a static library link in a header library by default this is an
embedded mode the assembly of the of the dependency will be will be copied into
the assembly of the consumer okay in this case we have gone full mode that
means hey you take the whole reference of the binary dependency which is the the version one three
two full the revision which is an automatic version that canon is adding which is
the hash of the contents you change a comma in your source code and even if you you’re not bumping the
version conan will compute a new revision for you so that means that changing something in the source code of
math okay will be will result in a new binary of engine which is exactly what you want hey
something change in math you need to rebuild engine because the binary will always be different
okay and even the package id of math is factored in into the into the equation
why because in the case we have the math packets with float accuracy and double
accuracy okay if you change that that change the package id of math you also need to rebuild you
need a different binary of the of the downstream consumer application so absolutely everything of the binary of
the dependency is factored in into the into the downstream consumer
and of course one of the things that we are trying to be very careful with column 2 is being flexible if you want to change any
of these defaults now you have full control you can individually change the embed mode the
non-embed node the python mode you have full control over the over these modes
that you can use okay we are good on time i will run a very quick demo
let’s let’s try it here is the phone good enough
okay so let’s go to the game uh first thing i’m going
to do is i’m going to run conan graph info
just to to show you the dependency graph format by the way this is conan 2 0
alpha 6 which is already released okay you can go there and you can start trying yourself
so i’m going to write the html format into my graph html
and i’m going to show you this this graph html and you will see that this is basically
the example that we have been seeing today okay we have the game application that depends on engine that in turn
depends on artificial intelligence and graphics well it’s a bit different but it’s more or less the the same idea that
in turns depend on mathly here i’m using matlib because when i play the demo uh
math.h is a different thing and i was getting system conflict so i renamed it to map to to avoid this
so this is my dependency graph i can create my game package with this syntax for
example version 1.0 and it will be basically fetching my
dependencies here we can see that we depend on on math leap
okay this is dependency to matlab when the my cmake code will be sorry
about this one second okay my c make will be telling a this it
created the matlab target for us so it’s actually getting the target from the math
transitive dependency and it is executing the game which is printing
one message for every every library there in this case it is executing this code it is written in the math library
okay and it’s been actually been called twice because it’s been called by the diamond uh dependencies so i’m
actually using the mat library in my game right and i know this is a very
small step uh but i’m really excited about this now i
can go to my game source code here i can introduce
the matlab.h that actually exists there
and if i do a conan creed again then my mechanism that the trades propagation
doesn’t make the headers for math available to game at all and this is what developers get they
are they can no longer introduce a hidden hidden header inclusion in the
game from math if they don’t explicitly define a requires from game to math
okay so let’s remove it
let’s make sure that this this works and the other thing that i’m going to demo now
is also a very very simple thing but it’s the thing that i’m really very excited
here i created my game package right so i can now let’s install it let’s install
requires equal game 1.0
and actually the thing that i’m most excited about is this one you see here the requirements
artificial intelligence skip i no longer need this binary my computer
to it’s all game engine package skip graphics package skip
package skip so when i’m installing game in this case i only need to fetch
the game executable binary and i will not even download all the other packages
from the servers if they are not needed at all okay this is the thing that will result in saving tons of time
for things not only this but this will also be applied for building
so conan 20 will be building way less things if you use versioning correctly
while not skipping changes if something changes that needs to be rebuilt it will guarantee that things get rebuilt
cool we are on time perfect about to finish
okay let’s jump to the conclusions so conan 2 has defined a very powerful
mechanism its base is built on three things first the requirement traits we
have seen a bunch of them headers library transitivity visibility build run and so there are there are others by
the way there is the test one that i haven’t talked about it there is the options for the first time you can
explicitly tell a this package here has these options okay
um that that wasn’t possible in column one to cope with the with the complexities
that could arise from only the required trades the package types
solve the 90 something percent of the cases that
will not add extra complexity to your recipes your requires will still be plain requires and just managing the
package types knowing that something is a static library or a cert library or something really brings the power the knowledge of
what conan should be doing in order to to do the right things and the mechanism of how how these
requirement traits are propagated from upstream to downstream uh taking into
account the package types is is the proposal here and with this proposal we
we can do amazing things we have seen on demo that we can hide the headers
visibility we can really avoid the linkage propagating the leak catch through shared libraries if
it’s not needed we can implement private dependencies very simple we can do
things like bootstrapping now we cannot have a cmake 320 package to require
cma319 for example if we want okay the much better package id will be
saving tons of time of developers and nci the tool requirements and executables is
managed in a better way conflicts and resolution is is also necessary and the smart skipping of of
dependencies but the real power of all of these is that for most cases nothing changes
for for developers so we have something that in many cases it will look like this
just very simple standard content recipes with options define it in most of the
cases when it is not necessary it will it will use the package tab explicitly just one attribute to help conan to
identify what is the package type and it will be this simple in most
situations but for really weird things that happens out there and i can tell you they happen with with c and c plus
plus building they happen out there you will be able to implement them you will be able to handle them because now
the the controls that you have over the dependency graph with the trades is much much bigger
okay so i wanted to uh end today saying thank you first for for
being there and um some pointers if you want more information about
anything the first entry point is conan.io of course that’s very simple
for uh you can go from everywhere from there to the documentation there is already
2.0 beta documentation out there okay you just need to go to the docs and go
to the versions and click on the two zero alpha actually and
if you have we have a training for conan one okay it’s good if you don’t know anything about conan it’s good go there
for training is a free training video format self-paced is good to get to know about conan one we will duplicate that
for conan two once conan two is is ga if you have any questions the best resource resources github you can submit
a github issue for a question it doesn’t need to be a bad report so please if you have any question any issue any problem
open a github and follow us in in social by the way i wanted to say what have
been showing to today is conan 2-0 alpha okay the plan is the following we plan to go
uh to zero alpha seven the next month that will be the last alpha and then we will
move to beta we plan to do at least four betas is the expected team and then
we’ll move to ga but beta is being there very very very close um i will be later in in gather town so
for the online people if you want to to try to reach me i try to be to be there
if not you can ping me you can reach me in twitter dlr los ala i’m open you have
any question not only today other day and you want to talk this about and
discuss anything about what we haven’t seen today please ping me and i’d love to to connect with you and talk to you
and and everything and now i guess uh time for some questions thank you
you