What’s new in Conan 2.0 @ Meeting C++ 2022

hello everyone I’m Christopher the Conan developer Advocate over here at jfrog and I’m thrilled to get to present to
you what’s new in Conan 2.0 my involvement in koning began almost three years ago as an open source
contributor and Enterprise user and I can assure you you’ll be just as excited as I am when you learn what’s coming
everything is new it’s been five years since the 1.0 release without any breaking changes
about 60 of the code is completely new from scratch all new and this is going
to really bring some huge improvements to Conan twenty percent of the code now is backwards and that’s what’s the most
important part for you one of the main principles that guided the 2.0 work was to provide a compatible syntax in the
1.x to help smooth that transition and adoption to 2.0 another huge principle was feedback from
users you can start to see just how vibrant and active the Conan Community is with just this Snapshot from the cpls
plus language slack you can see uh Khan is ranked number two in the
number of members who posted message and it’s the top three for how many message there actually are
the social engagement even looking at usage Conan has gotten nearly 700 000
downloads from Pi Pi in the last month we’ve also gained the Pi Pi critical
project designation which marks Conan is one of the top one percent projects even in the python ecosystem
and that’s not the whole picture this is just one of the installers that we offer there’s also an MSI for Windows
users Brew for Mac Debian Etc and you can check the full list on the website
being an open source project we can also look at the contributions over the past few years you can see in 2019 there is
roughly 750 pull requests to Conan which is like the main client portion itself and that number has remained strong over
the years but what I am proud to share is the number of pull requests in Conan Center this is where Conan users get to
contribute open source projects they need to make them available for everybody there has been an astonishing amount of
growth and as a community reviewer this graph doesn’t show and capture the enthusiasm that I see from contributors
every day in 2021 we saw well over 3 000 pull requests and this year we’re set to make
over 4 000 already and this is just so far
we’re seeing about 150 200 pull requests every single week so I think we should hit that 5000 mark this year and that
should be pretty freaking sweet
contributions are amazing and helpful uh but we also need to talk about support users have questions issues bug reports
and we see around 2 000 GitHub issues per year we spent over 100 hours in
video calls with customers and we have almost nearly daily requests that are message directly to us asking for help
and guidance you can see in the graph we have a growing number of jfrog artifactory
servers that use Conan in production as well and these are just the servers with Telemetry enabled and that are not
behind a firewall we have these amazing users actively
using Conan in production as part as their daily workflow these Conan experts who happen to work for companies like
TomTom RTI and Bloomberg just to name a few the Conan tribe has been invaluable
in helping us iron out the details for the Conan 2.0 launch their practical expertise turned into feedback for
development for these new features that I’ll be sharing today I’m going to be going over the
four lessons we have learned from the C plus plus ecosystem and how if we’ve Incorporated this into Conan 2.0 our
first lesson is learning to fly the initial takeoff point for Conan is really understanding the dependency
graphs in the C plus ecosystem we need to start off with a Conan file
or what we like to call our recipe this is the instructions for getting the source files building the project and
packaging it in a reusable way you can also set attributes for the common file class like the name and
version in this example so with this math project we are using the Conan create command to make a binary package
which we will typically include the library archives and header files that we would expect to have if we built
another project on top of that surprise the requirements have changed and now we’re trying to make a game that
requires an engine which uses the math Library which we just saw you can see these dependencies in the
cone files requires attribute now let’s say you were looking you were
working locally with Conan 1.54 you run the cone and install command and that cmake depth generator you see would have
outputted the configuration files for finding all the dependencies in the graph even the transitive ones
but what would you but what would be your expectations if the engine library was built to share if the engine
correctly encapsulated the math Library we would expect the game to no longer require math binaries
this sadly just isn’t possible in Kona 1.x currently you would have to get all
the transmitter dependencies even though linking to a shared engine would already include the math symbols
more so because the math libraries and header files were exposed developers could erroneously use them from the game
when trying to without explicitly having a dependency on them so why is this problem we could just not
always link statically well unfortunately the answer ends up being no about 25 of the time uh some of the
cases we often come across are long length times accidentally embedding unused copies bloating the binary size
just to name a few so this is the Conan 2.0 proposal if
this looks familiar it’s because it’s the same as konig1.x and you could run this without any error in Conan 1.54
yet we have added in 2.0 are requirement traits you can now explicitly set if the
requirements include headers libraries by default both are true and this opens
the door to possibilities so when both attributes are true the
cmake config files generated by Conan includes both the link libraries and the
include directory definitions for the math Library you can see them here in this example
if we were to explicitly set headers to false this would mean the engine project does not need to include any headers the
Conan generators would only populate the link libraries omitting the include
directories this way our engine would not be able to access the header files of math
on the other hand if we were to set Libs to false and leave headers to True our engine project would only have access to
the other files through the generated configuration files with this new informational flexibility
we should be able to gain access to a better model graph when looking at our graph in Conan 2.0
we can see there are direct and transitive dependencies now our engine recipe has a direct requirement for math
our game still has the direct dependency on game which also includes the transitive dependency for math
now on our engine we can finally change it to build only as a shared Library we
could explicitly tell Conan our requirement to the math Library will not be exposed by setting the transitive
Libs property to false this will allow Conan to build our dependency graph such that game will now have the transitive
requirement of math with headers and libraries set to false meaning the client will not expose the include
directories or the link libraries when generating our find config files this will prevent the library from being
passed at link time avoiding any penalty to the size which would have been incurred with Conan 1.x also if
developers were to use math in game they will now need to explicitly add a direct requirement to math
in order to simplify this and to provide some reasonable defaults we’ve also introduced the concept of package types
with Conan 2.0 this is one of those features that was directly influenced by the Conan tribe
there are three package types which we have defined they are static Library shared library
and application for the two Library types they can be determined by using the predefined
shared option depending on the value being true or false Conan will be able to generate and correctly pass the
requirement rates now I’ll switch over to doing a quick demonstration of this
so before we get into our very first demo for this evening I will be going to
explain one new Conan 2.0 feature that I’m taking advantage of so you might notice here I have this conan.rc file
and this is a new feature which allows uh each project to specify explicitly
the Conan home folder so previously in counting 1.x you could only change this
with environment variables but you can see here I am going to do this relative to my current directory and I’m going to
have a Conan to home folder which you can see is already here and there are
already some settings and profiles defined so if we go over to my terminal we can
type Conan version and you’ll see here I am running the
latest beta 5. clear now if we
so as you can note we are already in the game subdirectory so what we’ll do is a
Conan install and we will specify our option as engine
shared equals true now if we go up just a little bit you can
see here it’s Computing the necessary packages and engine was found in the
cache I had pre-populated those but the most important one here is math skit
so this is how Conan is showing you that it knows that the math requirement was not required
so if we go into our build generators folder and we look down the list you’ll
notice we have only the engine configuration targets which are required to load with the cmake fine package
which is this guy over here now
if we go back and we do this again
without our shared option so just with engine as static that is
our default you’ll notice it’s pulling up math from
the cache this time because it knows it needs this dependency
we go into our build folder and we go into our math
cmake Target release you can see there’s all the things that are populated and here if we look at the
paths that are being populated you’ll notice here we have the library directories which is being populated
here but the include directories are empty because
the Assumption here is that as a shared Library it should be encapsulating all
the dependencies and only the link libraries are required
with the improvements to the dependency graph in column 2.0 we can now Express correct linkage requirements correct
header visibility and possibly even have hidden or private dependencies and this is just scratching the surface if you
want to learn more I highly recommend Diego’s talk focusing on this and package mode improvements at the Accu
conference this year you can find that one on YouTube if you’re interested our second lesson is going to be
building a dam and building a dam is just the beginning you could also need to generate electricity and deliver it
to Consumers it’s no secret humans don’t like change and we have conversations that go like
this they I want the lips from cone independencies in my project folder us
no need for it you can just use the lids from the cache they but the dependencies should be in
the project not really many other package managers like Maven pip do not
put the dependencies in the project folder but it’s easy they say they don’t need why don’t you just put the dependency
Libs in my folder us it’s easy that they’ll conflict and have different versions The binaries might not match
there’s no metadata no synchronization and then you’re taking up double this disk space and they finally retort I
want the libs in my project folder they should be there yay um
all jokes aside there are some pretty valid use cases where something like this could be really beneficial
and we’ll be introducing deployers with this new Conan 2.0
there will be two built-in deployers full and direct but this is just the start deploys are extensible you can
write your own install them using the same Conan config install commands you’re probably already familiar with
just passing a URL or path these custom deployers only need to
define a deploy method and have access to the Conan file so you can access exactly what you need from the package
or even go over the entire dependency graph I will be demonstrating this here with
you now
so far the players demo uh we are going to pick up right where we left off for the
graph demo so I will clear my screen you can see we’re still working from the game
directory and let’s say our local game developer wanted to do that full eject style
um workflow
he could do Conan install and then he can do deploy
full
so that would have done our install so now if we go into our game folder we not
only have a build folder but we also have a host folder so here is our
generators which should look very familiar to the last demo but we also
have our host engine and this has all of our things so we have our
name version uh debug type and our architecture you can see we have an
include directory build directory as well as the info files and you can see math has a very similar
structure now if we go to our generators and we pick for instance the
x86 for our engine you can now see it’s pointing to our user Conan what’s new
Game host engine and this is the actual directory in my local structure so for
those who wanted to have Conan run and work with the
local directory and have everything locally in their own file they’re now able to do so
now having a deployer that just ejects all the content in your current directory is great
but I think one of the more interesting use cases is how we can use the players to create a installer now
if we go over to visual studio again and we look I have a deployers folder
that has a DOT Conan folder now this is my own naming convention and I think it
goes nicely with the dot Conan RC I have my extensions deploy and I have a
runtime zip deployer now you can see here this will search in
different directories and it’ll look for our shared runtime and executables and
it’ll package them in a zip file and give us that as a Target
so these are very easy to work with if we go into our deployer
folder we can show off this cute little demo by doing Conan
config install dot Conan because that’s the director we want to install and
you’ll notice here it’ll say copying runtime zip deploy to our Conan home directory extensions
deploy so now if I do Conan
install and we’re going to do a virtual graph so we’re going to explicitly require our game 1.0 and we’re again
going to do our engine shared now we’re going to specify deploy and then same
file name that we gave our deployer
so you’ll notice it would have done our lookup so we have our game we have our
engine and math which in this case was skipped because engine was shared
so if I go back to visual studio and we open our deployers now you can see a lot
new files so these are the regular files that you’ll see from our environments because we’re using virtual build in
Virtual run environments those are generated here but most importantly we have our zip file
now if we reveal and find her and we open this up
we can go and check and we have both our game executable and our shared copy of
our engine and you should be able to run these and it’ll work without any hitch
so that should give you a taste of what deployers can do there are a flexible way to extract artifacts from the cache
in addition they allow automation of post Conan tasks like creating installers
there are also not part of the recipe unlike the previous uh Kona 1.x Imports
mechanics each recipe needed to have its own import method with the logic implemented specifically for it
the new feature will enable scale ability deployment across many packages the user
can provide their customized way to extract exactly what they need and they
can and these deployers can be installed with the Conan config install command as a lot of the other options
the third lesson makes me feel attached guilty it’s death by a thousand bites
and this is our expression for the burden directly implementing specific use cases and workflows that we’ve been
asked over the years there was a slack message on the community Channel about one of the more
common helpers files.get which will which is used to download and verify archives so the source files of
different projects the author was asking why there was just so many arguments for the spawn function which should have been relatively easy
he discovered an old GitHub issue which made it clear how we got so many arcs clicking on the link and following it it
was an issue I created as an Enterprise user trying to meet my own specific needs of of course it had to be one of
mine the solution to this problem was quite simple actually which is usually a good
sign right just allow users to help themselves with Conan 2.0 we have worked
hard not to provide just a tool but also a framework a framework which allows users to build
solutions to tailored to their needs we are launching with several supported
plugins in today’s talk I will focus on the profile command wrapper and package
lining plugins the profile plugin allows checking validating and manipulating a profiles
in order to understand the need for such a feature though we’ll need to look back some of the challenges developers face
today let’s build our game example for the local developer system given the profile
of Linux with GCC version 8 and they’re using C plus 20.
but let’s say the team builds tests and ships the game for multiple different compilers
we want to be able to move forward and update the compiler versions regularly so we’re more agile
so we want to have a profile that allows us to build a certain set of compiler versions and we can do this with profile
templates since Conan 1.38 this has been supported and uses Jinja to render these templates
this enables an easy workflow where we only have one profile template which takes in several environment variables
you can note that OS get EnV call on our profile now this will allow us to dynamically change
the value to match the development environment that we’re using so let’s say our developer needs to work
on an older compiler and changes his environment variable in his terminal he punches in my GCC
version equals six so the profile pass the cone would then be using GCC version 6 with C plus 20.
now this should not actually work C plus plus 20 support was not added until GCC
version a so this could result in compiler errors about unknown Flags
so we do so what can we do to ensure that these profiles being used are valid
I’m sure you can imagine other situations where only a subset of the settings in Conan should be used for a
particular project and it’d be beneficial to make sure unsupported settings are not being passed for local
development or testing so let me introduce the profile plugin
similarly to the deployers we saw earlier these are also installed in the extensions folder using the Conan config
install command however this uses a different method name profile plugin and
pass the complete profile to the method before loading the graph there are a few built-in checks done by
this plugin one to ensure the CPP sdd is valid given the compiler version the
example we just went over in addition there is a second check to match the msvc compiler runtime to the build type
being used when it’s absent from the profile and settings so we are recreating our example on my
MacBook I have my normal profile with apple coin 14 using C plus plus 23.
however with my profile plugin if I specify compiler version equal 12 I
will get an easy to work with error message the provided compiler CPP sdd
equal 23 requires an apple client greater than or equal to 13 but version 12 was provided and this allows me to
quickly correctly issue I know exactly which setting is wrong and what value it needs to be
now changing gears not only can we validate settings but we can also fill in the blanks for
Windows development we often don’t think about the different runtimes we are using unless we have to that is
the typical default for compilers and Tool chains is to match releasing the bug compilation with the same runtime
optimization in other words building an optimized application will
link against an optimized system runtime it is however possible to mix them you
can build a debug application using a release runtime and there are various benefits to doing something like this
this is what the second part of the build profile plugin will do when the settings for the compiler runtime is not
set it will be assigned the same optimization for the build type that means changing from debug to release for
the build type will also change the target system runtime that will be linked against
so just to recap the profile plugin allows you to control the contents in a profile and there are two built-in
features of the profile plugin the first isn’t to ensure that the CPP STD is
supported by the compiler version which is an example of just validating the profile as coherent
the second is to ensure the compiler runtime matches the build type when it is not specified this serves as an
example of manipulating the profiles which is the one we just covered
of course these behaviors are completely optional unextensible you can modify the default rules to the specification or
build your own to enforce workplace or Project Specific conventions constraints and compliance
this is one plugin I am most excited to see what users will be able to achieve what problems it might help them
overcome the plugin I will be sharing is the command wrapper this was intended to
support external tools like incredibuild which need to be added to certain commands
the command wrapper plugin will be called each time self.run is called and
this is the entry point for Conan invoking things on the shell or command prompt you’ll be given the exact same
command line before Conan runs it and you’ll be able to manipulate it however you like
for a theoretical example let’s say you use C cash to speed up build times with modern cmake you can set a compiler
launcher which is prepended to the compiler invocation so here each time Conan calls cmake we
can make sure to set variables for cmx compiler launcher as ccash so when cmake
is called it will always know to use this tool this is far more convenient of a solution than trying to modify each
and every single project or multiple projects to change to use ccash in their
build scripts or even adding options to support doing so it can be applied to the entire dependency graph using this
plugin without modifying any of the underlying projects
arguably the most important plugin I will be introducing is package signing which I believe has the most chance to
bring Innovation to the C plus plus ecosystem supply chain security has been an
increasingly important topic it’s important to have traceability given condition distributes binaries
it’s in the right position to make a huge impact this is a question we got very often from prospective users typically most
companies already have some Implement I’ve already implemented this in some internal mechanism and this plugin will
allow them to seamlessly add it into their accounting packages that being said open source is extremely
important and we’re also working on some very exciting Integrations which will be coming shortly after the Conan 2.0
launch so stay tuned this diagram should look familiar by now
as the other extensions of plug-in you simply need to provide one file which can be installed as before the package
signing needs two methods a sign and a verify method they are called for a
single reference think project and have a few arguments which I will explain
the important thing to note here is when this plugin is called this is specifically for one the client is
interacting with a remote or registry some package hosting server external to the local client’s cache simply creating
a package will not sign in if you build it locally yourself hopefully you trust it
so our two plug-in methods are called when uploading packages to our Mount
this is when the sign method is called and given a reference an artifact path you’ll be able to perform the necessary
calculations and save the output which is most likely a signature to the signing metadata folder in the cache
when consuming packages for instance when a Conan install is called when the
packages are not in the local cache and a match is found in a remote the verify
method is called a bar plugin which will be called with the same parameters as before
this will allow us to compute the signature with what was downloaded and verify against that which was previously
captured in the cache and I’ll be demoing this for you shortly it’s worth mentioning because this is a
plot this is in Python you can very easily call a program to upload the signature to a central server or
download the new ones to verify with
so for the package signing demo we’re going to start off by looking at
Visual Studio code and we have a package signing demo and
we have our DOT cone folder we have our extensions plugins sign and
our sign.ui and this one simply just goes through all the files in the
artifact folder and then creates a signature for them and the verify does the does the same thing so it’s going
through looking for the signature and then making sure the contents match however this one has one extra detail
which is a extra remote.json so this is where we’re going to be
adding and uploading our packages so that we can then pull them down to verify and this here is in my uh super
frog instance of artifactory and there’s a bunch of them already in here
so if we grab our terminal and we go into our package sign-in folder we can do a Conan
config install dot Conan
it’s done we can then do a Conan upload
now we’re specifying our super frog which is the remote we want to do minus
C is just to skip the confirmations and minus f is because I have a bunch of junk in there
Force so you’ll notice it’s signing a bunch of different artifacts
and printing the references and then it’s uploading them
now if we clear we can do a Conan remove
everything yes
minus f there we go flush that out so now there’s no packages locally
now if we go back up and we do Conan create game and build missing
now it’ll start looking for the packages and you can see it doing the verify
so for some of the packages like engine which it was able to find with super
frog it was able to download and verify those signature contents
with these new plugins for managing profiles commands and signing users will have a real framework to build on top of
I do not share custom commands or the new python API coming in 2.0 I highly recommend watching Diego’s CPP contact
for those highlights sadly it’s not yet available so you’ll have to subscribe to that YouTube channel
we have also been working on packaging signing for Conan Center and we have some really exciting Integrations we’ve
been working on you might want to follow us on social media the last lesson we will be covering
today is repeating yourself how or how not to be repeating yourself depending
on your use case you can choose this can be extremely beneficial as a way to achieve reproducibility repeating the
same build tomorrow and in 10 years from now it’s something many of our clients have a business requirement a legal
requirement an obligation to do but also reusing existing valid binaries correctly capturing the binary
configuration and knowing binary compatibility between packages is often just as necessary
when we hear about binary compatibility we often get mixed up because it can mean different things to different
people so let me explain what it means from a Conan perspective we have two of our projects engine which
requires math each have their own package package has a unique ID that is
generated from the settings options and the graph it was created from regardless of the compatibility we can pin a binary
to exactly the one configuration it comes from if another package exists for a
different configuration its package ID must be different or else Cohen would not be able to distinguish
those two configurations now if these two packages are compatible that means Downstream consumers of those
projects are able to fall back to a package which is not an exact match for
the requested settings this means Conan can look for other package IDs when the exact one does not
exist so what does so what do these settings and options look like you can see here
an example from the recipe of fmt which is in Conan Center we have a collection
of settings and options and we often refer to this as the binary model of a package
any change to one of these will cause a different package ID to be assigned the
relationship between options is also important the header only and shared options are mutually exclusive and
that’s modeled in this recipe but there’s more Conan also has different package modes which can change
how the package IDs are computed so there’s a lot of nuance that is specific to each graph
but let’s take a step back and talk more generally binary compatibility in Conan means different package IDs of different
dependencies can be used to create the same package ID of the target project so
different input gives the same output if we reuse our math and engine projects
we should be able to demonstrate this idea we have two math packages 6A and two
three and we need to build our Engine with an ID of 79. if the required
package for a profile is 6a for math that package will be used
since it’s an exact match now if that was not the case let’s say both packages used a different minor
release of the compiler so GCC 7.2 and GCC 7.3 respectively and now let’s try
building Engine with GCC 7.4 if our compatibility policy is to look in
reverse reverse order of our minor releases so of the compiler so 7.4 7.3 7.2 then the
math 2 3 would be the selected to satisfy our graph
so since that example is a bit long let me give you a real world example instead one of the long-standing issues I’ve
personally been very interested in is the management of lib C versions with Conan where there is a backwards
compatibility if you build with g-lib C 3.28 you can reuse
3.23 and this would still output a 3.28 g-lib c ABI regardless of which versions
you are building your dependencies with so hopefully that gives you a good idea
with the compatibility definition understood let me introduce yet another plugin I’m sure
you are starting to see kona’s more of a framework for managing your dependencies now we have compatibility which is located
in the extensions folder similar to the plugin profile there is only one binary
compatibility that will be provided by default packages can fall back to any
CPP SD value as we saw with the deployers the plugin gets a cone file which means you can
access all of the dependencies settings Etc it’s also worth noting this acts as a
migration helper between 1.x and 2.0 the CPP SD value must be specified in 2.4 uh
2.0 where it was not required in 1.x so the default behaviors will be similar in
practice with this default built-in compatibility and this is what I’m going to demonstrate for you now
for our final demo we’re going to be going over binary compatibility with CPP
STD that is built into Conan 2.0 by default so if we do Conan
profile show minus PR
default you’ll notice we are going to be running with CVP STD 11.
and that’s going to be our starting point for our first few projects that we’re going to build we check in my cache I don’t have any
packages yet so we’re starting with a clean system
so the first thing we’re going to want to do is build math so we’re going to do colon creates math
version 1.0 build missing and we’re going to specify our default profile
this guy over here which is the one we just saw which is our cbbsd11 let me run this
beautiful and then we can do it again but this time we’ll do it for engine
you can see it compiles if we flick over to visual studio to
look at our cache we can look at our packages
this one is in 14e so if we go to 14e
package cone and info dot text you can see here our settings have our cppsd 11 and this
has the requirement on math which is very cool to see now if we want to do Conan create
game this time specifying our compiler as CPP
14 over here and we run this
if you notice we’re only going to see one build
arguments and if we scroll up a little bit more
we have Computing dependency graph you can see here all the things were in the cache
and here is Computing necessary packages you can see here for math it’s checking
11 compatible configurations and you can see here it started with um C plus 98
11 and then it found a matching binary so it stopped looking
and then here we have the same thing again but for engine and it stopped it did not find
the exact match but it did find a compatible one which is the one for cppsd11 over here
so our three two and then we have the same one here
now if I do a Conan remove everything
and then we build math for default
do our engine again but this time with 14
I only do minus s compiler
cppsdd equals 14.
it’s built now we can do cone and create game with our cpps to D14 over here
and this time if we go to Computing necessary packages
you’ll notice only math is being searched for now because game already had the
uh because engine already had the exact match so it did not need to do any searching
so what have we seen so far why Kona 2.0 your feedback
we’ve been listening to all the feedback you’ve been and have worked tirelessly to come up with better Solutions
some of the key features to do this the new graph resolution model we’ve added requirement trades to better model how
binaries are captured we’ve we’ve added applications type to provide good
defaults for the graph to use we looked at deployers you can finally eject everything into your current
directory if you so wish and you can create installers in a fashion that is much more scalable
I’ve also introduced countless new plugins for you to explore we have the profile plugin to validate or even
change profiles command wrapper so you can inject anything into the command
line which should be much easier to deploy new tooling instead of modifying individual project build scripts we have
package signing you’ll finally have traceability throughout your build on which artifacts are being pulled in
lastly there was the compatibility plugin this will allow you to have reproducible builds with exact settings
while opting into managing binary compatibility if you know a different setting could be used in place of a
matching one and this is only scratching the surface we have many other improvements with
Conan 2.0 some of my favorites are multi-revision caches lock files and
custom commands which will drastically improve my CI CD pipelines for my projects and I hope you’re just as
excited as I am for the pending 2.0 if you want to try it out yourself we
are at the fifth beta uh installation instructions are very easy just pip install
as for when it’ll be out we suspect there will be a sixth beta before we hit a general availability release so stay
tuned for that cheers everyone