The road to Terraform with JFrog

Video Transcript

Good morning, good afternoon good evening everyone depending on where you’re
joining us from today i’m Sean Pratt your friendly marketing manager and i’m going to be your moderator for today’s webinar the topic we have for you is: behind the curtain the road to terraform with Jfrog and we’re super excited to share with you how jfrog uses terraform and how we arrived at our new terraform support in Artifactory.
Just a few housekeeping items before we get started this webinar is being
recorded so don’t worry if you miss anything as we’ll be sending out a recording
within 24 hours.
If you have any questions regarding this webinar or further questions post event
please email webinars jfrog.com and we’ll get back to you
also as a reminder please use the q a chat window at the very bottom of your screen we have our experts on the call here today and they will be happy to answer your questions so don’t be shy feel free to put them in as you have them and we will answer them promptly so i’m going to go ahead and kick things off today by introducing our speakers.
Today we’re really fortunate to have with us amit danielle who’s our devops
team lead at jfrog and a software engineer at jfrog so without further ado let’s go ahead and get started amit why don’t you take it away thanks sean hi everyone great having you here with us today have a question for the audience how much time are you wasting initializing your terraform environments, I can share with you that we used to spend hours initializing our telephone environments and that’s exactly what we are going to discuss today we are going to present what we change and how it’s going to improve your daily work before we jump into the meat of the
presentation.
I just wanted to do a super short introduction to jfrog for those that are not familiar with us our mission is to make every software created successful by providing the best solution for fast and secure continuous releases we do that with jfrog devops platform which includes the different application you see here and today we will spend some time talking about artifactory in particular and lastly a little bit about our organization.
We served some of the largest companies in the world across nearly every vertical and there are over 1 000 frogs globally with offices across North America, Europe and asia let me give you a short introduction on jfrog infrastructure layer we have around 50 kubernetes clusters across three public cloud providers aws azure and gcp jfrog is running the workload in 40 different regions devops infra manage around 130 telephone workspaces and the scale is massive as you can see in other words we are heavy terraform users.
Today i’m here with you to share my experience and story since i joined JFrog what i did with terraform and what we learned through the process let’s talk about five main steps that improved our terraform usage aligned terror from current infrastructure to be managed only using infrastructure escort two major branches for development
workspaces and production workspaces and telephone flow where we can change
the module without breaking workspaces one master branch for the models
repository we tags per module adopt an orchestrator over telephone we
need to run all our telephone workspaces automatically every day to detect the drifts
split the permissions between different groups that consume our telephone code
run logical code after every step before or after the plan before or after the
apply etc and of course modules modules decoupling obviously we face some challenges managing the entire jfk sas infrastructure layer without proper infrastructure as code configure is almost impossible naming convention security standards delivery time without infrastructure as code we can’t monitor production changes misconfiguration in every resource massive changes etc it’s super how to find ruddy the change in a cloud resource without infrastructure as good not only did we have manual work we also faced terraform code spread over multiple branches before moving to use telephone with two major branches we found ourselves with a
branch per feature environment this causes a lot of drifts we couldn’t understand which configuration was running on every Environment the solution for that was splitting terraform code into two main repositories: one for the live code and one for the terraform modules for the live code all the development phases were against the dev branch moving the terraform modules to a dedicated repository with a single
branch and every module gets its own tag and the live code will consume it
in this case you can manage exactly which model will run on every workspace the solution to reduce the dependencies we had for every telephone workspace is modules decoupling now i have a question for the audience about that why can’t we add a database if we have drift in our kubernetes cluster workspace splitting our telephone modules into four main models networking layers that will contain the vpc subnet not gateway etc kubernetes layer using the networking resources as a data
databases layer and object storage this separation gave us the option to continue walking on a layer without fixing drift if any in another module make sure that we can build our infrastructure with no dependencies.
I have another question for the audience how many times have you built a resource
and you find that this resource has security vulnerabilities and now you can’t fix it here comes the magic… think that you need to create a database and the database instance is ready and we are happy customers start to use this database and they are also happy but now the security team found that this db is not encrypted
and now you need to create a new database and migrate all the workloads
to this db right now after the telephone plan we are running an open source application for infrastructure that’s called security vulnerability scanning and we are seeing the vulnerabilities before the resource is being created with this approach j4 creates a resource with no vulnerabilities also when you are scanning your plan and
State you can always see current vulnerabilities and mitigations
if there is one thing that i want that you will take from this webinar part is that we couldn’t able to get to this solution without a strong collaboration between devops and rnd in jfrob we prefer to drink our own whiskey and for that we partner the
entire development phase with rnd about the telephone features in artifactory
that helps us to achieve this amazing milestone and to continue talking about this process. I want to end it all over to barack.
Thank you, it was a nice story and a great lesson now let’s see how things are really happening.
Let’s start talking about the jfrog solution for terraform artifactory has local remote and virtual repositories implementation for the terraform registry all of them support
both modules and providers let’s elaborate about the benefits of
each repository type the remote repository which will most probably point to the artifactory to the hashicorp official terraform registry saves your outside network traffic gives you more stability and faster download speed the local repository pushes you to the transition from source control modules to binaries devops used to resolve their modules directly from their source Control now with artifactory those modules become binaries with all the benefits out of it you get checksum and checks and deploy immutability as opposed to source code without the git reaction how can you Tell which commit or tag is newer source code is mutable binaries are not and therefore they are more secured beside that you get the version in replication release bundles and many other advantages you’re getting with artifactory by using the latest release of the jfox cli we are also allowing you to publish your entire modules monorepo to our defactory terraform red repository in
one action and lastly we have the virtual repo solution that will let you aggregate all
your local and remote terraform repositories into one single source of
Truth now let’s take a little tour and see how things are really happening
i have already created us one local repository for the modules
one local repository for the providers one remote repository pointing to the official hashicorp terraform registry and one repository aggregating the three of them together here we have our monorepo of our in-house aws modules we’re about to publish artifactory first we will create the one-time configuration to choose the artifactory instance and the targeted repository so we choose the terraform modules local repo and here you can see the configuration folder it was created and now we will run the terraform publish action we will provide the namespace base provider folder modules and of course the tag perfect here we can see the destination the
modules were published to and now let’s explore those modules a Bit here we have our new kubernetes v10 Module in its artifactory info tab we can see some extra data like providers and module dependencies inputs outputs and even the readme file without downloading the entire module beside that you are also able to browse and search over all your published modules and Providers now let’s get back to the code and let’s create some simple module we will define a provider the aws Provider and one module and of course we want to resolve it from a to factory the source of the module is built from the artifactory domain the repository key double underscore
and finally the required module in the structure of namespace module name and
base provider now we are almost ready to initialize but let’s create a token with the native terraform of incommand first so artifactory support the native terraform login command it will open up the browser automatically and see since we are already logged in as admin we just need to approve and the token was created automatically we can close this page now and now we can run our terraform init action and all the the module and the provider is getting downloaded from artifactory as you can see we have installed our latest version of the efs module we have just deployed the v10
because we haven’t uh put any version inside the module itself so it took the latest tag now that we are able to initialize our environment with our custom modules let’s see how we can benefit out of the terraform backend repository solution
artifactory has a local repository implementation that supports terraform remote backend including workspaces and lock-in mechanisms in just a few minutes
you are able to migrate your existing back-end solution to artifactory and enjoy many other benefits it is easy to maintain managing all your users in one place it is safe all the state’s content is encrypted in the actual object storage you have a full state files history including publish time and the user who created it and you have a smart div between the state files to allow you monitor your changes easily when a problem spokes up and let’s take a tour over here as well so now we are going to migrate an
existing kubernetes workspace to use the artifactory backend first we will configure our backend to point to the artifactory tf back-end repository and second we will try to initialize the environment well we need to create a new workspace first so let’s create a workspace for our webinar and now we are ready to run the terraform in it again by the way we already have the token for the artifactory domain from the preview server form again so we don’t need to login again now we already have the token perfect finally we are ready to import our existing state to artifactory
Just push it and now we’re going to edit our module we’re going to add some new arbitrals so we’re going to have some difference from the existing state and let’s run some terraform plan. Okay perfect let’s apply those changes great let’s navigate back to the ui and explore our states.
We can see the state latest of our current active state and the state.timestamp of the previous state we can also see the user who created the state file when it was created and of course to read and search in the actual state content and lastly one of the best features is to compare the latest state with the old state so we can see the serial was changed but probably most more interesting for us is to see the arabic rule that we added so you can see how easy it is to uh seethe differences between the state files and monitor what was changed to those of you who survived we
presented here today the way jfrog is using terraform and the way jfrog makes your telephone solution work better faster and more reliably thank you for joining us today and may the frog be with you awesome thank you so much barack and amit for taking the time uh presenting this information to us and also you know putting it together.
To those on the call if you haven’t uh asked any questions or you want to
ask any questions now is your chance hopefully you’ve been putting them in
and had them answered as the session has gone on
but we have had a couple that have come in um
that we want to feel to our presenters here and we’re going to go ahead and do
that the first question i think is best for omit and the question is why do you
need to decouple your models if you’re running a daily
automation build.
Thanks Sean this is a great great question. It’s actually not just related to a terraform drift but also maintaining multiple layers of infrastructure. There is no relation between the kubernetes layer and the database layer so why they need to be in the same state is awesome.
yeah thanks for that great example amit.
barack another question that’s come in that i think maybe you could answer is what are the benefits um to using artifactory as the back end versus say a regular s3 back end okay that’s actually also a great question and well first we have the full state files history since the beginning of time and with the smart div capability it allows you to monitor your entire changes in your workspace perfectly so this is a great advantage and beside that you can manage your users and permissions in one place
instead of two and that makes things much more easier and and much uh much better to maintain so uh given that we’re running a little bit short on time we have uh the opportunity to answer one more question um i think this one is probably
a lot of the attendees here are curious about especially those that are uh
existing j4 customers but when will jfrog support infrastructure
as code scanning maybe brock you want to answer this one.
yeah sure uh well that’s an easy question to answer uh we have just announced that swamp up last week that infrastructure is called scanning is coming in q3 so keep an eye out for that feature coming soon awesome thanks barack great um so with that we’re going to go ahead and wrap up our webinar today if you didn’t have your questions answered no worries we’ll follow up with you directly i want to thank again uh barack and amit for presenting today and of course everyone for joining us uh be on the lookout for more great jfr webinars coming soon thanks everyone and we will catch you next time