Reusable Cloud Infrastructure as Code with Pulumi and JFrog Artifactory
This guest post is submitted by Chris Smith of Pulumi and is co-posted on the Pulumi blog.
Pulumi enables you to specify cloud infrastructure with code. This empowers you to program the cloud in your favorite language, and benefit from useful and familiar features of coding like static analysis, type checking, IDEs, and more.
One major advantage of the Pulumi model, is that you can create and reuse cloud infrastructure components using package managers like npm.
Creating a Cloud Application with Pulumi
Here’s a condensed view of a Pulumi program, showing the code-centric model in practice. (You can see the full source on GitHub.) The Pulumi application allows you to upload a video to the cloud, and then uses AI services to analyze that video for common objects, e.g. “stop sign” or “palm tree.” Then it extract a thumbnail image for each object found.
This example highlights the advantage of using code packages with Pulumi. Rather than programming directly against various products like Amazon Web Services’ Lambda, Fargate, and others, packages are used so that program is written at a higher-level of abstraction.
The Pulumi program uses the cloud package for storage (via Bucket) and arbitrary compute on Docker containers (via Task). For the more specialized task of object recognition, it uses a custom video package which relies on the Amazon’s Rekognition service.
By having the cloud infrastructure described in reusable packages, clients are free to spend more time on logic specific to the task at hand, leaving the details about cloud provider services and APIs to the package’s implementation.
const videoUploads = new cloud.Bucket(...);
const extractThumbnailTask = new cloud.Task(...);
const videoProcessor = new video.VideoLabelProcessor();
...
// Whenever an .mp4 file is uploaded to the S3 bucket, start a new analysis job.
videoUploads.onPut(
"onNewVideo",
(newBucketObject: BucketArgs) => {
videoProcessor.analyzeVideo(videoUploads.get(), newBucketObject.key);
...
},
{ keySuffix: ".mp4" });
...
// Whenever an analysis job completes, the onLabelComplete callback is called. The function
// defined here is run on the cloud using AWS Lambda, which then will kick off an AWS Fargate
// task to perform the image extraction.
videoProcessor.onLabelingComplete((file: string, labels: VideoLabels[]) => {
...
// Spin up our Docker container to perform the image extraction process,
// uses AWS Fargate under the hood.
extractThumbnailTask.run(...);
...
});
An architectural diagram for the Pulumi program can be seen below. The exportedVideoLabelProcessor class handles the interactions with AWS Rekogntion and SNS, so clients just need to call analyzeVideo and provide a callback for onLabelingComplete.
But the benefit of reusable packages isn’t just about productivity. By reusing components, it is easier to follow security best practices because any updates only need to be made to the package. Packages can also provide more advanced scenarios, such as handling the details hooking into your organization’s logging and alerting system.
One tradeoff using Pulumi for managing cloud infrastructure however, is that you now need to handle the versioning and distribution of these various packages. How can you ensure that people on the team or within the organization are using the latest version? Or that of the video package? Or that the Docker container is built using the right base image?
Artifactory to the Rescue
This is where where JFrog Artifactory can step in, and handle the entire package/dependency management problem.
Artifactory handles code dependencies and build artifacts of all kinds. Whether they are packages for a standard website, or for managing cloud infrastructure with Pulumi, Artifactory will work the same.
By using Artifactory as your npm registry to host the npm package for image recognition, we can reliably enforce versioning as well as provide a secure, reliable way for distributing packages. Similarly, Artifactory can be used as a Docker container registry too.
Pulumi provides a new way to think about and use software packages and JFrog Artifactory continues to be the best solution for storage, management, and distribution of those artifacts.
Ready to make coding cloud infrastructure as easy as “Programming 101”?
