Building cloud applications is hard. On the one hand, the number of services available from public cloud providers is astonishing, and these offerings can dramatically increase your team’s velocity. And yet, integrating these services into a product nearly always requires deep knowledge, since documentation is not always as clear or well-organized as you’d like. Integrating them into a real application is tough, all the more when a particular product feature might need to direct requests to 5 or 6 different AWS services, each of which requires piles of JSON to configure correctly.
For my work at the Washington Post, I was recently tasked with creating a proof-of-concept (POC) to test out a particular combination of AWS services (MediaLive+MediaPackage+CloudFront+MediaTailor). MediaLive does live video encoding; MediaPackage performs origination and packaging; CloudFront does content-delivery, and MediaTailor stitches in ads. Over the course of a two week sprint, I managed to put together a fully functional POC in Node/TypeScript that would create the resources needed in AWS for this particular video workflow. Here’s how I did it:
Amazon’s web console holds your hand. This is a good thing, but should be used with care. I knew from reading the AWS docs what services I’d probably need to use. So I opened up our Sandbox account and got to work. I created a MediaPackage channel and a MediaPackage origin endpoint. The web console helpfully offered to create a CloudFront web distribution to sit in front of this. I said, “sure!” Then I went in and started making the entities in MediaLive that I needed (Inputs, Input Security Groups, and a Channel). The MediaLive channel was by far the trickiest, since it includes transcoding profiles for every different resolution you want to produce (to get a since of the complexity in play, you can look here. It was more than 300 lines of JSON). Once I got the video working (I could stream video to an RTMP endpoint and watch the resulting HLS stream coming out the other side) I added in a MediaTailor to test ad stitching. This threw me for a loop for a while, because MediaPackage was unhelpfully squashing the ad markers. But once I found the right checkbox to flip, this started working too. At this point, it was time to capture what I’d made in code so it would be repeatable.
Trying to manually figure out how to encode all that I’d done in TypeScript client code would have taken forever. Even with TypeScript SDK docs, or CloudFormation docs to reference, I’d still have needed to map the values in the web UI to the code I was writing. Happily, there was an easier way: the AWS cli will simply dump out the configuration of whatever entity you’re interested in. For instance, this will give you JSON for the CloudFront distributions in your AWS account:
aws cloudfront list-distributions > distributions.json
The exact command varies depending on the service.
aws <servicename> help will generally lead you to the right command. Generally you want one that starts with
aws ec2 describe-instances
It’s helpful to remember that the
aws-cli tools are fundamentally just convenience wrappers to REST APIs, which is why they often aren’t terrible user friendly. REST APIs and traditional *nix tools have different structures and conventions.
I’ve since used the same strategy to put together some ECS blue/green deploy code. Juggling mountains of JSON is not fun. Creating what you need in the web console, by contrast, and then exporting via the CLI will speed up your develop of infrastructure-facing code, especially if you can’t find good examples to follow. Good luck!