Lambda Layers

AWS Lambda functions have been around for awhile now and used by thousands of people. But an often underused feature are Lambda Layers which were introduced back in 2018. AWS describes them as "With Lambda Layers, you can package and deploy libraries, custom runtimes, and other dependencies separately from your function code. Share your Layers with your other accounts or the whole world. For details, see Lambda Layers.

Yarn Zero Installs

Starting with Yarn 2, there is a .yarn/cache directory that stores all the packages installed. This directory can be optionally included in your repository and suddenly, the next developer to clone the project can immediately run it - no need for complicated logins, proxies, being on the right network (having a network!), etc. For details, see Yarn Zero Installs

So, what's the problem?

Lambdas run only the code that you provide, ideally business logic and nothing else. When it's a single file or even a few files that rely on each other, there's not much to worry about - the runtime environment (node in this case) takes care of it for you.

In my project, I needed to read some credentials from AWS Secrets Manager, so I ran yarn add @aws-sdk/client-secrets-manager to get a library with all the code I need to be able to do that. Now, we have to remember to bundle that package and any of it's dependencies together with your business logic when you upload it to Lambda. Some frameworks will bundle all your dependencies into the same file as all your business logic and upload that to Lambda. This works fine, but you're usually left with a file that's too big to render in the AWS Console, which limits your ability to debug on the fly. If you've got multiple Lambdas, they'll all have this same problem.

With Yarn Zero Installs, there's no node_modules directory. Instead, we have a .yarn/cache directory to work with, which is a collection of zip files, one for each dependency listed in package.json. You might have also noticed that there's a .pnp.js file at the root of your project directory that gets updated when you add/update/remove a dependency. This is the key to making it all work. The .pnp.js file automatically patches Node's fs module to add support for accessing zip files, making the whole process seamless.

Using Layers to our advantage

Layers give us the ability to bundle the .yarn/cache directory and .pnp.js file, so that they can be re-used across multiple Lambdas. It also allows us to keep the size of the deployed application small, which will enable Console editing for on the fly tweaks, debugging, etc.

Getting these files bundled correctly is a bit complicated. Unfortunately, we can't just replicate our local development environment in a Layer. Lambda will extract the contents of your Layers into the /opt directory (this is important later) and, more specifically, into certain PATH variable entries, depending on your environment. In this case, we're expecting to find our Layer contents in /opt/nodejs/.

The details

First up, we need something to zip our files. Archiver popped up during my hunt, but I couldn't get it working and a few StackOverflow posts from others with similar issues, made me abandon that approach. Instead, I found ADM-ZIP and it does what you expect it to do. This got the job done for my project, you might need something more advanced for your own project.

import AdmZip from 'adm-zip';
const createBundle = (destination: string) {
    const zip = new AdmZip();
    zip.addLocalFile(
        path.join(process.cwd(), '.pnp.js'),
        'nodejs'
    );
    zip.addLocalFolder(
        path.join(process.cwd(), '.yarn'),
        path.join('nodejs', '.yarn')
    );
    zip.writeZip(destination);
}

When you reference this zip as the code asset for your Layer, it will create a structure like the one below, in the Lambda environment. It's difficult to confirm this structure is present, even using the AWS Console:

opt/
    nodejs/
        .pnp.js
        .yarn/
            .cache
            ...
        business_logic/
            ...

We need to make one more modification, to the Lambda itself;

require('opt/.pnp.js').setup();

I do this at the top of the file that is the entry point for my lambda (the handler), so it should be the first peice of code that's executed. This is the recommendation directly from Yarn themselves. The main call out there is that yarn run will automatically call pnp setup, but the Lambda environment doesn't afford us that luxury, so we have to tell it explicitly. Setup will do the patching of fs so that our code can read the zip files in the .cache, just like local development. No further changes necessary and your imports will all still work in both your development environment and the Lambda environment, yay!

That's it?

Yep, that's it. It seems complicated and scary at first. Now though, I have the building blocks required to go wild. My project has about a dozen Lambdas, they'll all now use Layers and reference the same Layer that contains the yarn cache and pnp file. They all have shared code that I've had to duplicate, so now that'll now go in a second Layer that they can all reference too.

There's a few gotchas to be aware of;

  • A Lambda can reference a maximum of 5 Layers - that seem like quite a bit, I'm expecting to use 2 Layers in the vast majority of cases.
  • The total unzipped size of the Lambda and the Layers it references, cannot exceed 250MB - again, this seems generous.
  • If your Lambda is referencing a Layer from a different AWS Account and the Layer is deleted or permissions are revoked, your existing Lambda will continue to operate. New Lambdas cannot reference that Layer though.
  • Lambda extracts each Layer in the order you specify, merging any duplicate folders. Duplicate files will be overwritten, Lambda chooses the file that is currently being extracted as the one to keep.

Now you've got all the building blocks. Bolt Yarn Zero Installs onto your project (it makes the developer onboarding experience more enjoyable), get your hands dirty with Layers and most importantly, have fun!