Weapon of Choice

Lately, I’ve been developing a Lambda function to create a pair of keys, store one in a parameter and return the other to the user. Today I got my first clean run, and I’m writing this to celebrate.

As part of learning lots of things along the way, because I hadn’t developed anything on AWS before, I found out that AWS reports errors in many different ways. Here are those that occurred to me in the last few hours.

See: AWS Lambda Invoke Errors (documentation)

Error — InvalidZipFileException

{ InvalidZipFileException: Lambda was not able to unzip the file
       <stack trace...>
     message: 'Lambda was not able to unzip the file',
     code: 'InvalidZipFileException',
     time: 2017-09-16T16:04:04.558Z,
     requestId: '<request id...>',
     statusCode: 502,
     retryable: true } }

• This error occurred because the zip file I had uploaded was wrong: the files it contained were stored into a directory.
• This is a Lambda creation error, and it’s quite complete.
• Unhandled / Handled classification is limited to Lambda execution.

Error — Process exited

{ errorMessage: 'RequestId: <request id...> Process exited before completing request' }

• This error occurred because I threw a validation error.
• I could catch this error, but don’t, so it’s classified as Unhandled.
• The logs show my error and its stack trace.

17:12:39 START RequestId: <request id...> Version: $LATEST
17:12:39 2017-09-16T17:12:39.442Z   <request id...>    Error: Expected a pair ID for your keys. at setPairId (/var/task/createPairOfKeys.js:9832:15) at handler (/var/task/createPairOfKeys.js:9805:9)
17:12:39 END RequestId: <request id...>
17:12:39 REPORT RequestId: <request id...> Duration: 203.72 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 78 MB
17:12:39 RequestId: <request id...> Process exited before completing request

Error — Timed out

{ errorMessage: '2017-09-16T17:32:08.746Z <request id...> Task timed out after 10.00 seconds' }

• This error occurred because generating a pair of keys is not fast (bigint maths involved) and, in my few tests, it sometimes needed more than 40 seconds.
• Surprisingly, my MacBook is faster than AWS Lambda. A few times it took less than 1 second, most of the times less than 6 seconds, and only once it timed out at 10 seconds.
• I can’t catch this error, so it’s classified as Unhandled.

Error — AccessDeniedException

{ errorMessage: 'User: <user arn...> is not authorized to perform: ssm:PutParameter on resource: <resource arn...>',
        errorType: 'AccessDeniedException',
        stackTrace: [Object] } } }

• This error occurred because the role I had assigned to my Lambda function didn’t have the right to write SSM parameters.
• I catch this error and swallow it, so it’s classified as Handled.

Error — ParameterAlreadyExists

{ errorMessage: 'The parameter already exists. To overwrite this value, set the overwrite option in the request to true.',
        errorType: 'ParameterAlreadyExists',
        stackTrace: [Object] } } }

• This error occurred because I explicitly create my SSM parameter with overwrite set to false. Not gonna change it, this is by design.
• Notice how this is an exception whose name doesn’t end with Exception.
• I catch this error and swallow it, so it’s classified as Handled.

The AWS Lambda, Programming Model document states:

Your Lambda function code must be written in a stateless style, and have no affinity with the underlying compute infrastructure.

The AWS Lambda, Best Practices document recommends:

Take advantage of container re-use to improve the performance of your function.

Those contradictory statements make sense at scale.

When an AWS Lambda function completes (which means it terminates without unhandled exceptions, either with an error (also referred to as a failure) or a result (also referred to as a success)), the AWS Lambda service will automatically freeze its container and probably reuse it for a new invocation.

When an AWS Lambda function takes longer than the specified maximum execution time, the AWS Lambda service times it out by throwing an exception (unhandled by definition) and then manages it accordingly to some policies.

Given that AWS Lambda functions for a NodeJS runtime are NodeJS applications that, in general, add callbacks to the event loop, the AWS Lambda service will, by default, make sure that there is no pending work before considering the AWS Lambda function fully completed, independently from the fact that its callback was already called or not. To call the callback is, in fact, optional.

Unfortunately, if the AWS Lambda function somehow blocks the event loop (for example, a database connection which is being periodically polled), the AWS Lambda service won’t ever find the event loop empty and the AWS Lambda function will time out.

NodeJS V8 has an experimental Async Hooks module to help debug these situations, while V6 (used by AWS Lambda) has these two undocumented functions:

• process._getActiveHandles() gets you handles that are still alive
• process._getActiveRequests() gets you info about active libuv requests.

Conveniently, the AWS Lambda service calls the exports.handler function with a second argument set to the container context. In turn the context has a callbackWaitsForEmptyEventLoop property which can be set to false to signal to the AWS Lambda service that the callback call (in a way) marks the end of the AWS Lambda function.

In reality:

AWS Lambda will freeze the process, any state data and the events in the NodeJS event loop (any remaining events in the event loop [will be] processed when the Lambda function is called next and if AWS Lambda chooses to use the frozen process).

let expensiveResult;

exports.handler = (event, context, callback) => {
    endWhenTheCallbackIsCalledOrTheTimeoutIsReached(context);

    process(event, callback);
};

// ---

function process(event, done) {
    try {
        if (! expensiveResult) {
            // set expensiveResult
        }
        // use expensiveResult
        // then
        done(null, result);
    } catch (e) {
        done(JSON.stringify(e));
    }
}

function endWhenTheCallbackIsCalledOrTheTimeoutIsReached(context) {
    context.callbackWaitsForEmptyEventLoop = false;
}

function endWhenTheEventLoopIsEmptyOrTheTimeoutIsReached(context) {
    context.callbackWaitsForEmptyEventLoop = true;
}

 

This solution is based on Google Chrome, Gmail, and the Mailvelope extension for Chrome, which are all free.

First-time procedure

From the second time on, you only need Step 4 for storing and Step 6 for retrieving.

1. Register a new Gmail account or use one of yours. Be it `X@gmail.com`.
2. Install Mailvelope.
3. Generate a key pair for `X+secrets@gmail.com`.
4. Compose a message for `X+secrets@gmail.com`.
   1. Use a Subject like `Awesome Service` for your Awesome Service account.
   2. Click the Mailvelope icon that appears when you hover the body.
   3. You’ll see that a dialog appears with `X+secrets@gmail.com` in green.
   4. Enter account details, like your password.
   5. `Encrypt`
   6. You’ll see that the body of the message is now encrypted.
   7. Send the message.
5. Given that `X+secrets@gmail.com` is just an alias of `X@gmail.com` you got the message in your inbox.
6. Open it and confirm that it’s still encrypted.
   1. Click the Mailvelope icon that appears when you hover the body.
   2. Enter the password you specified in Step 3.
   3. You’ll see that the message is now decrypted.
   4. Close the message.

Why is this secure?

• The network connection between your browser and Gmail servers is encrypted by default, so only you and Google can read your messages. With this encryption, not even Google will be able to read them.
• When you send an email message protected by someone’s public encryption key, such a message can only be read by that someone using their matching private decryption key.
• Anyone who got access to your browser without your consent and tried to steal one of these secrets would have to enter your password to read them.

Why is this practical?

• It’s secure and free.
• You can add a filter to label these messages so they appear collected.
• You can search the subject or any other metadata you added to the bodies.
• You can access your secrets from any Gmail device, anywhere in the world.
• You can also keep all related secrets in a text file and send it as an encrypted attachment to `X+secrets@gmail.com`.