DynamoDB, Lambdas and API GW with localstack (or not)
My original goal for this post was to get a local version of DynamoDB, API Gateway and Lambdas up and running using localstack. Unfortunately localstack doesn't seem to play well with cloudformation + lambdas... At a certain point this tutorial will transition from a localstack to a AWS-native approach. I'm keeping the localstack related commentary because I think it's useful to people looking to do the same.
My goal for this post is to get DynamoDB, API Gateway and Lambdas up and running locally on AWS using cloudformation).
This is a very small expansion of this post: {% link dev.to/elthrasher/exploring-aws-cdk-loading.. %}
I had intended for it to be heavier on the local dev experience but that didn't pan out.
Table of Contents
- Getting Started
- Creating the DynamoDB table
- Creating the Lambdas
- API Gateway
- Deploying to AWS
- Clean-up
- Final thoughts...
Getting Started
The code for this is here:
{% github martzcodes/blog-cdk-localstack no-readme %}
Make sure you have cdk installed
Locally, I have this installed:
$ node --version
v12.13.1
$ npm --version
6.12.1
$ aws --version
aws-cli/2.0.23 Python/3.7.4 Darwin/19.5.0 botocore/2.0.0dev27
$ cdk --version
1.45.0 (build 0cfab15)
To initialize a project:
$ cdk init --language typescript
Applying project template app for typescript
Initializing a new git repository...
Executing npm install...
# Welcome to your CDK TypeScript project!
This is a blank project for TypeScript development with CDK.
The `cdk.json` file tells the CDK Toolkit how to execute your app.
## Useful commands
* `npm run build` compile typescript to js
* `npm run watch` watch for changes and compile
* `npm run test` perform the jest unit tests
* `cdk deploy` deploy this stack to your default AWS account/region
* `cdk diff` compare deployed stack with current state
* `cdk synth` emits the synthesized CloudFormation template
There are some dependencies that will need to get installed... let's just take care of all of that now. Run: npm i @aws-cdk/aws-apigateway @aws-cdk/aws-dynamodb @aws-cdk/aws-iam @aws-cdk/aws-lambda @aws-cdk/aws-s3 @aws-cdk/aws-s3-assets @aws-cdk/aws-lambda-nodejs --save. S3 is needed because that's where the lambda code will actually live. Ends up... probably could've skipped the S3 dependencies, but they don't hurt anything...
Creating the DynamoDB table
Now, let's start to get our feet wet with CDK... Time to create the DynamoDB table. For this project we're going to make a table that tracks a user's ratings of beer. In my lib/blog-cdk-localstack-stack.ts file I'm going to update the code to:
import { AttributeType, Table, BillingMode } from "@aws-cdk/aws-dynamodb";
import { Construct, RemovalPolicy, Stack, StackProps } from "@aws-cdk/core";
export class BlogCdkLocalstackStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const modelName = "Beer";
const dynamoTable = new Table(this, modelName, {
billingMode: BillingMode.PAY_PER_REQUEST,
partitionKey: {
name: `UserId`,
type: AttributeType.STRING,
},
sortKey: {
name: `${modelName}Id`,
type: AttributeType.STRING,
},
removalPolicy: RemovalPolicy.DESTROY,
tableName: modelName,
});
}
}
This creates the dynamoDB table with a partition key as the UserId and a SortKey with the BeerId. Since this is a user-focused app this will enable me to get all of a user's beer ratings by using the partition key, and if I want to filter down to a specific beer and its ratings I can do that (there's a LOT more you can do with partition / sort keys but it's not needed for this example).
DynamoDB + localstack
Next, I'll run cdk synth BlogCdkLocalstackStack --profile personal > BlogCdkLocalstackStack.template.yml which will output the CloudFormation template to create the DynamoDB table. Now I need to get localstack somewhat set up...
I'll start with a somewhat standard docker-compose file:
version: "3.8"
services:
localstack:
image: localstack/localstack
ports:
- "4566-4599:4566-4599"
- "8080:8080"
environment:
- SERVICES=s3,lambda,cloudformation,apigateway,dynamodb
- DEBUG=true
- DATA_DIR=/tmp/localstack/data
- LAMBDA_EXECUTOR=local
- LAMBDA_REMOTE_DOCKER=false
- DOCKER_HOST=unix:///var/run/docker.sock
- START_WEB=1
volumes:
- ./tmp:/tmp/localstack
- "/var/run/docker.sock:/var/run/docker.sock"
And run docker-compose up -d. After waiting (and periodically checking the logs) to make sure it's up and ready... I can now run:
aws cloudformation create-stack --stack-name BlogCdkLocalstackStack --template-body file://./BlogCdkLocalstackStack.template.yml --endpoint-url http://localhost:4566 which should initialize the DynamoDB table in localstack. If I check the logs I get some log statements that don't necessarily give me a warm and fuzzy:
localstack_1 | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource BlogCdkLocalstackStack/BeerE09E1431: None
localstack_1 | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Deploying CloudFormation resource (update=False, exists=False, updateable=False): {'Type': 'AWS::DynamoDB::Table', 'Properties': {'KeySchema': [{'AttributeName': 'UserId', 'KeyType': 'HASH'}, {'AttributeName': 'BeerId', 'KeyType': 'RANGE'}], 'AttributeDefinitions': [{'AttributeName': 'UserId', 'AttributeType': 'S'}, {'AttributeName': 'BeerId', 'AttributeType': 'S'}], 'BillingMode': 'PAY_PER_REQUEST', 'TableName': 'Beer'}, 'UpdateReplacePolicy': 'Delete', 'DeletionPolicy': 'Delete', 'Metadata': {'aws:cdk:path': 'BlogCdkLocalstackStack/Beer/Resource'}}
localstack_1 | 2020-06-18T14:39:51:DEBUG:localstack.utils.cloudformation.template_deployer: Running action "create" for resource type "DynamoDB::Table" id "BeerE09E1431"
localstack_1 | 2020-06-18T14:39:51:DEBUG:localstack.utils.cloudformation.template_deployer: Request for resource type "DynamoDB::Table" in region us-east-1: create_table {'TableName': 'Beer', 'AttributeDefinitions': [{'AttributeName': 'UserId', 'AttributeType': 'S'}, {'AttributeName': 'BeerId', 'AttributeType': 'S'}], 'KeySchema': [{'AttributeName': 'UserId', 'KeyType': 'HASH'}, {'AttributeName': 'BeerId', 'KeyType': 'RANGE'}], 'ProvisionedThroughput': {'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5}}
localstack_1 | 2020-06-18T14:39:51:WARNING:localstack.services.cloudformation.cloudformation_starter: Unable to determine physical_resource_id for resource <class 'moto.dynamodb2.models.Table'>
localstack_1 | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource BlogCdkLocalstackStack/CDKMetadata: None
localstack_1 | 2020-06-18T14:39:51:WARNING:moto: No Moto CloudFormation support for AWS::CDK::Metadata
localstack_1 | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Running CloudFormation stack deployment loop iteration 1
But if I run aws dynamodb list-tables --endpoint-url http://localhost:4566 the table is definitely listed. From here I can verify the stacks: aws cloudformation list-stacks --endpoint-url http://localhost:4566, delete the stack aws cloudformation delete-stack --stack-name BlogCdkLocalstackStack --endpoint-url http://localhost:4566 and attempt to verify that the delete happened... turns out deleting the stack did not delete the table... could be a quirk of localstack or I'm using the wrong command...
Creating the Lambdas
Ultimately, I'm going to want to have two endpoints... GET all user's and their beer ratings... and Update a User's beer ratings. Clearly we'd want some more fidelity there, but this is a simple demo.
Let's start with the update lambda. I'll be somewhat basing my code off of this example: github.com/aws-samples/aws-cdk-examples/tre.. but I want to use the @aws-cdk/aws-lambda-nodejs cdk module so my lambda can be stored as a typescript file.
import { DynamoDB } from "aws-sdk";
const db = new DynamoDB.DocumentClient();
export const handler = async (event: any): Promise<any> => {
if (!event.body) {
return {
statusCode: 400,
body: "invalid request, you are missing the parameter body",
};
}
const body = JSON.parse(event.body);
try {
const review = await db
.update({
TableName: 'Beer',
Key: {
UserId: event.pathParameters.user,
BeerId: event.pathParameters.beer,
},
UpdateExpression: "set rating = :r",
ExpressionAttributeValues: {
":r": `${body.rating || 0}`
},
ReturnValues: "ALL_NEW",
})
.promise();
console.log(`Update complete. ${JSON.stringify(review)}`);
return {
statusCode: 200,
headers: {},
body: JSON.stringify(review),
};
} catch (e) {
console.error("GET failed! ", e);
return {
statusCode: 400,
headers: {},
body: `Update failed: ${e}`,
};
}
};
The GET ALL lambda ends up being very similar except it uses a scan of the DB which 99% of the time you should absolutely NOT do since it actively reads every item in your table!
Next... the lambdas need to get tied into the main blog-cdk-localstack-stack.ts file. Where we left off we only had the table. After creating the table we need to create the lambdas. For that I'm using the experimental NodejsFunction which can directly compile typescript files (it's quite convenient, but could be minimized more).
const updateReview = new NodejsFunction(this, "updateReviewFunction", {
entry: `${lambdaPath}/update-review.ts`,
handler: "handler",
runtime: Runtime.NODEJS_12_X,
});
const getReviews = new NodejsFunction(this, "readReviewFunction", {
entry: `${lambdaPath}/get-reviews.ts`,
handler: "handler",
runtime: Runtime.NODEJS_12_X,
});
The lambdas also need to have access to dynamodb:
dynamoTable.grantReadWriteData(updateReview);
dynamoTable.grantReadData(getReviews);
Time to deploy this to localstack and invoke the functions manually...
Lambdas + localstack via cloudformation
At this point I figured out CloudFormation lambdas on localstack STILL do not work
I tried a number of things to no avail. The error has to do with where localstack / cloudformation is trying to store the lambda code. Lambda code is stored on S3... localstack had s3 turned on but the parameter was supposedly missing. This is the error for anyone curious:
localstack_1 | 2020-06-18T16:26:12:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response for CloudFormation action "CreateStack" (400) POST /: b'<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">\n <Error>\n <Type>Sender</Type>\n <Code>Missing Parameter</Code>\n <Message>Missing parameter AssetParameters7708d9acea8c2c3d47875fb4c6499e05f52897d0ff284eb6811ad1fc7d97eeadS3Bucket9261AF10</Message>\n </Error>\n <RequestId>cf4c737e-5ae2-11e4-a7c9-ad44eEXAMPLE</RequestId>\n</ErrorResponse>'
So, I'm going to skip localstack, for now...
API Gateway
Finally, we need to set up the api gateway to use the lambdas:
const api = new LambdaRestApi(this, 'beer-api',
{
handler: getReviews,
proxy: false
});
const users = api.root.addResource('users');
users.addMethod("GET", new LambdaIntegration(getReviews));
const user = users.addResource('{user}');
const userReview = user.addResource('{beer}');
userReview.addMethod("POST", new LambdaIntegration(updateReview));
First we create a lambda-based REST api. It requires a default handler but with proxy set to false nothing will actually go to it... they need to be manually defined which is what the following lines do.
Deploying to AWS
Since localstack is out of the picture, to get this deployed to AWS I'll need to do cdk bootstrap --profile personal first... Which will initialize the stack on my AWS account. If I follow that by cdk deploy --profile personal it will ask if I want to deploy the changeset (I do). After a minute or two of logs I get the result:
Outputs:
BlogCdkLocalstackStack.beerapiEndpointE86AAAC2 = https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/
Which, if I add add my api routes I can hit https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/users to get a 200 response with no items and alternately do a POST to https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/users/123/123 with the body {"rating": 8} which responds with the updated item:
{
"Attributes": {
"BeerId": "123",
"UserId": "123",
"rating": "8"
}
}
It's not local, but it WAS easy...
Clean-up
To remove everything in AWS it's as simple as running cdk destroy --profile personal
$ cdk destroy --profile personal
Are you sure you want to delete: BlogCdkLocalstackStack (y/n)? y
BlogCdkLocalstackStack: destroying...
0 | 10:12:39 PM | DELETE_IN_PROGRESS | AWS::CloudFormation::Stack | BlogCdkLocalstackStack User Initiated
0 | 10:12:42 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Account | beer-api/Account (beerapiAccount2152DC9C)
0 | 10:12:42 PM | DELETE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata
0 | 10:12:42 PM | DELETE_IN_PROGRESS | AWS::Lambda::Permission | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer86A593EA)
0 | 10:12:42 PM | DELETE_IN_PROGRESS | AWS::Lambda::Permission | beerapiusersuserGETApiPermissionBlogCdkLocalstackStackbeerapi01021E61GETusersuser199DA8EE
0 | 10:12:42 PM | DELETE_IN_PROGRESS | AWS::Lambda::Permission | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.Test.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer16885B45)
1 | 10:12:42 PM | DELETE_COMPLETE | AWS::ApiGateway::Account | beer-api/Account (beerapiAccount2152DC9C)
1 | 10:12:42 PM | DELETE_IN_PROGRESS | AWS::Lambda::Permission | beerapiusersuserGETApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61GETusersuser9933087C
1 | 10:12:43 PM | DELETE_IN_PROGRESS | AWS::IAM::Role | beer-api/CloudWatchRole (beerapiCloudWatchRole81F996BB)
2 | 10:12:43 PM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata
3 | 10:12:44 PM | DELETE_COMPLETE | AWS::IAM::Role | beer-api/CloudWatchRole (beerapiCloudWatchRole81F996BB)
4 | 10:12:52 PM | DELETE_COMPLETE | AWS::Lambda::Permission | beerapiusersuserGETApiPermissionBlogCdkLocalstackStackbeerapi01021E61GETusersuser199DA8EE
5 | 10:12:52 PM | DELETE_COMPLETE | AWS::Lambda::Permission | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.Test.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer16885B45)
6 | 10:12:52 PM | DELETE_COMPLETE | AWS::Lambda::Permission | beerapiusersuserGETApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61GETusersuser9933087C
7 | 10:12:52 PM | DELETE_COMPLETE | AWS::Lambda::Permission | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer86A593EA)
7 | 10:12:53 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Stage | beer-api/DeploymentStage.prod (beerapiDeploymentStageprod3F2FBBD3)
8 | 10:12:54 PM | DELETE_COMPLETE | AWS::ApiGateway::Stage | beer-api/DeploymentStage.prod (beerapiDeploymentStageprod3F2FBBD3)
8 | 10:12:55 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Deployment | beerapiDeployment3077C14B26da823256c4ecc7e0f6cf539683f3ef
9 | 10:12:56 PM | DELETE_COMPLETE | AWS::ApiGateway::Deployment | beerapiDeployment3077C14B26da823256c4ecc7e0f6cf539683f3ef
9 | 10:12:57 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Method | beerapiusersuserGETCE81253D
9 | 10:12:57 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Method | beer-api/Default/users/{user}/{beer}/POST (beerapiusersuserbeerPOST2D1FBB1F)
10 | 10:12:57 PM | DELETE_COMPLETE | AWS::ApiGateway::Method | beerapiusersuserGETCE81253D
11 | 10:12:58 PM | DELETE_COMPLETE | AWS::ApiGateway::Method | beer-api/Default/users/{user}/{beer}/POST (beerapiusersuserbeerPOST2D1FBB1F)
11 | 10:12:58 PM | DELETE_IN_PROGRESS | AWS::Lambda::Function | readReviewFunction (readReviewFunction94EC729D)
11 | 10:12:58 PM | DELETE_IN_PROGRESS | AWS::Lambda::Function | updateReviewFunction (updateReviewFunctionF0FF17E0)
11 | 10:12:58 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Resource | beer-api/Default/users/{user}/{beer} (beerapiusersuserbeerE37FAAD9)
12 | 10:12:59 PM | DELETE_COMPLETE | AWS::Lambda::Function | readReviewFunction (readReviewFunction94EC729D)
13 | 10:12:59 PM | DELETE_COMPLETE | AWS::ApiGateway::Resource | beer-api/Default/users/{user}/{beer} (beerapiusersuserbeerE37FAAD9)
14 | 10:12:59 PM | DELETE_COMPLETE | AWS::Lambda::Function | updateReviewFunction (updateReviewFunctionF0FF17E0)
14 | 10:13:00 PM | DELETE_IN_PROGRESS | AWS::IAM::Policy | readReviewFunction/ServiceRole/DefaultPolicy (readReviewFunctionServiceRoleDefaultPolicy2543AE62)
14 | 10:13:00 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Resource | beer-api/Default/users/{user} (beerapiusersuser40A97AB3)
14 | 10:13:00 PM | DELETE_IN_PROGRESS | AWS::IAM::Policy | updateReviewFunction/ServiceRole/DefaultPolicy (updateReviewFunctionServiceRoleDefaultPolicyD907B783)
15 | 10:13:00 PM | DELETE_COMPLETE | AWS::IAM::Policy | readReviewFunction/ServiceRole/DefaultPolicy (readReviewFunctionServiceRoleDefaultPolicy2543AE62)
16 | 10:13:00 PM | DELETE_COMPLETE | AWS::ApiGateway::Resource | beer-api/Default/users/{user} (beerapiusersuser40A97AB3)
17 | 10:13:01 PM | DELETE_COMPLETE | AWS::IAM::Policy | updateReviewFunction/ServiceRole/DefaultPolicy (updateReviewFunctionServiceRoleDefaultPolicyD907B783)
17 | 10:13:01 PM | DELETE_IN_PROGRESS | AWS::IAM::Role | readReviewFunction/ServiceRole (readReviewFunctionServiceRole77A9B597)
17 | 10:13:01 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::Resource | beer-api/Default/users (beerapiusers7623AC8A)
17 | 10:13:02 PM | DELETE_IN_PROGRESS | AWS::DynamoDB::Table | Beer (BeerE09E1431)
17 | 10:13:02 PM | DELETE_IN_PROGRESS | AWS::IAM::Role | updateReviewFunction/ServiceRole (updateReviewFunctionServiceRole57FA81FA)
18 | 10:13:02 PM | DELETE_COMPLETE | AWS::ApiGateway::Resource | beer-api/Default/users (beerapiusers7623AC8A)
19 | 10:13:03 PM | DELETE_COMPLETE | AWS::IAM::Role | readReviewFunction/ServiceRole (readReviewFunctionServiceRole77A9B597)
19 | 10:13:04 PM | DELETE_IN_PROGRESS | AWS::ApiGateway::RestApi | beer-api (beerapi67AE41CF)
20 | 10:13:04 PM | DELETE_COMPLETE | AWS::IAM::Role | updateReviewFunction/ServiceRole (updateReviewFunctionServiceRole57FA81FA)
21 | 10:13:04 PM | DELETE_COMPLETE | AWS::ApiGateway::RestApi | beer-api (beerapi67AE41CF)
✅ BlogCdkLocalstackStack: destroyed
Final thoughts...
It's definitely possible to run everything in localstack by manually running all the commands... but that ultimately defeats the purpose of using CDK.
Do you have any tips for doing local development across API Gateway -> Lambdas -> DynamoDB? With or without localstack?
