Configure DynamoDB in Serverless

jayair · May 9, 2018, 11:12pm

Link to chapter - https://serverless-stack.com/chapters/configure-dynamodb-in-serverless.html

Acidfabric · May 27, 2018, 12:26am

OS: darwin
Node Version: 8.11.2
Serverless Version: 1.26.0

Problem is I can’t access env variables with process.env.tableQuestions
Function returns error message:
Invalid table/index name. Table/index names must be between 3 and 255 characters long, and may contain only the characters a-z, A-Z, 0-9, ‘_’, ‘-’, and ‘.’

Here is part of serverless.yml file:

custom:
  stage: ${opt:stage, self:provider.stage}
  tableThroughputs:
    prod: 5
    default: 1
  tableThroughput: ${self:custom.tableThroughputs.${self:custom.stage}, self:custom.tableThroughputs.default}
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true
  dynamodb:
    start:
      port: 8000
      inMemory: true
      migrate: true
      seed: true
      noStart: true
    seed:
      test:
        sources:
          - table: ${env:tableQuestions}
            rawsources: [./offline/seeds/questions.json]

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: eu-central-1
  environment:
    tableQuestions:
      Ref: QuestionsTable
    tableReplays:
      Ref: ReplaysTable

iamRoleStatements:
  - Effect: Allow
    Action:
      - dynamodb:DescribeTable
      - dynamodb:Query
      - dynamodb:Scan
      - dynamodb:GetItem
      - dynamodb:PutItem
      - dynamodb:UpdateItem
      - dynamodb:DeleteItem
    Resource:
      - "Fn::GetAtt": [ QuestionsTable, Arn ]
      - "Fn::GetAtt": [ ReplaysTable, Arn ]

And this is my dynamoDb Resource configuration:

Resources:
    QuestionsTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: questionId
            AttributeType: S
        KeySchema:
          - AttributeName: questionId
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: ${self:custom.tableThroughput}
          WriteCapacityUnits: ${self:custom.tableThroughput}
        TableName: ${self:custom.stage}-questions

    ReplaysTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: userId
            AttributeType: S
        KeySchema:
          - AttributeName: userId
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: ${self:custom.tableThroughput}
          WriteCapacityUnits: ${self:custom.tableThroughput}
        TableName: ${self:custom.stage}-replays

jayair · May 27, 2018, 10:16pm

It looks okay. What does process.env.tableQuestions & process.env.tableReplays resolve to in your Lambda if you do a console.log?

Acidfabric · May 31, 2018, 9:53pm

Don’t know. Stuck at function invoke, error message:

{
    "statusCode": 500,
    "headers": {
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Credentials": true
    },
    "body": "{\"message\":\"1 validation error detected: Value '[object Object]' at 'tableName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+\"}"
}

I need to check if a problem persits in production enviroment, at the moment I’m trying to invoke functions localy

Acidfabric · May 31, 2018, 10:10pm

I can confirm it works in production. But it’s incompatible with serverless-offline plugin

jayair · June 4, 2018, 5:16pm

Oh how are you invoking it locally through the serverless-offline plugin?

Acidfabric · June 4, 2018, 6:31pm

Yes. I’m running dynamoDb docker container and serverless-offline plugin to test my api

jayair · June 4, 2018, 6:42pm

And what’s the command you are running that gave you the error?

aaditya-panik · June 9, 2018, 1:14am

I’m fairly new to this, so I may be incorrect. After some debugging, I discovered that the tableName env variable gets assigned the [object Object] value, maybe because it doesn’t evaluate the ‘Ref’ intrinsic function (Not played around with serverless as much but thought this might be helpful).
Anyways, you can resolve the issue by replacing

tableName: Ref NotesTable

with,

tableName: notes-${self:custom.stage}

jayair · June 10, 2018, 2:15am

Hmmm that is weird. If you’ve configured your resources correctly, it should pick it up. The Ref is simply pointing to ${self:custom.stage}-notes.

github.com

AnomalyInnovations/serverless-stack-demo-api/blob/master/serverless.yml#L129


  events:
    - http:
        path: billing
        method: post
        cors: true
        authorizer: aws_iam


# Create our resources with separate CloudFormation templates
resources:
# DynamoDB
- ${file(resources/dynamodb-table.yml)}
# S3
- ${file(resources/s3-bucket.yml)}
# Cognito
- ${file(resources/cognito-user-pool.yml)}
- ${file(resources/cognito-identity-pool.yml)}

aaditya-panik · June 10, 2018, 6:16pm

An issue of this nature has already been posted on the Repo issues: https://github.com/serverless/serverless/issues/3080

Following that issue thread, you can install a plugin serverless-export-env to properly initialize all environment variables with Cloudformation template references.

Again, I’m new to this so I’ve encountered this problem when I use the serverless invoke local rather than serverless-offline plugin. Not sure if it works different for the plugin.

jayair · June 11, 2018, 5:34pm

That’s a really old issue. Which version of Serverless Framework are you using?

khrome83 · August 2, 2018, 1:39am

What is the best way to share Dynamo DB across multiple services. According to Amazon, having a single table is idea, which I can do.

I need to modify different parts of the object, hence difference services that interact with the document differently.

How would you setup the resource in a monorepo?

Separate Resource Service (how does that work in Seed, race condition?)
Each serverless.yml references the same /resources/dynamoDb.yml configuration and import it?

Additionally, do IAM roles for DynamoDB impact all requests in the service? Can you restrict delete actions to the /delete path for example. May need a different user credentials to delete for example?

Sorry still very new to this.

jayair · August 3, 2018, 6:48pm

I talked about the resource sharing in the other thread - Organizing Serverless Projects.

But for the IAM roles, it really depends on the level at which you want to control this. You can use Cognito User Groups. Or you can manage these internally in your code if you have some slightly complicated business logic to resolve permissions.

seth-wat · August 8, 2018, 8:51pm

Is there a good documentation resource on configuring DynamoDB in Serverless’s yaml? I’m checking out the docs here: https://serverless.com/framework/docs/providers/aws/guide/resources/ but it’s not very comprehensive

jayair · August 10, 2018, 8:52pm

Yeah I listed one in this chapter.

It’s a guide dedicated to DynamoDB - https://www.dynamodbguide.com/.

ependydad · September 8, 2018, 1:32am

I’m experiencing the same as @acidfabric and the fix shown by @aaditya-panik is what I had to do to get past it for now. I really dislike the duplicative naming and I’m sure I’m doing something wrong, but I’m not seeing it.

Some background:

I read through part 1 in the past 2 weeks
I started with part 2 to create a base project and literally just tonight cloned the repo
I have made minor modifications to reflect the real-world project that I plan on using it (namely table + field names)
I’m using WebStorm as my IDE

Here’s how I invoke the method:

serverless invoke local --function create --path mocks/create-event.json

Here’s the error that gets returned:

      message: '1 validation error detected: Value \'[object Object]\' at \'tableName\' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+',
      code: 'ValidationException',
      time: 2018-09-08T01:28:37.270Z,

When I do console.log(process.env.TripsTableName);, I simply get:
[object Object]

Here is my serverless.yml:

service: routinator

# Use the serverless-webpack plugin to transpile ES6
plugins:
  - serverless-webpack
  - serverless-offline

# serverless-webpack configuration
# Enable auto-packing of external modules
custom:
  # Our stage is based on what is passed in when running serverless
  # commands. Or fallsback to what we have set in the provider section.
  stage: ${opt:stage, self:provider.stage}
  # Set our DynamoDB throughput for prod and all other non-prod stages.
  tableThroughputs:
    prod: 5
    default: 1
  tableThroughput: ${self:custom.tableThroughputs.${self:custom.stage}, self:custom.tableThroughputs.default}
  # Load our webpack config
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: us-east-1

  # These environment variables are made available to our functions
  # under process.env.
  environment:
    TripsTableName:
      Ref: TripsTable

  # 'iamRoleStatement' defines the permission policy for the Lambda function.
  # In this case Lambda functions are granted with permissions to access DynamoDB.
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeTable
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      # Restrict our IAM role permissions to
      # the specific table for the stage
      Resource:
        - "Fn::GetAtt": [ TripsTable, Arn ]

functions:
  # Defines an HTTP API endpoint that calls the main function in create.js
  # - path: url path is /trips
  # - method: POST request
  # - cors: enabled CORS (Cross-Origin Resource Sharing) for browser cross
  #     domain api call
  # - authorizer: authenticate using the AWS IAM role
  create:
    handler: trips/create.main
    events:
      - http:
          path: trips
          method: post
          cors: true
          authorizer: aws_iam

  get:
    # Defines an HTTP API endpoint that calls the main function in get.js
    # - path: url path is /trips/{id}
    # - method: GET request
    handler: trips/get.main
    events:
      - http:
          path: trips/{id}
          method: get
          cors: true
          authorizer: aws_iam

  list:
    # Defines an HTTP API endpoint that calls the main function in list.js
    # - path: url path is /trips
    # - method: GET request
    handler: trips/list.main
    events:
      - http:
          path: trips
          method: get
          cors: true
          authorizer: aws_iam

  update:
    # Defines an HTTP API endpoint that calls the main function in update.js
    # - path: url path is /trips/{id}
    # - method: PUT request
    handler: trips/update.main
    events:
      - http:
          path: trips/{id}
          method: put
          cors: true
          authorizer: aws_iam

  delete:
    # Defines an HTTP API endpoint that calls the main function in delete.js
    # - path: url path is /trips/{id}
    # - method: DELETE request
    handler: trips/delete.main
    events:
      - http:
          path: trips/{id}
          method: delete
          cors: true
          authorizer: aws_iam

# Create our resources with separate CloudFormation templates
resources:
  # DynamoDB
  - ${file(resources/dynamodb-tables.yml)}
  # S3
  - ${file(resources/s3-bucket.yml)}
  # Cognito
  - ${file(resources/cognito-user-pool.yml)}

Here is my resources/dynamodb-tables.yml:

Resources:
  TripsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      # Generate a name based on the stage
      TableName: ${self:custom.stage}-trips
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: tripId
          AttributeType: S
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: tripId
          KeyType: RANGE
      # Set the capacity based on the stage
      ProvisionedThroughput:
        ReadCapacityUnits: ${self:custom.tableThroughput}
        WriteCapacityUnits: ${self:custom.tableThroughput}

It works when I change serverless.yml to:

  environment:
    TripsTableName: ${self:custom.stage}-trips

Any thoughts?

jayair · September 10, 2018, 5:51pm

At a glance, most of your stuff looks okay. Have you deployed yet or are you just running it locally for now?

Also which version of Serverless are you using?

mking · September 12, 2018, 5:55pm

For two or more tables, how would the the environment block and iamrolestatements Resource definition change?

jayair · September 13, 2018, 7:42pm

It would just need more lines in the Resource section:

  # These environment variables are made available to our functions
  # under process.env.
  environment:
    tableName:
      Ref: NotesTable

  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeTable
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      # Restrict our IAM role permissions to
      # the specific table for the stage
      Resource:
        - "Fn::GetAtt": [ NotesTable1, Arn ]
        - "Fn::GetAtt": [ NotesTable2, Arn ]
        - "Fn::GetAtt": [ NotesTable3, Arn ]

Topic		Replies	Views
Use Environment Variables in Lambda Functions Chapter Comments	6	2584	March 11, 2019
Deploy the Resources Repo Chapter Comments	4	683	November 12, 2020
Multi-Repo and Referencing Table names and other resources Chapter Comments	4	541	October 28, 2018
Deployment error	2	476	April 28, 2020
Error: is not authorized to perform: dynamodb:Query :(	4	4534	August 24, 2020

Configure DynamoDB in Serverless

Related Topics