Configure DynamoDB in Serverless


#1

Link to chapter - https://serverless-stack.com/chapters/configure-dynamodb-in-serverless.html


#2

OS: darwin
Node Version: 8.11.2
Serverless Version: 1.26.0

Problem is I can’t access env variables with process.env.tableQuestions
Function returns error message:
Invalid table/index name. Table/index names must be between 3 and 255 characters long, and may contain only the characters a-z, A-Z, 0-9, ‘_’, ‘-’, and ‘.’

Here is part of serverless.yml file:

custom:
  stage: ${opt:stage, self:provider.stage}
  tableThroughputs:
    prod: 5
    default: 1
  tableThroughput: ${self:custom.tableThroughputs.${self:custom.stage}, self:custom.tableThroughputs.default}
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true
  dynamodb:
    start:
      port: 8000
      inMemory: true
      migrate: true
      seed: true
      noStart: true
    seed:
      test:
        sources:
          - table: ${env:tableQuestions}
            rawsources: [./offline/seeds/questions.json]

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: eu-central-1
  environment:
    tableQuestions:
      Ref: QuestionsTable
    tableReplays:
      Ref: ReplaysTable

iamRoleStatements:
  - Effect: Allow
    Action:
      - dynamodb:DescribeTable
      - dynamodb:Query
      - dynamodb:Scan
      - dynamodb:GetItem
      - dynamodb:PutItem
      - dynamodb:UpdateItem
      - dynamodb:DeleteItem
    Resource:
      - "Fn::GetAtt": [ QuestionsTable, Arn ]
      - "Fn::GetAtt": [ ReplaysTable, Arn ]

And this is my dynamoDb Resource configuration:

Resources:
    QuestionsTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: questionId
            AttributeType: S
        KeySchema:
          - AttributeName: questionId
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: ${self:custom.tableThroughput}
          WriteCapacityUnits: ${self:custom.tableThroughput}
        TableName: ${self:custom.stage}-questions

    ReplaysTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: userId
            AttributeType: S
        KeySchema:
          - AttributeName: userId
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: ${self:custom.tableThroughput}
          WriteCapacityUnits: ${self:custom.tableThroughput}
        TableName: ${self:custom.stage}-replays

#3

It looks okay. What does process.env.tableQuestions & process.env.tableReplays resolve to in your Lambda if you do a console.log?


#4

Don’t know. Stuck at function invoke, error message:

{
    "statusCode": 500,
    "headers": {
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Credentials": true
    },
    "body": "{\"message\":\"1 validation error detected: Value '[object Object]' at 'tableName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+\"}"
}

I need to check if a problem persits in production enviroment, at the moment I’m trying to invoke functions localy


#5

I can confirm it works in production. But it’s incompatible with serverless-offline plugin


#6

Oh how are you invoking it locally through the serverless-offline plugin?


#7

Yes. I’m running dynamoDb docker container and serverless-offline plugin to test my api


#8

And what’s the command you are running that gave you the error?


#9

I’m fairly new to this, so I may be incorrect. After some debugging, I discovered that the tableName env variable gets assigned the [object Object] value, maybe because it doesn’t evaluate the ‘Ref’ intrinsic function (Not played around with serverless as much but thought this might be helpful).
Anyways, you can resolve the issue by replacing

tableName: Ref NotesTable

with,

tableName: notes-${self:custom.stage}


#10

Hmmm that is weird. If you’ve configured your resources correctly, it should pick it up. The Ref is simply pointing to ${self:custom.stage}-notes.


#11

An issue of this nature has already been posted on the Repo issues: https://github.com/serverless/serverless/issues/3080

Following that issue thread, you can install a plugin serverless-export-env to properly initialize all environment variables with Cloudformation template references.

Again, I’m new to this so I’ve encountered this problem when I use the serverless invoke local rather than serverless-offline plugin. Not sure if it works different for the plugin.


#12

That’s a really old issue. Which version of Serverless Framework are you using?


#13

What is the best way to share Dynamo DB across multiple services. According to Amazon, having a single table is idea, which I can do.

I need to modify different parts of the object, hence difference services that interact with the document differently.

How would you setup the resource in a monorepo?

  1. Separate Resource Service (how does that work in Seed, race condition?)
  2. Each serverless.yml references the same /resources/dynamoDb.yml configuration and import it?

Additionally, do IAM roles for DynamoDB impact all requests in the service? Can you restrict delete actions to the /delete path for example. May need a different user credentials to delete for example?

Sorry still very new to this.


#14

I talked about the resource sharing in the other thread - Organizing Serverless Projects.

But for the IAM roles, it really depends on the level at which you want to control this. You can use Cognito User Groups. Or you can manage these internally in your code if you have some slightly complicated business logic to resolve permissions.


#15

Is there a good documentation resource on configuring DynamoDB in Serverless’s yaml? I’m checking out the docs here: https://serverless.com/framework/docs/providers/aws/guide/resources/ but it’s not very comprehensive


#16

Yeah I listed one in this chapter.

It’s a guide dedicated to DynamoDB - https://www.dynamodbguide.com/.


#17

I’m experiencing the same as @acidfabric and the fix shown by @aaditya-panik is what I had to do to get past it for now. I really dislike the duplicative naming and I’m sure I’m doing something wrong, but I’m not seeing it.

Some background:

  • I read through part 1 in the past 2 weeks
  • I started with part 2 to create a base project and literally just tonight cloned the repo
  • I have made minor modifications to reflect the real-world project that I plan on using it (namely table + field names)
  • I’m using WebStorm as my IDE

Here’s how I invoke the method:

serverless invoke local --function create --path mocks/create-event.json

Here’s the error that gets returned:

      message: '1 validation error detected: Value \'[object Object]\' at \'tableName\' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+',
      code: 'ValidationException',
      time: 2018-09-08T01:28:37.270Z,

When I do console.log(process.env.TripsTableName);, I simply get:
[object Object]

Here is my serverless.yml:

service: routinator

# Use the serverless-webpack plugin to transpile ES6
plugins:
  - serverless-webpack
  - serverless-offline

# serverless-webpack configuration
# Enable auto-packing of external modules
custom:
  # Our stage is based on what is passed in when running serverless
  # commands. Or fallsback to what we have set in the provider section.
  stage: ${opt:stage, self:provider.stage}
  # Set our DynamoDB throughput for prod and all other non-prod stages.
  tableThroughputs:
    prod: 5
    default: 1
  tableThroughput: ${self:custom.tableThroughputs.${self:custom.stage}, self:custom.tableThroughputs.default}
  # Load our webpack config
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: us-east-1

  # These environment variables are made available to our functions
  # under process.env.
  environment:
    TripsTableName:
      Ref: TripsTable

  # 'iamRoleStatement' defines the permission policy for the Lambda function.
  # In this case Lambda functions are granted with permissions to access DynamoDB.
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeTable
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      # Restrict our IAM role permissions to
      # the specific table for the stage
      Resource:
        - "Fn::GetAtt": [ TripsTable, Arn ]

functions:
  # Defines an HTTP API endpoint that calls the main function in create.js
  # - path: url path is /trips
  # - method: POST request
  # - cors: enabled CORS (Cross-Origin Resource Sharing) for browser cross
  #     domain api call
  # - authorizer: authenticate using the AWS IAM role
  create:
    handler: trips/create.main
    events:
      - http:
          path: trips
          method: post
          cors: true
          authorizer: aws_iam

  get:
    # Defines an HTTP API endpoint that calls the main function in get.js
    # - path: url path is /trips/{id}
    # - method: GET request
    handler: trips/get.main
    events:
      - http:
          path: trips/{id}
          method: get
          cors: true
          authorizer: aws_iam

  list:
    # Defines an HTTP API endpoint that calls the main function in list.js
    # - path: url path is /trips
    # - method: GET request
    handler: trips/list.main
    events:
      - http:
          path: trips
          method: get
          cors: true
          authorizer: aws_iam

  update:
    # Defines an HTTP API endpoint that calls the main function in update.js
    # - path: url path is /trips/{id}
    # - method: PUT request
    handler: trips/update.main
    events:
      - http:
          path: trips/{id}
          method: put
          cors: true
          authorizer: aws_iam

  delete:
    # Defines an HTTP API endpoint that calls the main function in delete.js
    # - path: url path is /trips/{id}
    # - method: DELETE request
    handler: trips/delete.main
    events:
      - http:
          path: trips/{id}
          method: delete
          cors: true
          authorizer: aws_iam

# Create our resources with separate CloudFormation templates
resources:
  # DynamoDB
  - ${file(resources/dynamodb-tables.yml)}
  # S3
  - ${file(resources/s3-bucket.yml)}
  # Cognito
  - ${file(resources/cognito-user-pool.yml)}

Here is my resources/dynamodb-tables.yml:

Resources:
  TripsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      # Generate a name based on the stage
      TableName: ${self:custom.stage}-trips
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: tripId
          AttributeType: S
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: tripId
          KeyType: RANGE
      # Set the capacity based on the stage
      ProvisionedThroughput:
        ReadCapacityUnits: ${self:custom.tableThroughput}
        WriteCapacityUnits: ${self:custom.tableThroughput}

It works when I change serverless.yml to:

  environment:
    TripsTableName: ${self:custom.stage}-trips

Any thoughts?


#18

At a glance, most of your stuff looks okay. Have you deployed yet or are you just running it locally for now?

Also which version of Serverless are you using?


#19

For two or more tables, how would the the environment block and iamrolestatements Resource definition change?


#20

It would just need more lines in the Resource section:

  # These environment variables are made available to our functions
  # under process.env.
  environment:
    tableName:
      Ref: NotesTable

  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeTable
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      # Restrict our IAM role permissions to
      # the specific table for the stage
      Resource:
        - "Fn::GetAtt": [ NotesTable1, Arn ]
        - "Fn::GetAtt": [ NotesTable2, Arn ]
        - "Fn::GetAtt": [ NotesTable3, Arn ]