Simplify cross-account stream processing with AWS Lambda and Amazon DynamoDB

Organizations typically use a multi-account structure for safety and isolation. Nonetheless, along with your Amazon DynamoDB tables now in a single account, you may must course of their stream occasions in one other. Till just lately, this meant routing via Amazon Kinesis Knowledge Streams or constructing customized relay infrastructure with cross-account AWS Id and Entry Administration (IAM) roles, including undesirable complexity. Useful resource-based insurance policies for Amazon DynamoDB Streams now helps you keep away from these workarounds. Your AWS Lambda features can instantly eat streams throughout accounts with no customized infrastructure required.

DynamoDB is a serverless, absolutely managed, distributed NoSQL database with single-digit millisecond efficiency at scale. You possibly can construct fashionable, high-performance purposes with out managing infrastructure. Certainly one of its key options is DynamoDB Streams, which captures knowledge modifications in close to actual time. This functionality helps use circumstances reminiscent of audit logging, search indexing, cross-Area replication, anomaly detection, and real-time analytics.

Lambda is a serverless compute service you should use to run code with out provisioning or managing servers. Lambda integrates with DynamoDB Streams, so you’ll be able to mechanically set off features in response to desk updates. This integration is useful to be used circumstances like knowledge replication, materialized views, analytics pipelines, and event-driven architectures.

On this submit, we discover the best way to use resource-based insurance policies with DynamoDB Streams to allow cross-account Lambda consumption. We give attention to a standard sample the place utility workloads dwell in remoted accounts, and stream processing occurs in a centralized or analytics account.

Advantages of cross-account Lambda with DynamoDB streams

With resource-based insurance policies for DynamoDB streams, you’ll be able to grant a Lambda operate in a single AWS account direct entry to learn from a DynamoDB stream in one other account. No customized relay infrastructure is required.

With this function, now you can simplify multi-account event-driven architectures, enhance safety, and cut back operational burden. Lambda manages the ingestion, filtering, supply, retries, and error dealing with simply because it does with similar account occasion supply mappings.

There are lots of causes you may wish to course of DynamoDB stream occasions in a unique AWS account, reminiscent of in the event you’re working a SaaS providing, operating a centralized analytics pipeline, or managing remoted environments for growth, staging, and manufacturing. Useful resource-based coverage help for DynamoDB streams makes it easy to implement these patterns whereas sustaining sturdy entry management.

This functionality allows a number of architectural patterns:

• Centralized knowledge processing – Route streams from a number of utility accounts to a central analytics or knowledge lake account the place Lambda features carry out aggregation, transformation, and loading into your knowledge warehouse.
• Shared providers – Construct reusable audit logging, compliance monitoring, or notification providers in a devoted account that eat streams from tables throughout your group.
• Multi-tenant architectures – Permit tenant-specific Lambda features in remoted accounts to course of knowledge from a centralized DynamoDB desk, sustaining safety boundaries whereas supporting event-driven workflows.

Resolution overview

The cross-account entry mannequin makes use of two parts:

• Useful resource-based coverage on the DynamoDB stream (within the supply account) – Grants permission to the Lambda execution function within the consuming account
• IAM execution function (within the consuming account) – Permits the Lambda operate to learn from the stream

Once you configure an occasion supply mapping between a Lambda operate and a cross-account DynamoDB stream, entry is granted utilizing the mixed permissions of the stream’s useful resource coverage and the Lambda operate’s execution function. The stream’s resource-based coverage authorizes the Lambda execution function, and the execution function supplies the precise permissions wanted to learn and course of stream information.

For instance how this works, take into account a SaaS supplier that gives doc processing providers to enterprise prospects. Every buyer operates in a devoted AWS account for improved isolation and safety. When a doc is processed, the tenant writes a document to a DynamoDB desk with DynamoDB streams enabled. The SaaS providing workforce needs to combination these information in a central AWS account for billing and analytics. With cross-account Lambda and DynamoDB streams, this integration turns into easy and absolutely managed.

The next diagram illustrates the structure.

On this submit, we arrange a Lambda operate in Account A to course of DynamoDB stream occasions from a desk in Account B. We create the Lambda execution function first, then arrange the DynamoDB desk with streams, configure the cross-account permissions, and at last join them with an occasion supply mapping.

The next instructions use variables to make the setup reusable and cut back handbook copying of Amazon Useful resource Names (ARNs) between steps.

Stipulations

To deploy this answer, you have to have two AWS accounts and permission to create the mandatory assets. This answer makes use of the AWS Command Line Interface (CLI), which you’ll set up utilizing these steps.

Outline variables

Use the next code to outline the variables:

# AWS Area
REGION=us-east-1

# Useful resource names
TABLE_NAME=Orders
ROLE_NAME=CrossAccountStreamProcessor
FUNCTION_NAME=ProcessOrders

Create Lambda execution function in Account A

In Account A, create an IAM function that can function the Lambda execution function:

ROLE_ARN="$(aws iam create-role 
--role-name "$ROLE_NAME" 
--assume-role-policy-document '{
  "Model": "2012-10-17",
  "Assertion": [{
    "Effect": "Allow",
    "Principal": {"Service": "lambda.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}' 
--query 'Position.Arn' 
--output textual content)"


aws iam attach-role-policy 
--role-name “$ROLE_NAME” 
--policy-arn arn:aws:iam::aws:coverage/service-role/AWSLambdaDynamoDBExecutionRole

echo "Position ARN: $ROLE_ARN"

Create Lambda operate in Account A

Create the Lambda operate code:

cat > index.py << 'EOF'
import json

def handler(occasion, context):
    print(json.dumps(occasion, indent=2))
    return {'statusCode': 200}
EOF

Bundle the code:

zip operate.zip index.py

Create the Lambda operate:

aws lambda create-function 
--function-name “$FUNCTION_NAME” 
--runtime python3.12 
--role “$ROLE_ARN” 
--handler index.handler 
--zip-file fileb://operate.zip 
--region “$REGION”

Create DynamoDB desk with streams in Account B

In Account B, create a DynamoDB desk with streams enabled:

STREAM_ARN="$(aws dynamodb create-table 
--table-name "$TABLE_NAME" 
--attribute-definitions AttributeName=OrderId,AttributeType=S 
--key-schema AttributeName=OrderId,KeyType=HASH 
--billing-mode PAY_PER_REQUEST 
--stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES 
--region "$REGION" 
--query 'TableDescription.LatestStreamArn' 
--output textual content)"


echo "Stream ARN: $STREAM_ARN"

Connect resource-based coverage to stream in Account B

In Account B, connect a resource-based coverage to your DynamoDB stream.First, create the JSON coverage:

cat > stream-policy.json << EOF
{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Sid": "AllowCrossAccountStreamAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "$ROLE_ARN"
      },
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator"
      ],
      "Useful resource": "*"
    }
  ]
}
EOF

Subsequent, apply the coverage:

aws dynamodb put-resource-policy 
  --resource-arn "$STREAM_ARN" 
  --policy file://stream-policy.json 
  --region "$REGION"

Create occasion supply mapping in Account A

In Account A, create an occasion supply mapping between your Lambda operate and the cross-account stream:

aws lambda create-event-source-mapping 
--function-name “$FUNCTION_NAME” 
--event-source-arn “$STREAM_ARN” 
--starting-position LATEST 
--region “$REGION”

Issues

Take note the next when deploying this answer:

• Each the DynamoDB desk and Lambda operate should be in the identical AWS Area
• Normal DynamoDB Streams and Lambda pricing applies; there aren’t any further expenses for cross-account entry
• The stream’s resource-based coverage makes use of the Lambda execution function ARN because the principal for fine-grained entry management
• The stream’s resource-based coverage helps customary IAM coverage options, together with circumstances and coverage variables
• Be certain to incorporate the next required actions in your coverage: dynamodb:DescribeStream, dynamodb:GetRecords, and dynamodb:GetShardIterator
• This function works with each new and present DynamoDB tables with streams enabled
• This function doesn’t help Amazon managed keys

Cleansing up

To keep away from incurring future expenses, take away the assets created on this walkthrough:

1. In Account A, delete the assets you created:

   aws lambda list-event-source-mappings 
   --function-name “$ProcessOrders” 
   --region “$REGION”
   
   aws lambda delete-event-source-mapping 
   --uuid  
   --region “$REGION”

   Delete the Lambda operate:

   aws lambda delete-function 
   --function-name “$ProcessOrders” 
   --region “$REGION”

   Delete the IAM function:

   aws iam detach-role-policy 
   --role-name “$ROLE_NAME” 
   --policy-arn arn:aws:iam::aws:coverage/service-role/AWSLambdaDynamoDBExecutionRole
   
   aws iam delete-role 
   --role-name “$ROLE_NAME”

2. In Account B, delete the DynamoDB desk:

   aws dynamodb delete-table 
   --table-name “$TABLE_NAME”

Conclusion

Useful resource-based coverage help for DynamoDB streams offers you a strong new solution to construct event-driven methods throughout AWS account boundaries. With this function, you’ll be able to create safe, scalable pipelines with out writing customized integration logic. This answer will help in the event you’re operating a SaaS providing, consolidating logs, or processing change knowledge centrally.

DynamoDB streams with Lambda supplies a managed, dependable path for real-time stream processing. Begin constructing with cross-account Lambda and DynamoDB streams in the present day and simplify your event-driven structure.

Concerning the authors