A guide to architecting resilient payment systems that scale and the infrastructure decisions that determine whether they survive production.
Building a payment system that handles millions of transactions while maintaining security, compliance, and cost efficiency isn't just about moving money it's about mastering system architecture principles that apply across any complex, mission-critical platform.
If you are looking to adopt agents into your payment infrastructure, read the
5 Stages of Deploying Agent-Based Payment Systems →
→ Before you design — find out what your current infrastructure is costing you Run the free Payment Risk Estimator →
What Are the Key Components of a Payment System Architecture?
A robust payment system architecture consists of five critical layers that work together to process transactions securely and reliably:
API Gateway Layer
Your entry point handles authentication, rate limiting, and request validation. This layer protects your system from malicious traffic while ensuring legitimate requests flow smoothly through your infrastructure.
→ The free Infrastructure Readiness Score assesses your API gateway configuration against PCI DSS 4.0 requirements — no login required Run the free Infrastructure Readiness Score →
Orchestration Layer
Step Functions or similar workflow engines coordinate complex multi-step processes. Instead of services calling each other directly creating brittle chains, centralised orchestration provides visibility, retry logic, and failure recovery.
Every payment involves 5-7 steps. Each step can fail. Orchestration built around the assumption that it won't is orchestration that creates incidents.
Business Logic Layer
Lambda functions or containerised services handle the core payment processing logic, fraud detection, and business rules. This layer transforms requests into actionable business operations.
Serverless vs containers decision:
Lambda: Spiky traffic, millisecond fraud decisions, webhook handlers
Containers (ECS Fargate): Continuous settlement streams, large model files, legacy SOAP integrations
Hybrid: Use both based on specific service requirements
Data Persistence Layer
DynamoDB for high-throughput operations, RDS for complex queries, and specialised storage for audit trails. Your data architecture must handle both transactional consistency and analytical workloads.
DynamoDB: High-speed transaction state, idempotency keys
RDS: Compliance reporting, reconciliation, complex joins
ElastiCache: Fraud scores, processor availability, rate limits
S3: Audit log archival, 7-year retention
Analytics and Monitoring Layer
Real-time event streaming through Kinesis feeds into analytics platforms, while comprehensive monitoring ensures system health and regulatory compliance. Build this from day one — not as an afterthought.
How Do You Handle Payment System Failures and Retries?
Payment systems fail in predictable patterns. Your architecture must handle each failure type appropriately and agent-based payment systems introduce failure modes that human-initiated flows never encounter.
Smart Retry Strategies
Not all failures should trigger retries:
Retriable: Network timeouts, 5xx server errors, temporary service unavailability
Non-retriable: Invalid card numbers, insufficient funds, authentication failures
Rate-limited: Implement circuit breakers and adaptive retry delays
For agent-based systems, the retry logic is more complex, an agent retries at machine speed, without human judgment between attempts. Your circuit breaker needs to fire before the agent creates a cascade.
Idempotency Keys
Every payment request must include a unique idempotency key to prevent duplicate charges. Your system should store these keys with request outcomes, ensuring that retried requests return the same result without reprocessing.
At agent execution speed, a 0.1% duplicate rate on 100K monthly transactions is 100 disputes per month — £2,500-5,000/month in overhead before customer trust damage.
Table: idempotency_keys
Partition key: idempotency_key
Attributes: transaction_id, result, status, created_at
TTL: 24 hours
Conditional write: attribute_not_exists(idempotency_key)
→ The Sync Your Cloud Idempotency Safety Rails tool validates your retry logic against agent-speed failure modes Access with Sync membership →
State Machine Design
Model your payment flow as a finite state machine with explicit transitions. Each state should have clear error handling paths and timeout mechanisms:
INITIATED → VALIDATED → AUTHORISED → CAPTURED → SETTLED → RECONCILED
Invalid transitions: SETTLED → AUTHORISED
Compensation flows: AUTHORISED → VOID (on capture failure)
This approach makes complex flows easier to reason about and debug and makes it impossible for a payment to enter an undefined state.
What Does Payment System Security Architecture Look Like?
Security in payment systems requires defence in depth across multiple layers. A single layer failing should never result in a breach.
Edge Protection with WAF
Web Application Firewalls filter malicious traffic before it reaches your application layer. Configure rules for:
SQL injection and XSS patterns
Rate limiting (100 requests/minute per IP)
Geographic restrictions where applicable
Custom rule: block requests with credit card patterns in URLs (PCI violation prevention)
API Security
Implement OAuth 2.0 with proper token validation. Rate limiting prevents abuse. For agent-based systems, every agent instance needs a scoped identity credential — not a shared API key across execution contexts.
IAM role per agent type — not per environment
Conditions: aws:PrincipalTag/AgentType matches payment scope
Scope: specific processor ARNs only
Data Encryption
Encrypt sensitive data both in transit and at rest. Use KMS with separate keys per environment and per data classification. Never store card data in plain text — use tokenisation to minimise PCI scope.
KMS keys: separate per environment (dev/staging/prod)
separate per data classification (PII, PCI, general)
Rotation: automatic annual rotation enabled
Network Isolation
Deploy sensitive components in private VPCs:
Public subnets: API Gateway, ALB only
Private subnets: All payment agents, databases
Isolated subnets: PCI-sensitive operations (tokenisation)
VPC endpoints keep AWS service traffic private — even if an agent is compromised, payment data never traverses the public internet.
→ Assess your security architecture against PCI DSS 4.0 requirements Run the free Infrastructure Readiness Score →
How Much Does It Cost to Build a Payment System?
Understanding payment system costs helps you make informed architectural decisions. AWS costs change — always verify current pricing directly with AWS.
Transaction Processing Costs
| Stage |
Cost per Transaction |
| Early (speed over cost) |
£0.006-£0.02 |
| Optimised scale |
£0.001-£0.004 |
| Highly optimised |
£0.000107 |
Infrastructure Costs by Volume
| Volume |
Monthly AWS Cost |
| 0-100K transactions |
£200-400 |
| 100K-1M transactions |
£800-1,500 |
| 1M-10M transactions |
£3,000-6,000 |
| 10M+ transactions |
£10,000-30,000 |
Hidden Costs Most Teams Miss
PCI DSS compliance infrastructure: £800-1,200/month overhead
Emergency PCI remediation when controls drift: £15,000-50,000
Duplicate settlement incidents: £15,000-40,000 per incident
Regional failure with orphaned transactions: £10,000-30,000
→ Calculate what your current infrastructure is actually costing you — free, 60 seconds Run the free Payment Risk Estimator →
Cost Optimisation Strategies
Right-size resources based on actual usage patterns
Implement data lifecycle policies — move old data to S3 Infrequent Access, then Glacier
Use reserved capacity for predictable workloads, Lambda for spiky traffic
Tag everything for cost attribution — when your CFO asks "how much does fraud detection cost per transaction?" you need the answer immediately
How Do You Ensure Payment System Compliance?
Compliance isn't a checkbox it's an architectural constraint that shapes your system design from day one. Retrofitting compliance is 3-5x more expensive than building it in.
PCI DSS 4.0 Compliance Architecture
PCI DSS 4.0 introduced continuous monitoring requirements. For agent-based payment systems running 24 hours a day, this is an architectural requirement — not an administrative one.
63 individual controls across 6 control objectives need to be mapped to your specific infrastructure and monitored continuously:
Secure Network & Systems: 2 requirements, 12 controls
Protect Account Data: 2 requirements, 14 controls
Vulnerability Management: 2 requirements, 8 controls
Strong Access Control: 3 requirements, 16 controls
Monitor & Test Networks: 2 requirements, 9 controls
Security Policy: 1 requirement, 4 controls
Controls that fail most often in agent-based environments:
Requirement 10 (Audit Logging): Agent volumes overwhelm log retention policies
Requirement 6 (Secure Development): Agent config changes bypass change management
Requirement 8 (Identity Management): Agent identities share credentials
→ Run the free PCI DSS Gap Analysis — all 63 controls, no login required Run the free Infrastructure Readiness Score →
Audit Trail Requirements
Every transaction must have a complete audit trail. Use structured logging with a defined schema — not scattered log.info() calls:
{
"transaction_id": "txn_abc123",
"agent_id": "fraud-agent-v2",
"decision": "approved",
"authorisation_scope": "payment:fraud:read",
"timestamp": "2026-05-31T09:14:23Z",
"amount_pence": 4999,
"currency": "GBP"
}
Separate audit log stream from operational logs. 7-year retention for financial records. Different access controls. Different integrity requirements.
Data Retention and Privacy
Implement automated data lifecycle management to meet regulatory retention requirements while respecting GDPR's right to be forgotten. These requirements conflict design the resolution explicitly before you go live.
What Are the Best Practices for Payment System Monitoring?
Effective monitoring turns invisible system behaviour into actionable insights. Build for audit, not just debugging.
Key Metrics to Track
Payment success rate: Target >99.5% — alert below 99%
Authorisation latency P99: Target <800ms — alert above 1 second
Agent error rate: Alert above 0.5%
DLQ message depth: Alert immediately on any message
Cost per transaction: Alert on 20% increase
Compliance control drift: Alert immediately
Alerting Strategy
Alert on business impact, not just technical metrics:
PagerDuty (immediate): Success rate drops, DLQ messages, latency spikes, compliance drift
Slack (warning): Cost increases, traffic anomalies, connection pool pressure
A DLQ with messages means a payment is stuck right now. If you don't have DLQ depth wired into immediate alerting, you won't know until a customer calls.
→ The Sync Your Cloud Observability Pack provides pre-configured monitoring patterns for payment systems Access with Sync membership →
Observability Architecture
Standard observability answers: why did this break?
Audit observability answers: can you demonstrate, for any transaction, the complete chain of decisions that led to it?
These are different requirements. Build both from day one:
Operational logs: 30-day retention, debugging
Audit logs: 7-year retention, compliance
X-Ray distributed tracing: end-to-end payment execution chain
CloudTrail: all infrastructure changes affecting payment agents
How Do You Scale a Payment System Architecture?
Horizontal Scaling Patterns
Design stateless services that scale independently. Use event-driven architectures to decouple services and handle traffic spikes gracefully.
For agent-based systems, scaling introduces a specific challenge — more agent instances means more concurrent payment executions, which means higher idempotency table write throughput and more aggressive spending control validation.
Database Scaling Strategies
0-100K/month: DynamoDB on-demand, RDS t3.small
100K-1M/month: DynamoDB provisioned (25 WCU/50 RCU), RDS r5.large + read replica
1M-10M/month: DynamoDB auto-scaling (100-500 WCU), RDS r5.xlarge multi-AZ
10M+/month: DynamoDB global tables, RDS Aurora global
Geographic Distribution
Deploy payment processing closer to your users to reduce latency. Consider:
Regulatory requirements mandating data residency (UK FPS requires UK data residency)
PCI DSS requirements for cross-region data transfer
Multi-region failover — design active-active from the start, not active-passive
→ Build your multi-region failure playbook before you need it Run the free Failure Playbook →
What Technologies Should You Use for Payment Systems?
Serverless vs Containers
| Use Case |
Recommended |
| Fraud detection (spiky, ms decisions) |
Lambda |
| Settlement (continuous, predictable) |
ECS Fargate |
| Webhook handlers (unpredictable volume) |
Lambda |
| ML fraud agents (large model files) |
ECS Fargate |
| High-volume event processing |
Kinesis + Lambda |
Database Selection
DynamoDB: High-throughput transactional data, idempotency keys, session state
RDS PostgreSQL: Compliance reporting, reconciliation, complex analytical queries
ElastiCache Redis: Fraud scores (5-min TTL), processor status (1-min TTL), rate limits
Message Processing
SQS Standard: Primary message transport for agent communication
SQS FIFO: Ordered operations (settlement sequences)
EventBridge: Service-to-service routing without tight coupling
Kinesis: High-throughput event streaming for analytics
How Do You Test Payment Systems?
Testing Strategies
Unit tests: Business logic and edge cases — especially idempotency logic
Integration tests: Service interactions and data flow
End-to-end tests: Complete user journeys including failures
Chaos engineering: Kill random agents mid-transaction, verify recovery
Chaos engineering for payment systems isn't optional — it's how you discover whether your compensation flows actually work before production does it for you.
Test Data Management
Use synthetic test data that mimics production patterns without exposing real customer information. Implement test payment gateways that simulate:
Successful authorisation then settlement failure
Network timeout after authorisation
Duplicate request with same idempotency key
Regional failover mid-transaction
What Are Common Payment System Architecture Mistakes?
Technical Mistakes
Synchronous agent chains: API → Fraud Agent waits → Auth Agent waits → Settlement waits. Total latency equals sum of all agents. Single agent failure breaks entire flow. Fix: Step Functions orchestration with parallel execution where possible.
Missing idempotency: The most expensive mistake at scale. Duplicate charges at 0.1% rate on 1M monthly transactions is 1,000 disputes per month.
No DLQ monitoring: Silent failure killer. Every DLQ message is a stuck payment. If it's not wired to immediate alerting, you find out from customers.
Undersized database connections: 500 Lambda instances trying to connect to an RDS instance configured for 100 max connections. Fix: RDS Proxy for connection pooling.
Storing sensitive data in logs: PCI violation waiting to happen. Implement log sanitisation at agent level. Use CloudWatch Logs data protection policies.
Business Mistakes
Compliance as an afterthought: Retrofitting PCI DSS compliance costs 3-5x more than building it in from the start.
No cost attribution: When your CFO asks what fraud detection costs per transaction, "we don't track that" is the wrong answer.
Over-engineering early: Complex multi-region active-active before you have 10K monthly transactions. Build for the scale you have, design for the scale you're heading to.
How Do You Migrate to a New Payment System Architecture?
Strangler Fig Pattern
Gradually replace old system components while keeping the system operational. Route new features to the new architecture while maintaining existing functionality. Never big-bang migrate a payment system.
Data Migration Strategies
Dual-write: Write to both old and new systems during transition — verify consistency before cutover
Event replay: Rebuild new system state from historical events
Gradual cutover: Migrate by customer segment, 5% → 25% → 100%
Risk Mitigation
Feature flags for instant rollback of any migration step
Monitor business metrics — success rate, latency, dispute rate — not just technical metrics
Detailed rollback procedures tested before migration starts
Never migrate during high-traffic periods
Start With What Your Infrastructure Is Costing You
Before redesigning your payment architecture, understand what your current setup is actually costing you — not just your AWS bill, but the hidden costs of compliance gaps, manual reconciliation, and infrastructure that wasn't built for agent-based execution.
Free tools — no login required:
Calculate Your Payment Infrastructure Risk → 60 seconds. Quantifies your monthly risk exposure based on your payment service count.
Run the Agentic Readiness Assessment → 21 questions. Scored gap analysis across 7 dimensions of agent payment infrastructure readiness.
Build Your Failure Playbook → Documented recovery procedures for the failure modes that matter in production.
Need architecture support?
Sync Your Cloud gives engineering teams access to 26 purpose-built tools for AWS payment infrastructure — from PCI DSS gap analysis with all 63 controls, to agent flow simulation, to live AWS account analysis against your actual environment.
Plans from £999/month. Simulation mode available — no execution risk, full decision logs, complete evidence pack for your risk and compliance teams.
Explore Sync Your Cloud →
Sync Your Cloud is the infrastructure readiness platform for engineering teams deploying agent-based payment systems. Built on AWS. Validated for payment infrastructure.
