Online data flow описывает синхронные потоки данных в системе — запросы от клиентов через API Gateway к микросервисам и обратно. Используется для операций, требующих немедленного ответа.
Архитектура
Ingress NGINX] Gateway[API Gateway
Envoy Proxy] end subgraph "Service Layer" Identity[Identity Service
gRPC] Credential[Credential Service
gRPC] Auth[Auth Service
gRPC] Account[Account Service
gRPC] end subgraph "Data Layer" PostgresIdentity[(PostgreSQL
Identity DB)] PostgresCredential[(PostgreSQL
Credential DB)] RedisCache[(Redis
Cache & Sessions)] end WebApp -->|HTTPS/REST| LB MobileApp -->|HTTPS/REST| LB CLI -->|HTTPS/REST| LB LB --> Gateway Gateway -->|gRPC| Identity Gateway -->|gRPC| Credential Gateway -->|gRPC| Auth Gateway -->|gRPC| Account Identity --> PostgresIdentity Identity --> RedisCache Credential --> PostgresCredential Credential --> RedisCache Auth --> RedisCache Account --> PostgresIdentity style Gateway fill:#4A90E2 style PostgresIdentity fill:#51B749 style RedisCache fill:#DC382C
Типы online потоков
1. REST API (Public)
Протокол: HTTP/1.1, HTTP/2 Формат: JSON Аутентификация: JWT Bearer tokens
Примеры endpoints:
1 2 3 4 5 6 7 | |
Request/Response Example:
1 2 3 4 5 6 7 8 9 10 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
2. gRPC (Internal)
Протокол: HTTP/2 with Protocol Buffers Аутентификация: Service-to-service mutual TLS (планируется)
Service Definitions:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
Python Client Example:
1 2 3 4 5 6 7 8 9 10 11 | |
Детальные flow patterns
Pattern 1: User Registration
{username, email, password} Note over Gateway: 1. Validate request body Note over Gateway: 2. Rate limit check Gateway->>Identity: CreateUser(username) Identity->>PostgresI: BEGIN TRANSACTION Identity->>PostgresI: INSERT INTO users
(id, username, status) Identity->>Outbox: INSERT INTO outbox_events
(UserCreatedEvent) Identity->>PostgresI: COMMIT Identity-->>Gateway: User{id, username, status} Gateway->>Credential: SetPassword(user_id, password) Note over Credential: Hash password with Argon2id Credential->>PostgresC: BEGIN TRANSACTION Credential->>PostgresC: INSERT INTO password_credentials
(user_id, password_hash) Credential->>Outbox: INSERT INTO outbox_events
(PasswordSetEvent) Credential->>PostgresC: COMMIT Credential-->>Gateway: PasswordCredential{user_id} Gateway->>Identity: AttachIdentifier(user_id, email, challenge_id) Identity->>PostgresI: INSERT INTO identifiers
(user_id, type, value, verified_at) Identity-->>Gateway: Identifier{id, verified_at} Gateway->>Gateway: CreateSession + GenerateTokens Gateway->>Redis: SET session:{id} {data} EX 2592000 Gateway-->>Client: 201 Created
{user, tokens} Note over Outbox: Background worker publishes events to Kafka
Характеристики: - Latency: ~200-500ms (зависит от Argon2id hashing) - Transactions: 2 отдельные БД транзакции (Identity, Credential) - Consistency: Eventually consistent через events
Pattern 2: Authentication (Login)
{username, password} Gateway->>Identity: GetUserByUsername(username) Identity->>Redis: GET user:username:{username} alt Cache hit Redis-->>Identity: Cached user data else Cache miss Identity->>PostgresI: SELECT * FROM users
WHERE username = ? PostgresI-->>Identity: User record Identity->>Redis: SETEX user:username:{username}
TTL=300 end Identity-->>Gateway: User{id, status} alt User not found or disabled Gateway-->>Client: 401 Unauthorized end Gateway->>Credential: VerifyPassword(user_id, password) Note over Credential: Load password_hash from DB Credential->>PostgresC: SELECT password_hash
FROM password_credentials
WHERE user_id = ? PostgresC-->>Credential: password_hash Note over Credential: Argon2id verification
(constant-time) alt Invalid password Credential-->>Gateway: {is_valid: false} Gateway-->>Client: 401 Unauthorized end Credential-->>Gateway: {is_valid: true} Gateway->>Auth: CreateSession(user_id, metadata) Auth->>Redis: SET session:{id} {user_id, ...} EX 2592000 Auth->>Auth: GenerateTokens(user_id, session_id) Auth-->>Gateway: {access_token, refresh_token} Gateway-->>Client: 200 OK
{access_token, refresh_token}
Характеристики: - Latency: ~150-300ms (с кешем), ~300-600ms (без кеша) - Cache strategy: Username → User lookup кешируется на 5 минут - Security: Constant-time password verification
Pattern 3: Authenticated API Request
Authorization: Bearer JWT" Note over Gateway: "1. Extract JWT from header" Note over Gateway: "2. Verify signature (JWKS)" Note over Gateway: "3. Check exp, iss, aud" Note over Gateway: "4. Extract user_id from sub" Gateway->>Service: "GET /users/me
x-user-id, x-session-id" Service->>Redis: "GET user key" alt Cache hit Redis-->>Service: Cached user data else Cache miss Service->>PostgresS: SELECT user WHERE id PostgresS-->>Service: User record Service->>Redis: "SETEX user key TTL=300" end Service-->>Gateway: User data Gateway-->>Client: "200 OK + user data"
Характеристики:
- Latency: ~10-50ms (с кешем), ~50-150ms (без кеша)
- Cache TTL: 5 минут для user data
- Validation: JWT валидируется на Gateway, сервис доверяет x-user-id
Pattern 4: Data Mutation with Cache Invalidation
Authorization Bearer JWT
body username" Gateway->>Service: "PUT /users/id
x-user-id" Service->>PostgresS: BEGIN TRANSACTION Service->>PostgresS: "UPDATE users SET username WHERE id" Service->>PostgresS: COMMIT Note over Service: Invalidate caches Service->>Redis: "DEL user key" Service->>Redis: "DEL user username key" Service-->>Gateway: Updated user data Gateway-->>Client: "200 OK + updated user"
Характеристики: - Cache invalidation: Удаление всех связанных cache keys - Consistency: Write-through pattern (сначала DB, потом cache)
Performance Optimizations
1. Connection Pooling
PostgreSQL (через PgBouncer):
1 2 3 4 5 6 7 8 | |
Redis:
1 2 3 4 5 6 7 8 | |
2. Caching Strategy
Multi-level cache:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | |
Cache invalidation patterns: - Write-through: DB write → cache invalidate - TTL-based: Auto-expire после N секунд - Event-based: Kafka event → invalidate cache
3. Request Batching (DataLoader pattern)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
4. Read Replicas (planned)
Read/Write)] Replica1[(PostgreSQL Replica 1
Read-only)] Replica2[(PostgreSQL Replica 2
Read-only)] Service -->|Writes| Primary Service -->|Reads| Replica1 Service -->|Reads| Replica2 Primary -.->|Replication| Replica1 Primary -.->|Replication| Replica2 style Primary fill:#51B749 style Replica1 fill:#90EE90 style Replica2 fill:#90EE90
Observability
Request Tracing
OpenTelemetry spans:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Metrics
Prometheus metrics:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
Error Handling & Retry
Circuit Breaker
1 2 3 4 5 6 7 | |
Retry with Exponential Backoff
1 2 3 4 5 6 7 8 9 | |
Связанные страницы
- Event Flow — асинхронные потоки данных через Kafka
- Data Ownership — кто владеет какими данными
- API Design Principles — REST API guidelines