−
+
Reset
Wheel to zoom · drag to pan
100%
CYBERSIREN — SYSTEM ARCHITECTURE
Event-Driven Microservices over Apache Kafka · 11 Services · 12 Kafka Topics · PostgreSQL + Redis · v2.2
TI FEED LAYER
PhishTank
OpenPhish
URLhaus
MalwareBazaar (hashes)
Abuse.ch ThreatFox
Custom Feeds
TI Feed Sync
SVC-11 · Go · every 6h
fetch · parse · dedup · UPSERT
W: ti_indicators, attachment_library
DATA STORES
PostgreSQL 15
emails, verdicts, campaigns, rules, rule_hits,
enriched_threats, ti_indicators, email_url_ti_matches
attachment_library · current_verdicts VIEW + 5 MVs
MV refresh
Redis 7
dedup · TI cache · aggregator state
rate limiting · SimHash storage
Apache Kafka
12 topics · partition key: email_id
consumer groups per service
retention: 24h–7d
Object Storage
S3 / MinIO · attachment binaries · 90d TTL
EMAIL PIPELINE (KAFKA-CENTRIC)
EMAIL SOURCES
Gmail API
Outlook API
IMAP
REST API Upload
Custom Adapter
Email Ingestion Service
SVC-01 · Go · normalize · dedup · publish
Redis: dedup:{org_id}:{message_id} · DB: api_keys (R)
emails.raw
Email Parser Service
SVC-02 · Go · MIME parse · extract · fan-out
DB W: emails, email_urls, enriched_threats (bare),
attachment_library, email_attachments, email_recipients
analysis.urls
analysis.headers
analysis.attachments
analysis.text
analysis.plans
PARALLEL ANALYSIS (SCATTER)
Each service consumes analysis.* and emits scores.* (key = email_id)
URL Analysis
SVC-03 · Go + Python
TI feed match · enrich · ML
WHOIS / SSL / DNS / XGBoost
R: ti_indicators, enriched_threats · W: enriched_threats
Header Analysis
SVC-04 · Go
auth / reputation / structural
weights: research pending
R: rules, ti_indicators · W: rule_hits
Attachment Scorer
SVC-05 · Go
SHA256 TI hash lookup
entropy + MIME + VirusTotal
R/W: attachment_library
NLP Service
SVC-06 · Python
urgency / intent / impersonation
models: research pending
DB: none (stateless)
scores.url
scores.header
scores.attachment
scores.nlp
analysis.plans
Score Aggregator (gather only)
SVC-07 · Go · stateless workers + Redis state
Collects component scores into single payload · Does NOT compute final score · timeout: 30s
emails.scored
Decision Engine
SVC-08 · Go · score · verdict · campaign
1. Combine component scores (method: research pending)
2. Rule engine adjustments + verdict determination
3. Campaign fingerprint + SimHash (per-org scoped)
4. Campaign history informs future scoring
DB W: emails, verdicts, rule_hits, campaigns (single tx)
emails.verdict
OUTPUT LAYER
Notification Service
SVC-09 · Go · emails.verdict consumer
email / webhook / Slack / Teams
rate limit: 1/campaign/org/hr via Redis
API / Dashboard
SVC-10 · Go REST + React + WebSocket
R: all tables + MVs
W: analyst verdicts, rules, audit_log
LEGEND
Service (microservice)
Kafka Topic
External Source
Data flow (Kafka / direct)
DB/Redis access annotated inside each service box
KEY PROPERTIES
- All topics keyed by email_id: ordering within each topic, NO cross-topic co-location
- Steps 3a–3d run concurrently; Aggregator gathers scores only (no scoring logic)
- Decision Engine: final scoring + verdict + campaign management (method: research pending)
- Campaign history feeds back into scoring of future emails matching same campaign
- Verdicts are append-only (verdicts table + current_verdicts VIEW)
- TI Sync is independent (6h schedule), refreshes all 5 materialized views
- Privacy: email bodies purged after retention; Kafka bounded (24h–7d)
- All services connect to PostgreSQL / Redis / Kafka as shown in Data Stores panel
VERDICT MAPPING (preliminary thresholds, subject to calibration)
0–25 benign 26–50 suspicious 51–75 phishing 76–100 phishing (high) / malware
CyberSiren System Architecture Diagram — ARCH-DIAG-v2.2 — 2026-03-15