feat: add end-to-end observability (metrics, health, mobile RUM, dashboards, alerts)
Run Tests on Branches / Detect Changes (push) Successful in 10s
Run Tests on Branches / Backend Tests (push) Failing after 20s
Run Tests on Branches / Frontend Tests (push) Has been skipped
Run Tests on Branches / Frontend Mobile Tests (push) Failing after 20s

- Backend: OTel metrics via OTLP -> Alloy -> Prometheus (OTEL_METRICS_EXPORTER=otlp)
- New business metrics (src/metrics.js): searches, CIMA latency/errors, cache
  hits/misses, logins, rate-limits, pharmacy writes/links, push sent/failed,
  DB + Redis timings/errors, HTTP req count/duration, heartbeat
- Backend health endpoints /healthz and /readyz
- Mobile (Expo): Grafana Faro RUM via @grafana/faro-react-native
- redis/postgres exporters in docker-compose + Prometheus scrape jobs
- Grafana dashboards (backend, datastores, mobile RUM, overview)
- Prometheus alert rules (farmafinder_*) -> existing Alertmanager (Telegram)
- Design/spec saved to docs/superpowers/specs/
This commit is contained in:
Antoni Nuñez Romeu
2026-07-13 15:57:52 +02:00
parent 2713be85a3
commit 839c64c12a
16 changed files with 907 additions and 27 deletions
@@ -0,0 +1,145 @@
{
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"id": 1, "title": "Request rate (req/s)", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum(rate(http_requests_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "req/s" }
]
},
{
"id": 2, "title": "Requests by status class", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum by (status_class) (rate(http_requests_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "{{status_class}}" }
]
},
{
"id": 3, "title": "5xx error rate", "type": "stat",
"gridPos": { "h": 6, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum(rate(http_requests_total{status_class=\"5xx\",job=\"farmafinder-backend\"}[5m])) / sum(rate(http_requests_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "5xx ratio" }
],
"fieldConfig": { "defaults": { "unit": "percentunit", "thresholds": { "steps": [ { "color": "green", "value": 0 }, { "color": "yellow", "value": 0.01 }, { "color": "red", "value": 0.05 } ] } } }
},
{
"id": 4, "title": "HTTP p95 latency (route)", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 6 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_ms_bucket{job=\"farmafinder-backend\"}[5m])))", "legendFormat": "{{route}} p95" }
],
"fieldConfig": { "defaults": { "unit": "ms" } }
},
{
"id": 5, "title": "Medicine searches / CIMA requests", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 6 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum(rate(medicine_searches_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "searches/s" },
{ "expr": "sum(rate(cima_requests_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "cima req/s" }
]
},
{
"id": 6, "title": "Cache hit ratio", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 0, "y": 12 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum(rate(cache_hits_total{job=\"farmafinder-backend\"}[5m])) / (sum(rate(cache_hits_total{job=\"farmafinder-backend\"}[5m])) + sum(rate(cache_misses_total{job=\"farmafinder-backend\"}[5m])))", "legendFormat": "hit ratio" }
],
"fieldConfig": { "defaults": { "unit": "percentunit" } }
},
{
"id": 7, "title": "CIMA API errors", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 8, "y": 12 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum(rate(cima_requests_total{status=\"error\",job=\"farmafinder-backend\"}[5m]))", "legendFormat": "errors/s" }
]
},
{
"id": 8, "title": "Logins (success / failure)", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 16, "y": 12 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum(rate(login_success_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "success/s" },
{ "expr": "sum(rate(login_failure_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "failure/s" }
]
},
{
"id": 9, "title": "Rate-limit rejections (by route)", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 18 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum by (route) (rate(rate_limit_rejected_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "{{route}}" }
]
},
{
"id": 10, "title": "Push notifications (sent / failed)", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 18 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum by (channel) (rate(push_sent_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "sent {{channel}}" },
{ "expr": "sum by (channel) (rate(push_failed_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "failed {{channel}}" }
]
},
{
"id": 11, "title": "DB query p95 latency & errors", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 24 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "histogram_quantile(0.95, sum by (le, engine) (rate(db_query_duration_ms_bucket{job=\"farmafinder-backend\"}[5m])))", "legendFormat": "p95 {{engine}}" },
{ "expr": "sum by (engine) (rate(db_errors_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "errors {{engine}}" }
],
"fieldConfig": { "defaults": { "unit": "ms" } }
},
{
"id": 12, "title": "Redis cmd p95 & errors", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 24 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "histogram_quantile(0.95, sum by (le) (rate(redis_cmd_duration_ms_bucket{job=\"farmafinder-backend\"}[5m])))", "legendFormat": "p95 redis" },
{ "expr": "sum(rate(redis_errors_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "redis errors/s" }
],
"fieldConfig": { "defaults": { "unit": "ms" } }
},
{
"id": 13, "title": "Admin ops (pharmacy writes / links)", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 30 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "sum by (op) (rate(pharmacy_write_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "pharmacy {{op}}" },
{ "expr": "sum by (op) (rate(pharmacy_medicine_link_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "link {{op}}" }
]
},
{
"id": 14, "title": "Liveness (heartbeat)", "type": "stat",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 30 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "max(app_heartbeat_total{job=\"farmafinder-backend\"})", "legendFormat": "heartbeat total" }
]
}
],
"schemaVersion": 39,
"style": "dark",
"tags": ["farmafinder", "backend"],
"templating": { "list": [] },
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "FarmaFinder — Backend",
"uid": "farmafinder-backend",
"version": 1,
"weekStart": ""
}
@@ -0,0 +1,85 @@
{
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"id": 1, "title": "Redis up", "type": "stat",
"gridPos": { "h": 5, "w": 6, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "redis_up{job=\"farmafinder-redis\"}", "legendFormat": "redis" } ]
},
{
"id": 2, "title": "Connected clients", "type": "timeseries",
"gridPos": { "h": 5, "w": 9, "x": 6, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "redis_connected_clients{job=\"farmafinder-redis\"}", "legendFormat": "clients" } ]
},
{
"id": 3, "title": "Memory used", "type": "timeseries",
"gridPos": { "h": 5, "w": 9, "x": 15, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "redis_memory_used_bytes{job=\"farmafinder-redis\"}", "legendFormat": "bytes" } ],
"fieldConfig": { "defaults": { "unit": "bytes" } }
},
{
"id": 4, "title": "Commands/sec", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 5 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "rate(redis_commands_processed_total{job=\"farmafinder-redis\"}[5m])", "legendFormat": "cmd/s" } ]
},
{
"id": 5, "title": "Keyspace hit ratio", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 5 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "sum(rate(redis_keyspace_hits_total{job=\"farmafinder-redis\"}[5m])) / (sum(rate(redis_keyspace_hits_total{job=\"farmafinder-redis\"}[5m])) + sum(rate(redis_keyspace_misses_total{job=\"farmafinder-redis\"}[5m])))", "legendFormat": "hit ratio" } ],
"fieldConfig": { "defaults": { "unit": "percentunit" } }
},
{
"id": 6, "title": "Postgres up", "type": "stat",
"gridPos": { "h": 5, "w": 6, "x": 0, "y": 11 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "pg_up{job=\"farmafinder-postgres\"}", "legendFormat": "pg" } ]
},
{
"id": 7, "title": "Active connections", "type": "timeseries",
"gridPos": { "h": 5, "w": 9, "x": 6, "y": 11 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "pg_stat_activity_count{job=\"farmafinder-postgres\"}", "legendFormat": "connections" } ]
},
{
"id": 8, "title": "Tuples returned / fetched", "type": "timeseries",
"gridPos": { "h": 5, "w": 9, "x": 15, "y": 11 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "rate(pg_stat_database_tup_returned{job=\"farmafinder-postgres\"}[5m])", "legendFormat": "returned/s" }, { "expr": "rate(pg_stat_database_tup_fetched{job=\"farmafinder-postgres\"}[5m])", "legendFormat": "fetched/s" } ]
},
{
"id": 9, "title": "Deadlocks / conflicts", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 16 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "rate(pg_stat_database_deadlocks{job=\"farmafinder-postgres\"}[5m])", "legendFormat": "deadlocks/s" }, { "expr": "rate(pg_stat_database_conflicts{job=\"farmafinder-postgres\"}[5m])", "legendFormat": "conflicts/s" } ]
},
{
"id": 10, "title": "Cache hit ratio (blocks)", "type": "timeseries",
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 16 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "pg_stat_database_blk_hit{job=\"farmafinder-postgres\"} / (pg_stat_database_blk_hit{job=\"farmafinder-postgres\"} + pg_stat_database_blk_read{job=\"farmafinder-postgres\"})", "legendFormat": "block hit ratio" } ],
"fieldConfig": { "defaults": { "unit": "percentunit" } }
}
],
"schemaVersion": 39,
"style": "dark",
"tags": ["farmafinder", "datastores"],
"templating": { "list": [] },
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "FarmaFinder — Datastores",
"uid": "farmafinder-datastores",
"version": 1,
"weekStart": ""
}
@@ -0,0 +1,58 @@
{
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"id": 1, "title": "Events/sec", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "sum(count_over_time({app=\"farmafinder-mobile\"}[5m]))", "legendFormat": "events/s" } ]
},
{
"id": 2, "title": "Active sessions", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "count(count by (session_id) (count_over_time({app=\"farmafinder-mobile\"}[5m])))", "legendFormat": "sessions" } ]
},
{
"id": 3, "title": "Errors/sec", "type": "timeseries",
"gridPos": { "h": 6, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "sum(count_over_time({app=\"farmafinder-mobile\"} |~ \"(?i)error|exception|crash\"[5m]))", "legendFormat": "errors/s" } ]
},
{
"id": 4, "title": "Events by type", "type": "timeseries",
"gridPos": { "h": 7, "w": 12, "x": 0, "y": 6 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "sum by (type) (count_over_time({app=\"farmafinder-mobile\"}[5m]))", "legendFormat": "{{type}}" } ]
},
{
"id": 5, "title": "Top error messages", "type": "timeseries",
"gridPos": { "h": 7, "w": 12, "x": 12, "y": 6 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "sum by (message) (count_over_time({app=\"farmafinder-mobile\"} |~ \"(?i)error|exception|crash\"[5m]))", "legendFormat": "{{message}}" } ]
},
{
"id": 6, "title": "Recent logs", "type": "logs",
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 13 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "{app=\"farmafinder-mobile\"}", "legendFormat": "" } ]
}
],
"schemaVersion": 39,
"style": "dark",
"tags": ["farmafinder", "mobile", "rum"],
"templating": { "list": [] },
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "FarmaFinder — Mobile RUM",
"uid": "farmafinder-mobile-rum",
"version": 1,
"weekStart": ""
}
@@ -0,0 +1,64 @@
{
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"id": 1, "title": "Backend req/s", "type": "stat",
"gridPos": { "h": 5, "w": 6, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "sum(rate(http_requests_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "req/s" } ]
},
{
"id": 2, "title": "Backend 5xx ratio", "type": "stat",
"gridPos": { "h": 5, "w": 6, "x": 6, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "sum(rate(http_requests_total{status_class=\"5xx\",job=\"farmafinder-backend\"}[5m])) / sum(rate(http_requests_total{job=\"farmafinder-backend\"}[5m]))", "legendFormat": "5xx" } ],
"fieldConfig": { "defaults": { "unit": "percentunit", "thresholds": { "steps": [ { "color": "green", "value": 0 }, { "color": "yellow", "value": 0.01 }, { "color": "red", "value": 0.05 } ] } } }
},
{
"id": 3, "title": "CIMA errors/s", "type": "stat",
"gridPos": { "h": 5, "w": 6, "x": 12, "y": 0 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [ { "expr": "sum(rate(cima_requests_total{status=\"error\",job=\"farmafinder-backend\"}[5m]))", "legendFormat": "cima err/s" } ]
},
{
"id": 4, "title": "Mobile events/s", "type": "stat",
"gridPos": { "h": 5, "w": 6, "x": 18, "y": 0 },
"datasource": { "type": "loki", "uid": "bfowk7qjoccu8b" },
"targets": [ { "expr": "sum(count_over_time({app=\"farmafinder-mobile\"}[5m]))", "legendFormat": "mobile/s" } ]
},
{
"id": 5, "title": "Datastore health", "type": "stat",
"gridPos": { "h": 5, "w": 12, "x": 0, "y": 5 },
"datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
"targets": [
{ "expr": "redis_up{job=\"farmafinder-redis\"}", "legendFormat": "redis" },
{ "expr": "pg_up{job=\"farmafinder-postgres\"}", "legendFormat": "postgres" }
]
},
{
"id": 6, "title": "Navigation", "type": "text",
"gridPos": { "h": 5, "w": 12, "x": 12, "y": 5 },
"options": {
"mode": "markdown",
"content": "**Dashboards**\n- [Backend](/d/farmafinder-backend/farmafinder-backend)\n- [Datastores](/d/farmafinder-datastores/farmafinder-datastores)\n- [Mobile RUM](/d/farmafinder-mobile-rum/farmafinder-mobile-rum)\n- [Faro Web RUM](/d/faro-overview/faro-farmafinder-rum-overview)\n\n**Explore**\n- [Tempo (traces)](/explore?orgId=1&left=%5B%22now-6h%22,%22now%22,%22tempo%22,%7B%7D%5D)\n- [Loki (logs)](/explore?orgId=1&left=%5B%22now-6h%22,%22now%22,%22loki%22,%7B%7D%5D)\n- [Prometheus (metrics)](/explore?orgId=1&left=%5B%22now-6h%22,%22now%22,%22PBFA97CFB590B2093%22,%7B%7D%5D)"
}
}
],
"schemaVersion": 39,
"style": "dark",
"tags": ["farmafinder", "overview"],
"templating": { "list": [] },
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "FarmaFinder — Overview",
"uid": "farmafinder-overview",
"version": 1,
"weekStart": ""
}