FarmaFinder/docs/superpowers/specs/2026-07-13-observability-design.md

FarmaFinder — End-to-End Observability Plan & Progress

Purpose: Single source of truth for adding metrics + monitoring across the FarmaFinder stack. If the session runs out of tokens, a fresh session can read this file and continue from the Status section.

Created: 2026-07-13

---

1. Current State (verified)

1.1 Monitoring stack — srv84-macos:/root/Monitoring (Docker Compose)

Running services (verified live):

Service	Port	Role
Prometheus v2.55.1	9090	metrics + alerting (config /etc/prometheus/)
Grafana grafana/grafana	3000	dashboards (admin foobar)
Alertmanager v0.27.0	9093	Telegram notifications (already configured)
Loki latest	3100	logs (/otlp receiver enabled)
Tempo latest	3200	traces
cAdvisor	8080	container metrics
node-exporter	9100	host metrics (srv84-macos + srv102-hosting + srv104-omv)
Alloy v1.6.1	4317 gRPC / 4318 HTTP / 12345 self	unified OTel collector

Alloy pipeline (alloy-config.alloy): otelcol.receiver.otlp (gRPC+HTTP) → batch →

• traces → otelp.tempo (Tempo:4317)
• logs → otlphttp.loki (Loki:3100/otlp)
• metrics → exporter.prometheus → prometheus.remote_write → http://prometheus:9090/api/v1/write

So any OTLP metrics sent to Alloy land in Prometheus automatically — no new scrape job needed for app services.

1.2 Grafana datasources (provisioned, sufficient)

Prometheus (default), Tempo, Loki, PostgreSQL, Alertmanager.

1.3 Grafana plugins (installed, sufficient)

Built-in panels + Drilldown apps: grafana-lokiexplore-app, grafana-metricsdrilldown-app, grafana-exploretraces-app, grafana-pyroscope-app. No extra plugins required.

1.4 Existing app instrumentation

App	Traces	Metrics	Logs	Health	Notes
backend (Express)	✅ OTel→Alloy	❌ none	✅ pino→Alloy	❌ none	src/tracing.js boots NodeSDK; only traces+logs exported
frontend (React/Vite PWA)	✅ Faro→Alloy:4318	❌	✅ Faro→Loki	—	src/utils/faro.js; dashboards faro-overview/-web-vitals/-js-errors exist
frontend-mobile (Expo RN)	❌	❌	console only	—	no observability at all
pip-platform (FastAPI)	✅ OTel	✅ /metrics (prometheus_client)	✅ structlog	✅ /system/health,/ready,/status	not scraped by Prometheus yet
scraper (Puppeteer)	✅ OTel	❌	console	—

1.5 Prometheus scrape jobs (today)

prometheus, cadvisor, node-exporter (x3 hosts). No farmafinder/pip/redis/postgres jobs. Comment in prometheus.yml: app metrics are expected via OTLP→Alloy→remote_write.

1.6 Existing Prometheus alerts (alert.rules)

service_down, high_load, high_memory, disk_almost_full, pm2_process_down, container_down. All infra-level.

---

2. Decisions (confirmed with user)

• Metrics transport = OTLP for backend + mobile (aligns with stack design; no new app scrape jobs). Infra exporters (Redis/Postgres/pip) stay as scrape jobs.
• Mobile Faro endpoint: env-configurable EXPO_PUBLIC_FARO_URL, default http://grafana.hacecalor.net:4318. User will open router port 4318 → srv84-macos:4318 (and/or fix Nginx Proxy Manager /faro proxy to Alloy:4318). Devices are NOT on the docker host, so localhost will NOT work.
• Alertmanager: Telegram already configured — no contact-point change. New rules just route to it.

---

3. Design / Tasks

Task A — Backend metrics (OTel → Alloy → Prometheus)

Files: apps/backend/src/tracing.js, apps/backend/src/metrics.js (new), apps/backend/package.json, docker-compose.yml.

1. Add metrics SDK to tracing.js:
   • MeterProvider + OTLPMetricExporter + PeriodicExportingMetricReader (interval 15s).
   • Reuse existing resource + getNodeAutoInstrumentations (already includes http/express/redis → gives http.server.request.duration, http.client.request.duration, redis.client.commands.duration, etc.).
   • Keep PinoInstrumentation. Gate on NODE_ENV !== 'test'.
2. New src/metrics.js using @opentelemetry/api metrics:
   • medicine_searches_total (counter)
   • cima_request_duration_ms (histogram), cima_errors_total (counter)
   • cache_hits_total / cache_misses_total (counters)
   • login_success_total / login_failure_total (counters)
   • rate_limit_rejected_total (counter, label route)
   • pharmacy_write_total (counter, label op)
   • pharmacy_medicine_link_total (counter)
   • push_sent_total / push_failed_total (counters, label channel=expo|web)
   • db_query_duration_ms (histogram), db_errors_total (counter, label engine=sqlite|pg)
   • redis_errors_total (counter), redis_cmd_duration_ms (histogram)
   • heartbeat_total (counter, incremented every 30s) → drives "backend down" alert
   • Node runtime metrics via views/runtime instrumentation (event loop lag, heap, gc).
3. Instrument the relevant handlers in server.js (search, auth, admin, notifications, db, redis) by importing the meters.
4. docker-compose.yml: add OTEL_METRICS_EXPORTER: otlp under backend env (alongside existing traces/logs exporters).
5. package.json: ensure @opentelemetry/sdk-metrics + @opentelemetry/exporter-metrics-otlp-grpc are present (add if missing; align versions with existing ^0.55.0).

Task B — Backend health endpoints

File: apps/backend/server.js.

• GET /healthz → {status:"ok"} (liveness).
• GET /readyz → ping Redis + run a trivial DB query; return 200/503 JSON.
• Used by Docker healthcheck / k8s; not scraped.

Task C — Mobile RUM (Expo / React Native)

Files: apps/frontend-mobile/package.json, apps/frontend-mobile/app/_layout.tsx (root), .env.example.

• Add dependency @grafana/faro-react-native (^1.2.1).
• Init in _layout.tsx (or App.tsx) entry:

  import { initializeFaro } from '@grafana/faro-react-native';
  initializeFaro({
    app: { name: 'farmafinder-mobile', version: '1.0.0', environment: 'production' },
    url: process.env.EXPO_PUBLIC_FARO_URL || 'http://grafana.hacecalor.net:4318',
    sessionTracking: { enabled: true },
    // captures crashes, JS errors, console, ANR(Android), app-start, memory vitals
  });
  

• EXPO_PUBLIC_FARO_URL in .env.example (default the public URL above).
• Optional follow-up: @grafana/faro-react-native-tracing for distributed traces + @grafana/faro-metro-plugin for source maps (post-MVP).

Task D — Infra exporters (scrape jobs on srv84-macos)

Files (remote): /root/Monitoring/docker-compose.yml, /root/Monitoring/prometheus/prometheus.yml.

• Add containers redis-exporter (→ farmafinder redis) and postgres-exporter (→ farmafinder postgres) to the Monitoring compose. Reach farmafinder's redis/postgres via the docker gateway / service DNS on the same host.
• Add scrape jobs in prometheus.yml:
  • farmafinder-redis → redis-exporter
  • farmafinder-postgres → postgres-exporter
  • pip-platform → http://<pip-host>:8000/metrics (pip already exposes it)
• Reload Prometheus (POST /-/reload).

Task E — Grafana dashboards (provisioned JSON)

Files: /root/Monitoring/grafana/provisioning/dashboards/*.json + dashboard.yml entry.

1. farmafinder-backend.json — req rate, error %, latency p50/p95/p99 by route, CIMA latency/errors, cache hit rate, Redis/DB errors, push counts, auth failures, rate-limit hits, Node runtime.
2. farmafinder-datastores.json — Redis + Postgres + (SQLite via app metric) panels.
3. farmafinder-mobile-rum.json — Faro RN data from Loki (app.name="farmafinder-mobile").
4. farmafinder-overview.json — combined RUM + backend metrics + deep-links to Tempo (traces) & Loki (logs). (Keep existing faro-* PWA dashboards.)

Task F — Alerts

Files (remote): /root/Monitoring/prometheus/alert.rules + Grafana-managed alert. Prometheus rules (group farmafinder):

• farmafinder_backend_down — absent(heartbeat_total{job="farmafinder-backend"}) > 0 for 5m → critical
• farmafinder_backend_5xx — rate(http.server.request.errors ...) ratio > 5% over 5m → warning
• farmafinder_cima_errors — CIMA error rate high → warning
• farmafinder_high_latency — p95 http.server.request.duration > threshold → warning
• farmafinder_redis_down / farmafinder_postgres_down / pip_platform_down — up==0 Grafana-managed alert (Loki): mobile/PWA JS-error spike — count of Faro exceptions over 10m > N. All route to existing Alertmanager (Telegram).

Task G — Mobile Faro reachability (user action)

• Open router TCP 4318 → srv84-macos:4318, or fix Nginx Proxy Manager grafana.hacecalor.net/faro → http://<alloy-host>:4318 ensuring OTLP HTTP paths (/v1/logs,/v1/metrics,/v1/traces) are proxied. Verify from a phone / external curl that POST http://grafana.hacecalor.net:4318/v1/logs is accepted.

---

4. Status (update as you go)

Task	Status	Notes
A. Backend metrics	✅ done (code)	src/metrics.js, tracing.js meter provider, instrumented server.js/cima-service.js/redis-client.js. Needs backend rebuild+redeploy to take effect.
B. Backend health	✅ done (code)	/healthz, /readyz added.
C. Mobile Faro	✅ done (code)	services/faro.ts + _layout.tsx init + dep @grafana/faro-react-native. Requires dev/EAS build (native module).
D. Infra exporters	✅ done	redis/postgres exporters added to farmafinder docker-compose.yml; Prometheus scrape jobs + alert rules applied on srv84-macos. Needs farmafinder redeploy.
E. Dashboards	✅ done + LIVE	4 dashboards pushed to /root/Monitoring/grafana/provisioning/dashboards/ and provisioned in Grafana.
F. Alerts	✅ done + LIVE	farmafinder group applied to alert.rules; routes to existing Alertmanager (Telegram).
G. Mobile Faro URL	⏳ user action	open :4318 → srv84-macos:4318, or fix Nginx Proxy Manager /faro.

5. Deployment & Verification

Backend (apply on next deploy)

The running farmafinder-backend-1 still runs old code. To activate metrics:

# from repo root on srv84-macos (or wherever farmafinder is deployed)
docker compose build backend && docker compose up -d backend
# or push to main and let .gitea deploy

Verify:

# after ~30s, a heartbeat sample should exist
curl -s "http://srv84-macos:9090/api/v1/query?query=app_heartbeat_total{job=%22farmafinder-backend%22}"
# health probes
curl -s http://localhost:3001/healthz ; curl -s http://localhost:3001/readyz
# dashboards populate: Grafana → FarmaFinder — Backend

Mobile (user)

1. cd apps/frontend-mobile && npm install (pulls @grafana/faro-react-native).
2. Build a dev/EAS build (not Expo Go — native module).
3. Set EXPO_PUBLIC_FARO_URL (default http://grafana.hacecalor.net:4318) reachable from device.
4. Verify in Grafana → FarmaFinder — Mobile RUM (Loki {app="farmafinder-mobile"}).

Datastore exporters (apply on farmafinder redeploy)

docker compose up -d brings up redis-exporter (:9121) and postgres-exporter (:9187); Prometheus already scrapes them. Verify: up{job="farmafinder-redis"}, up{job="farmafinder-postgres"}.

6. Verification checklist (end)

• 4 Grafana dashboards live (backend, datastores, mobile-rum, overview)
• Prometheus scrape jobs farmafinder-redis / farmafinder-postgres present
• Alert rules farmafinder_* loaded (no config errors)
• Backend redeployed → app_heartbeat_total, http_requests_total, cima_requests_total, etc. appear
• /healthz, /readyz return 200
• Mobile build → Faro data in Loki + Mobile RUM dashboard
• Trigger a test alert (stop backend) → Telegram receives it

7. Key commands / references

• Remote stack: ssh root@srv84-macos, configs under /root/Monitoring.
• Prometheus reload: curl -X POST http://srv84-macos:9090/-/reload.
• Grafana dashboards reload: curl -u admin:foobar -X POST http://srv84-macos:3000/api/admin/provisioning/dashboards/reload.
• Grafana API: curl -u admin:foobar http://srv84-macos:3000/api/... (admin password foobar).
• Backend OTel env (in docker-compose.yml): OTEL_SERVICE_NAME=farmafinder-backend, OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317, OTEL_TRACES_EXPORTER=otlp, OTEL_METRICS_EXPORTER=otlp (added), OTEL_LOGS_EXPORTER=otlp.
• New metrics module: apps/backend/src/metrics.js.
• Dashboards source: monitoring/dashboards/*.json (mirrored on the host).