feat: add end-to-end observability (metrics, health, mobile RUM, dashboards, alerts)

- Backend: OTel metrics via OTLP -> Alloy -> Prometheus (OTEL_METRICS_EXPORTER=otlp) - New business metrics (src/metrics.js): searches, CIMA latency/errors, cache hits/misses, logins, rate-limits, pharmacy writes/links, push sent/failed, DB + Redis timings/errors, HTTP req count/duration, heartbeat - Backend health endpoints /healthz and /readyz - Mobile (Expo): Grafana Faro RUM via @grafana/faro-react-native - redis/postgres exporters in docker-compose + Prometheus scrape jobs - Grafana dashboards (backend, datastores, mobile RUM, overview) - Prometheus alert rules (farmafinder_*) -> existing Alertmanager (Telegram) - Design/spec saved to docs/superpowers/specs/
2026-07-13 15:57:52 +02:00
parent 2713be85a3
commit 839c64c12a
16 changed files with 907 additions and 27 deletions
@@ -0,0 +1,208 @@
+# FarmaFinder — End-to-End Observability Plan & Progress
+
+> **Purpose:** Single source of truth for adding metrics + monitoring across the
+> FarmaFinder stack. If the session runs out of tokens, a fresh session can read
+> this file and continue from the **Status** section.
+>
+> Created: 2026-07-13
+
+---
+
+## 1. Current State (verified)
+
+### 1.1 Monitoring stack — `srv84-macos:/root/Monitoring` (Docker Compose)
+Running services (verified live):
+
+| Service | Port | Role |
+|---|---|---|
+| Prometheus `v2.55.1` | 9090 | metrics + alerting (config `/etc/prometheus/`) |
+| Grafana `grafana/grafana` | 3000 | dashboards (admin `foobar`) |
+| Alertmanager `v0.27.0` | 9093 | **Telegram** notifications (already configured) |
+| Loki `latest` | 3100 | logs (`/otlp` receiver enabled) |
+| Tempo `latest` | 3200 | traces |
+| cAdvisor | 8080 | container metrics |
+| node-exporter | 9100 | host metrics (srv84-macos + srv102-hosting + srv104-omv) |
+| **Alloy `v1.6.1`** | 4317 gRPC / 4318 HTTP / 12345 self | unified OTel collector |
+
+**Alloy pipeline (`alloy-config.alloy`):**
+`otelcol.receiver.otlp` (gRPC+HTTP) → `batch` →
+- traces → `otelp.tempo` (Tempo:4317)
+- logs → `otlphttp.loki` (Loki:3100/otlp)
+- metrics → `exporter.prometheus` → `prometheus.remote_write` → `http://prometheus:9090/api/v1/write`
+
+So **any OTLP metrics sent to Alloy land in Prometheus automatically** — no new scrape job needed for app services.
+
+### 1.2 Grafana datasources (provisioned, sufficient)
+Prometheus (default), Tempo, Loki, PostgreSQL, Alertmanager.
+
+### 1.3 Grafana plugins (installed, sufficient)
+Built-in panels + **Drilldown apps**: `grafana-lokiexplore-app`, `grafana-metricsdrilldown-app`, `grafana-exploretraces-app`, `grafana-pyroscope-app`.
+**No extra plugins required.**
+
+### 1.4 Existing app instrumentation
+| App | Traces | Metrics | Logs | Health | Notes |
+|---|---|---|---|---|---|
+| backend (Express) | ✅ OTel→Alloy | ❌ **none** | ✅ pino→Alloy | ❌ none | `src/tracing.js` boots NodeSDK; only traces+logs exported |
+| frontend (React/Vite PWA) | ✅ Faro→Alloy:4318 | ❌ | ✅ Faro→Loki | — | `src/utils/faro.js`; dashboards `faro-overview/-web-vitals/-js-errors` exist |
+| frontend-mobile (Expo RN) | ❌ | ❌ | console only | — | **no observability at all** |
+| pip-platform (FastAPI) | ✅ OTel | ✅ `/metrics` (prometheus_client) | ✅ structlog | ✅ /system/health,/ready,/status | **not scraped by Prometheus yet** |
+| scraper (Puppeteer) | ✅ OTel | ❌ | console | — | |
+
+### 1.5 Prometheus scrape jobs (today)
+`prometheus`, `cadvisor`, `node-exporter` (x3 hosts). **No farmafinder/pip/redis/postgres jobs.**
+Comment in `prometheus.yml`: app metrics are expected via OTLP→Alloy→remote_write.
+
+### 1.6 Existing Prometheus alerts (`alert.rules`)
+`service_down`, `high_load`, `high_memory`, `disk_almost_full`, `pm2_process_down`, `container_down`. All infra-level.
+
+---
+
+## 2. Decisions (confirmed with user)
+
+- **Metrics transport = OTLP** for backend + mobile (aligns with stack design; no new app scrape jobs). Infra exporters (Redis/Postgres/pip) stay as scrape jobs.
+- **Mobile Faro endpoint:** env-configurable `EXPO_PUBLIC_FARO_URL`, default
+  `http://grafana.hacecalor.net:4318`. User will **open router port 4318 → srv84-macos:4318**
+  (and/or fix Nginx Proxy Manager `/faro` proxy to Alloy:4318). Devices are NOT on the docker
+  host, so `localhost` will NOT work.
+- **Alertmanager:** Telegram already configured — **no contact-point change**. New rules just route to it.
+
+---
+
+## 3. Design / Tasks
+
+### Task A — Backend metrics (OTel → Alloy → Prometheus)
+**Files:** `apps/backend/src/tracing.js`, `apps/backend/src/metrics.js` (new), `apps/backend/package.json`, `docker-compose.yml`.
+
+1. Add metrics SDK to `tracing.js`:
+   - `MeterProvider` + `OTLPMetricExporter` + `PeriodicExportingMetricReader` (interval 15s).
+   - Reuse existing `resource` + `getNodeAutoInstrumentations` (already includes http/express/redis → gives `http.server.request.duration`, `http.client.request.duration`, `redis.client.commands.duration`, etc.).
+   - Keep `PinoInstrumentation`. Gate on `NODE_ENV !== 'test'`.
+2. New `src/metrics.js` using `@opentelemetry/api` `metrics`:
+   - `medicine_searches_total` (counter)
+   - `cima_request_duration_ms` (histogram), `cima_errors_total` (counter)
+   - `cache_hits_total` / `cache_misses_total` (counters)
+   - `login_success_total` / `login_failure_total` (counters)
+   - `rate_limit_rejected_total` (counter, label `route`)
+   - `pharmacy_write_total` (counter, label `op`)
+   - `pharmacy_medicine_link_total` (counter)
+   - `push_sent_total` / `push_failed_total` (counters, label `channel=expo|web`)
+   - `db_query_duration_ms` (histogram), `db_errors_total` (counter, label `engine=sqlite|pg`)
+   - `redis_errors_total` (counter), `redis_cmd_duration_ms` (histogram)
+   - `heartbeat_total` (counter, incremented every 30s) → drives "backend down" alert
+   - Node runtime metrics via `views`/runtime instrumentation (event loop lag, heap, gc).
+3. Instrument the relevant handlers in `server.js` (search, auth, admin, notifications, db, redis) by importing the meters.
+4. `docker-compose.yml`: add `OTEL_METRICS_EXPORTER: otlp` under backend env (alongside existing traces/logs exporters).
+5. `package.json`: ensure `@opentelemetry/sdk-metrics` + `@opentelemetry/exporter-metrics-otlp-grpc` are present (add if missing; align versions with existing `^0.55.0`).
+
+### Task B — Backend health endpoints
+**File:** `apps/backend/server.js`.
+- `GET /healthz` → `{status:"ok"}` (liveness).
+- `GET /readyz` → ping Redis + run a trivial DB query; return 200/503 JSON.
+- Used by Docker healthcheck / k8s; not scraped.
+
+### Task C — Mobile RUM (Expo / React Native)
+**Files:** `apps/frontend-mobile/package.json`, `apps/frontend-mobile/app/_layout.tsx` (root), `.env.example`.
+- Add dependency `@grafana/faro-react-native` (`^1.2.1`).
+- Init in `_layout.tsx` (or `App.tsx`) entry:
+  ```ts
+  import { initializeFaro } from '@grafana/faro-react-native';
+  initializeFaro({
+    app: { name: 'farmafinder-mobile', version: '1.0.0', environment: 'production' },
+    url: process.env.EXPO_PUBLIC_FARO_URL || 'http://grafana.hacecalor.net:4318',
+    sessionTracking: { enabled: true },
+    // captures crashes, JS errors, console, ANR(Android), app-start, memory vitals
+  });
+  ```
+- `EXPO_PUBLIC_FARO_URL` in `.env.example` (default the public URL above).
+- Optional follow-up: `@grafana/faro-react-native-tracing` for distributed traces + `@grafana/faro-metro-plugin` for source maps (post-MVP).
+
+### Task D — Infra exporters (scrape jobs on `srv84-macos`)
+**Files (remote):** `/root/Monitoring/docker-compose.yml`, `/root/Monitoring/prometheus/prometheus.yml`.
+- Add containers `redis-exporter` (→ farmafinder redis) and `postgres-exporter` (→ farmafinder postgres) to the Monitoring compose. Reach farmafinder's redis/postgres via the docker gateway / service DNS on the same host.
+- Add scrape jobs in `prometheus.yml`:
+  - `farmafinder-redis` → redis-exporter
+  - `farmafinder-postgres` → postgres-exporter
+  - `pip-platform` → `http://<pip-host>:8000/metrics` (pip already exposes it)
+- Reload Prometheus (`POST /-/reload`).
+
+### Task E — Grafana dashboards (provisioned JSON)
+**Files:** `/root/Monitoring/grafana/provisioning/dashboards/*.json` + `dashboard.yml` entry.
+1. `farmafinder-backend.json` — req rate, error %, latency p50/p95/p99 by route, CIMA latency/errors, cache hit rate, Redis/DB errors, push counts, auth failures, rate-limit hits, Node runtime.
+2. `farmafinder-datastores.json` — Redis + Postgres + (SQLite via app metric) panels.
+3. `farmafinder-mobile-rum.json` — Faro RN data from Loki (`app.name="farmafinder-mobile"`).
+4. `farmafinder-overview.json` — combined RUM + backend metrics + deep-links to Tempo (traces) & Loki (logs).
+(Keep existing `faro-*` PWA dashboards.)
+
+### Task F — Alerts
+**Files (remote):** `/root/Monitoring/prometheus/alert.rules` + Grafana-managed alert.
+Prometheus rules (group `farmafinder`):
+- `farmafinder_backend_down` — `absent(heartbeat_total{job="farmafinder-backend"}) > 0` for 5m → critical
+- `farmafinder_backend_5xx` — `rate(http.server.request.errors ...)` ratio > 5% over 5m → warning
+- `farmafinder_cima_errors` — CIMA error rate high → warning
+- `farmafinder_high_latency` — p95 `http.server.request.duration` > threshold → warning
+- `farmafinder_redis_down` / `farmafinder_postgres_down` / `pip_platform_down` — `up==0`
+Grafana-managed alert (Loki): **mobile/PWA JS-error spike** — count of Faro exceptions over 10m > N.
+All route to existing Alertmanager (Telegram).
+
+### Task G — Mobile Faro reachability (user action)
+- Open router `TCP 4318 → srv84-macos:4318`, **or** fix Nginx Proxy Manager `grafana.hacecalor.net/faro` → `http://<alloy-host>:4318` ensuring OTLP HTTP paths (`/v1/logs`,`/v1/metrics`,`/v1/traces`) are proxied. Verify from a phone / external curl that `POST http://grafana.hacecalor.net:4318/v1/logs` is accepted.
+
+---
+
+## 4. Status (update as you go)
+
+| Task | Status | Notes |
+|---|---|---|
+| A. Backend metrics | ✅ done (code) | `src/metrics.js`, `tracing.js` meter provider, instrumented server.js/cima-service.js/redis-client.js. Needs backend rebuild+redeploy to take effect. |
+| B. Backend health | ✅ done (code) | `/healthz`, `/readyz` added. |
+| C. Mobile Faro | ✅ done (code) | `services/faro.ts` + `_layout.tsx` init + dep `@grafana/faro-react-native`. Requires dev/EAS build (native module). |
+| D. Infra exporters | ✅ done | redis/postgres exporters added to farmafinder `docker-compose.yml`; Prometheus scrape jobs + alert rules applied on srv84-macos. Needs farmafinder redeploy. |
+| E. Dashboards | ✅ done + LIVE | 4 dashboards pushed to `/root/Monitoring/grafana/provisioning/dashboards/` and provisioned in Grafana. |
+| F. Alerts | ✅ done + LIVE | `farmafinder` group applied to `alert.rules`; routes to existing Alertmanager (Telegram). |
+| G. Mobile Faro URL | ⏳ user action | open :4318 → srv84-macos:4318, or fix Nginx Proxy Manager `/faro`. |
+
+## 5. Deployment & Verification
+
+### Backend (apply on next deploy)
+The running `farmafinder-backend-1` still runs old code. To activate metrics:
+```bash
+# from repo root on srv84-macos (or wherever farmafinder is deployed)
+docker compose build backend && docker compose up -d backend
+# or push to main and let .gitea deploy
+```
+Verify:
+```bash
+# after ~30s, a heartbeat sample should exist
+curl -s "http://srv84-macos:9090/api/v1/query?query=app_heartbeat_total{job=%22farmafinder-backend%22}"
+# health probes
+curl -s http://localhost:3001/healthz ; curl -s http://localhost:3001/readyz
+# dashboards populate: Grafana → FarmaFinder — Backend
+```
+
+### Mobile (user)
+1. `cd apps/frontend-mobile && npm install` (pulls `@grafana/faro-react-native`).
+2. Build a **dev/EAS build** (not Expo Go — native module).
+3. Set `EXPO_PUBLIC_FARO_URL` (default `http://grafana.hacecalor.net:4318`) reachable from device.
+4. Verify in Grafana → FarmaFinder — Mobile RUM (Loki `{app="farmafinder-mobile"}`).
+
+### Datastore exporters (apply on farmafinder redeploy)
+`docker compose up -d` brings up `redis-exporter` (`:9121`) and `postgres-exporter` (`:9187`);
+Prometheus already scrapes them. Verify: `up{job="farmafinder-redis"}`, `up{job="farmafinder-postgres"}`.
+
+## 6. Verification checklist (end)
+- [x] 4 Grafana dashboards live (backend, datastores, mobile-rum, overview)
+- [x] Prometheus scrape jobs `farmafinder-redis` / `farmafinder-postgres` present
+- [x] Alert rules `farmafinder_*` loaded (no config errors)
+- [ ] Backend redeployed → `app_heartbeat_total`, `http_requests_total`, `cima_requests_total`, etc. appear
+- [ ] `/healthz`, `/readyz` return 200
+- [ ] Mobile build → Faro data in Loki + Mobile RUM dashboard
+- [ ] Trigger a test alert (stop backend) → Telegram receives it
+
+## 7. Key commands / references
+- Remote stack: `ssh root@srv84-macos`, configs under `/root/Monitoring`.
+- Prometheus reload: `curl -X POST http://srv84-macos:9090/-/reload`.
+- Grafana dashboards reload: `curl -u admin:foobar -X POST http://srv84-macos:3000/api/admin/provisioning/dashboards/reload`.
+- Grafana API: `curl -u admin:foobar http://srv84-macos:3000/api/...` (admin password `foobar`).
+- Backend OTel env (in `docker-compose.yml`): `OTEL_SERVICE_NAME=farmafinder-backend`, `OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317`, `OTEL_TRACES_EXPORTER=otlp`, **`OTEL_METRICS_EXPORTER=otlp`** (added), `OTEL_LOGS_EXPORTER=otlp`.
+- New metrics module: `apps/backend/src/metrics.js`.
+- Dashboards source: `monitoring/dashboards/*.json` (mirrored on the host).