# FarmaFinder — End-to-End Observability Plan & Progress

> **Purpose:** Single source of truth for adding metrics + monitoring across the
> FarmaFinder stack. If the session runs out of tokens, a fresh session can read
> this file and continue from the **Status** section.
>
> Created: 2026-07-13

---

## 1. Current State (verified)

### 1.1 Monitoring stack — `srv84-macos:/root/Monitoring` (Docker Compose)
Running services (verified live):

| Service | Port | Role |
|---|---|---|
| Prometheus `v2.55.1` | 9090 | metrics + alerting (config `/etc/prometheus/`) |
| Grafana `grafana/grafana` | 3000 | dashboards (admin `foobar`) |
| Alertmanager `v0.27.0` | 9093 | **Telegram** notifications (already configured) |
| Loki `latest` | 3100 | logs (`/otlp` receiver enabled) |
| Tempo `latest` | 3200 | traces |
| cAdvisor | 8080 | container metrics |
| node-exporter | 9100 | host metrics (srv84-macos + srv102-hosting + srv104-omv) |
| **Alloy `v1.6.1`** | 4317 gRPC / 4318 HTTP / 12345 self | unified OTel collector |

**Alloy pipeline (`alloy-config.alloy`):**
`otelcol.receiver.otlp` (gRPC+HTTP) → `batch` →
- traces → `otelp.tempo` (Tempo:4317)
- logs → `otlphttp.loki` (Loki:3100/otlp)
- metrics → `exporter.prometheus` → `prometheus.remote_write` → `http://prometheus:9090/api/v1/write`

So **any OTLP metrics sent to Alloy land in Prometheus automatically** — no new scrape job needed for app services.

### 1.2 Grafana datasources (provisioned, sufficient)
Prometheus (default), Tempo, Loki, PostgreSQL, Alertmanager.

### 1.3 Grafana plugins (installed, sufficient)
Built-in panels + **Drilldown apps**: `grafana-lokiexplore-app`, `grafana-metricsdrilldown-app`, `grafana-exploretraces-app`, `grafana-pyroscope-app`.
**No extra plugins required.**

### 1.4 Existing app instrumentation
| App | Traces | Metrics | Logs | Health | Notes |
|---|---|---|---|---|---|
| backend (Express) | ✅ OTel→Alloy | ❌ **none** | ✅ pino→Alloy | ❌ none | `src/tracing.js` boots NodeSDK; only traces+logs exported |
| frontend (React/Vite PWA) | ✅ Faro→Alloy:4318 | ❌ | ✅ Faro→Loki | — | `src/utils/faro.js`; dashboards `faro-overview/-web-vitals/-js-errors` exist |
| frontend-mobile (Expo RN) | ❌ | ❌ | console only | — | **no observability at all** |
| pip-platform (FastAPI) | ✅ OTel | ✅ `/metrics` (prometheus_client) | ✅ structlog | ✅ /system/health,/ready,/status | **not scraped by Prometheus yet** |
| scraper (Puppeteer) | ✅ OTel | ❌ | console | — | |

### 1.5 Prometheus scrape jobs (today)
`prometheus`, `cadvisor`, `node-exporter` (x3 hosts). **No farmafinder/pip/redis/postgres jobs.**
Comment in `prometheus.yml`: app metrics are expected via OTLP→Alloy→remote_write.

### 1.6 Existing Prometheus alerts (`alert.rules`)
`service_down`, `high_load`, `high_memory`, `disk_almost_full`, `pm2_process_down`, `container_down`. All infra-level.

---

## 2. Decisions (confirmed with user)

- **Metrics transport = OTLP** for backend + mobile (aligns with stack design; no new app scrape jobs). Infra exporters (Redis/Postgres/pip) stay as scrape jobs.
- **Mobile Faro endpoint:** env-configurable `EXPO_PUBLIC_FARO_URL`, default
  `http://grafana.hacecalor.net:4318`. User will **open router port 4318 → srv84-macos:4318**
  (and/or fix Nginx Proxy Manager `/faro` proxy to Alloy:4318). Devices are NOT on the docker
  host, so `localhost` will NOT work.
- **Alertmanager:** Telegram already configured — **no contact-point change**. New rules just route to it.

---

## 3. Design / Tasks

### Task A — Backend metrics (OTel → Alloy → Prometheus)
**Files:** `apps/backend/src/tracing.js`, `apps/backend/src/metrics.js` (new), `apps/backend/package.json`, `docker-compose.yml`.

1. Add metrics SDK to `tracing.js`:
   - `MeterProvider` + `OTLPMetricExporter` + `PeriodicExportingMetricReader` (interval 15s).
   - Reuse existing `resource` + `getNodeAutoInstrumentations` (already includes http/express/redis → gives `http.server.request.duration`, `http.client.request.duration`, `redis.client.commands.duration`, etc.).
   - Keep `PinoInstrumentation`. Gate on `NODE_ENV !== 'test'`.
2. New `src/metrics.js` using `@opentelemetry/api` `metrics`:
   - `medicine_searches_total` (counter)
   - `cima_request_duration_ms` (histogram), `cima_errors_total` (counter)
   - `cache_hits_total` / `cache_misses_total` (counters)
   - `login_success_total` / `login_failure_total` (counters)
   - `rate_limit_rejected_total` (counter, label `route`)
   - `pharmacy_write_total` (counter, label `op`)
   - `pharmacy_medicine_link_total` (counter)
   - `push_sent_total` / `push_failed_total` (counters, label `channel=expo|web`)
   - `db_query_duration_ms` (histogram), `db_errors_total` (counter, label `engine=sqlite|pg`)
   - `redis_errors_total` (counter), `redis_cmd_duration_ms` (histogram)
   - `heartbeat_total` (counter, incremented every 30s) → drives "backend down" alert
   - Node runtime metrics via `views`/runtime instrumentation (event loop lag, heap, gc).
3. Instrument the relevant handlers in `server.js` (search, auth, admin, notifications, db, redis) by importing the meters.
4. `docker-compose.yml`: add `OTEL_METRICS_EXPORTER: otlp` under backend env (alongside existing traces/logs exporters).
5. `package.json`: ensure `@opentelemetry/sdk-metrics` + `@opentelemetry/exporter-metrics-otlp-grpc` are present (add if missing; align versions with existing `^0.55.0`).

### Task B — Backend health endpoints
**File:** `apps/backend/server.js`.
- `GET /healthz` → `{status:"ok"}` (liveness).
- `GET /readyz` → ping Redis + run a trivial DB query; return 200/503 JSON.
- Used by Docker healthcheck / k8s; not scraped.

### Task C — Mobile RUM (Expo / React Native)
**Files:** `apps/frontend-mobile/package.json`, `apps/frontend-mobile/app/_layout.tsx` (root), `.env.example`.
- Add dependency `@grafana/faro-react-native` (`^1.2.1`).
- Init in `_layout.tsx` (or `App.tsx`) entry:
  ```ts
  import { initializeFaro } from '@grafana/faro-react-native';
  initializeFaro({
    app: { name: 'farmafinder-mobile', version: '1.0.0', environment: 'production' },
    url: process.env.EXPO_PUBLIC_FARO_URL || 'http://grafana.hacecalor.net:4318',
    sessionTracking: { enabled: true },
    // captures crashes, JS errors, console, ANR(Android), app-start, memory vitals
  });
  ```
- `EXPO_PUBLIC_FARO_URL` in `.env.example` (default the public URL above).
- Optional follow-up: `@grafana/faro-react-native-tracing` for distributed traces + `@grafana/faro-metro-plugin` for source maps (post-MVP).

### Task D — Infra exporters (scrape jobs on `srv84-macos`)
**Files (remote):** `/root/Monitoring/docker-compose.yml`, `/root/Monitoring/prometheus/prometheus.yml`.
- Add containers `redis-exporter` (→ farmafinder redis) and `postgres-exporter` (→ farmafinder postgres) to the Monitoring compose. Reach farmafinder's redis/postgres via the docker gateway / service DNS on the same host.
- Add scrape jobs in `prometheus.yml`:
  - `farmafinder-redis` → redis-exporter
  - `farmafinder-postgres` → postgres-exporter
  - `pip-platform` → `http://<pip-host>:8000/metrics` (pip already exposes it)
- Reload Prometheus (`POST /-/reload`).

### Task E — Grafana dashboards (provisioned JSON)
**Files:** `/root/Monitoring/grafana/provisioning/dashboards/*.json` + `dashboard.yml` entry.
1. `farmafinder-backend.json` — req rate, error %, latency p50/p95/p99 by route, CIMA latency/errors, cache hit rate, Redis/DB errors, push counts, auth failures, rate-limit hits, Node runtime.
2. `farmafinder-datastores.json` — Redis + Postgres + (SQLite via app metric) panels.
3. `farmafinder-mobile-rum.json` — Faro RN data from Loki (`app.name="farmafinder-mobile"`).
4. `farmafinder-overview.json` — combined RUM + backend metrics + deep-links to Tempo (traces) & Loki (logs).
(Keep existing `faro-*` PWA dashboards.)

### Task F — Alerts
**Files (remote):** `/root/Monitoring/prometheus/alert.rules` + Grafana-managed alert.
Prometheus rules (group `farmafinder`):
- `farmafinder_backend_down` — `absent(heartbeat_total{job="farmafinder-backend"}) > 0` for 5m → critical
- `farmafinder_backend_5xx` — `rate(http.server.request.errors ...)` ratio > 5% over 5m → warning
- `farmafinder_cima_errors` — CIMA error rate high → warning
- `farmafinder_high_latency` — p95 `http.server.request.duration` > threshold → warning
- `farmafinder_redis_down` / `farmafinder_postgres_down` / `pip_platform_down` — `up==0`
Grafana-managed alert (Loki): **mobile/PWA JS-error spike** — count of Faro exceptions over 10m > N.
All route to existing Alertmanager (Telegram).

### Task G — Mobile Faro reachability (user action)
- Open router `TCP 4318 → srv84-macos:4318`, **or** fix Nginx Proxy Manager `grafana.hacecalor.net/faro` → `http://<alloy-host>:4318` ensuring OTLP HTTP paths (`/v1/logs`,`/v1/metrics`,`/v1/traces`) are proxied. Verify from a phone / external curl that `POST http://grafana.hacecalor.net:4318/v1/logs` is accepted.

---

## 4. Status (update as you go)

| Task | Status | Notes |
|---|---|---|
| A. Backend metrics | ✅ done (code) | `src/metrics.js`, `tracing.js` meter provider, instrumented server.js/cima-service.js/redis-client.js. Needs backend rebuild+redeploy to take effect. |
| B. Backend health | ✅ done (code) | `/healthz`, `/readyz` added. |
| C. Mobile Faro | ✅ done (code) | `services/faro.ts` + `_layout.tsx` init + dep `@grafana/faro-react-native`. Requires dev/EAS build (native module). |
| D. Infra exporters | ✅ done | redis/postgres exporters added to farmafinder `docker-compose.yml`; Prometheus scrape jobs + alert rules applied on srv84-macos. Needs farmafinder redeploy. |
| E. Dashboards | ✅ done + LIVE | 4 dashboards pushed to `/root/Monitoring/grafana/provisioning/dashboards/` and provisioned in Grafana. |
| F. Alerts | ✅ done + LIVE | `farmafinder` group applied to `alert.rules`; routes to existing Alertmanager (Telegram). |
| G. Mobile Faro URL | ⏳ user action | open :4318 → srv84-macos:4318, or fix Nginx Proxy Manager `/faro`. |

## 5. Deployment & Verification

### Backend (apply on next deploy)
The running `farmafinder-backend-1` still runs old code. To activate metrics:
```bash
# from repo root on srv84-macos (or wherever farmafinder is deployed)
docker compose build backend && docker compose up -d backend
# or push to main and let .gitea deploy
```
Verify:
```bash
# after ~30s, a heartbeat sample should exist
curl -s "http://srv84-macos:9090/api/v1/query?query=app_heartbeat_total{job=%22farmafinder-backend%22}"
# health probes
curl -s http://localhost:3001/healthz ; curl -s http://localhost:3001/readyz
# dashboards populate: Grafana → FarmaFinder — Backend
```

### Mobile (user)
1. `cd apps/frontend-mobile && npm install` (pulls `@grafana/faro-react-native`).
2. Build a **dev/EAS build** (not Expo Go — native module).
3. Set `EXPO_PUBLIC_FARO_URL` (default `http://grafana.hacecalor.net:4318`) reachable from device.
4. Verify in Grafana → FarmaFinder — Mobile RUM (Loki `{app="farmafinder-mobile"}`).

### Datastore exporters (apply on farmafinder redeploy)
`docker compose up -d` brings up `redis-exporter` (`:9121`) and `postgres-exporter` (`:9187`);
Prometheus already scrapes them. Verify: `up{job="farmafinder-redis"}`, `up{job="farmafinder-postgres"}`.

## 6. Verification checklist (end)
- [x] 4 Grafana dashboards live (backend, datastores, mobile-rum, overview)
- [x] Prometheus scrape jobs `farmafinder-redis` / `farmafinder-postgres` present
- [x] Alert rules `farmafinder_*` loaded (no config errors)
- [ ] Backend redeployed → `app_heartbeat_total`, `http_requests_total`, `cima_requests_total`, etc. appear
- [ ] `/healthz`, `/readyz` return 200
- [ ] Mobile build → Faro data in Loki + Mobile RUM dashboard
- [ ] Trigger a test alert (stop backend) → Telegram receives it

## 7. Key commands / references
- Remote stack: `ssh root@srv84-macos`, configs under `/root/Monitoring`.
- Prometheus reload: `curl -X POST http://srv84-macos:9090/-/reload`.
- Grafana dashboards reload: `curl -u admin:foobar -X POST http://srv84-macos:3000/api/admin/provisioning/dashboards/reload`.
- Grafana API: `curl -u admin:foobar http://srv84-macos:3000/api/...` (admin password `foobar`).
- Backend OTel env (in `docker-compose.yml`): `OTEL_SERVICE_NAME=farmafinder-backend`, `OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317`, `OTEL_TRACES_EXPORTER=otlp`, **`OTEL_METRICS_EXPORTER=otlp`** (added), `OTEL_LOGS_EXPORTER=otlp`.
- New metrics module: `apps/backend/src/metrics.js`.
- Dashboards source: `monitoring/dashboards/*.json` (mirrored on the host).