FarmaFinder/PLAN.md

Monitoring Integration Plan for Pharmacy Integration Platform (PIP)

This document outlines how to integrate the PIP application with an existing Prometheus/Grafana/Alertmanager stack running on a remote host. The PIP application already exposes Prometheus metrics at /metrics and includes OpenTelemetry instrumentation for tracing.

Prerequisites

1. Running PIP instance - The application must be accessible from the monitoring host (via network).
2. Existing monitoring stack - Prometheus server, Grafana, and Alertmanager already deployed and reachable.
3. Network access - Ensure the monitoring host can scrape the PIP application's metrics endpoint (default port 8000).
4. Credentials (if applicable) - Any authentication tokens or basic auth required for scraping.

1. Metrics Endpoint

The PIP application exposes Prometheus-formatted metrics at:

http://<pip-host>:8000/metrics

This endpoint is already mounted in the FastAPI application (see src/main.py line 64-65). No further changes are required unless you need to change the path or add authentication.

Optional: Add Basic Auth to Metrics Endpoint

If your Prometheus server requires authentication, you can wrap the metrics endpoint with middleware. Example:

# In src/main.py after creating the app
from starlette.basic_auth import BasicAuthMiddleware
from src.infrastructure.config.settings import settings

if settings.METRICS_AUTH_USERNAME and settings.METRICS_AUTH_PASSWORD:
    app.add_middleware(
        BasicAuthMiddleware,
        username=settings.METRICS_AUTH_USERNAME,
        password=settings.METRICS_AUTH_PASSWORD,
    )

Add the corresponding environment variables to your deployment.

2. Prometheus Configuration

Add a scrape job for the PIP service in your Prometheus configuration file (typically prometheus.yml).

scrape_configs:
  - job_name: 'pip-api'
    static_configs:
      - targets: ['<pip-host>:8000']   # replace with actual host/IP
    # Optional: if you enabled basic auth on /metrics
    # basic_auth:
    #   username: <username>
    #   password: <password>
    # Optional: TLS configuration if using HTTPS
    # scheme: https
    # tls_config:
    #   ca_file: /path/to/ca.pem
    #   cert_file: /path/to/client-cert.pem
    #   key_file: /path/to/client-key.pem
    metric_relabel_configs:
      # Optional: filter or relabel metrics as needed
      - source_labels: [__name__]
        regex: 'pip_.*'
        action: keep

After updating the configuration, reload Prometheus:

curl -X POST http://<prometheus-host>:9090/-/reload

3. Grafana Dashboards

You can import a pre-built dashboard JSON (provided below) or create your own panels using the exposed metrics.

Example Dashboard JSON

Save this as pip-dashboard.json and import via Grafana UI (+ Import → Paste JSON).

{
  "dashboard": {
    "id": null,
    "title": "Pharmacy Integration Platform",
    "timezone": "browser",
    "schemaVersion": 38,
    "version": 1,
    "refresh": "10s",
    "panels": [
      {
        "type": "graph",
        "title": "HTTP Requests Rate",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum by (method, endpoint, http_status) (rate(pip_http_requests_total[5m]))",
            "legendFormat": "{{method}} {{endpoint}} {{http_status}}",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "ops",
            "label": "req/s"
          }
        ],
        "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "Request Duration (seconds)",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum by (le, method, endpoint) (rate(pip_http_request_duration_seconds_bucket[5m])))",
            "legendFormat": "{{method}} {{endpoint}} 95th percentile",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "seconds",
            "label": "seconds"
          }
        ],
        "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "Database Query Duration",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum by (le, operation) (rate(pip_db_query_duration_seconds_bucket[5m])))",
            "legendFormat": "{{operation}} 95th percentile",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "seconds",
            "label": "seconds"
          }
        ],
        "gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "Redis Operations",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum by (operation) (rate(pip_redis_operations_total[5m]))",
            "legendFormat": "{{operation}}",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "ops",
            "label": "ops/s"
          }
        ],
        "gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "RabbitMQ Messages Published",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum by (exchange, routing_key) (rate(pip_rabbitmq_messages_published_total[5m]))",
            "legendFormat": "{{exchange}} -> {{routing_key}}",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "ops",
            "label": "msgs/s"
          }
        ],
        "gridPos": { "x": 0, "y": 16, "w": 24, "h": 8 }
      }
    ],
    "templating": {
      "list": []
    },
    "annotations": {
      "list": []
    }
  },
  "overwrite": true
}

Import Steps

1. In Grafana, click the + icon → Import.
2. Paste the JSON above or upload the file.
3. Select your Prometheus data source.
4. Click Import.

4. Alerting Rules (Optional)

Create a rule file for Alertmanager (e.g., pip_alerts.yml) and add it to your Prometheus server.

groups:
  - name: pip-alerts
    rules:
      - alert: HighRequestLatency
        expr: histogram_quantile(0.95, sum by (le, method, endpoint) (rate(pip_http_request_duration_seconds_bucket[5m]))) > 2
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High request latency on {{ $labels.endpoint }}"
          description: "95th percentile latency is above 2 seconds for {{ $labels.method }} {{ $labels.endpoint }}."

      - alert: APIErrorRateHigh
        expr: sum by (http_status) (rate(pip_http_requests_total{http_status=~"5.."}[5m])) / sum by (http_status) (rate(pip_http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate ({{ $value | printf \"%.2f\" }})"
          description: "More than 5% of requests are returning 5xx errors."

      - alert: RedisDown
        expr: up{job="pip-api"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "PIP application scrape failed"
          description: "Prometheus cannot scrape the PIP application metrics endpoint."

      - alert: DatabaseConnectionFailures
        expr: increase(pip_db_connection_errors_total[5m]) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Database connection errors detected"
          description: "There have been {{ $value }} database connection errors in the last 5 minutes."

Add the file to your Prometheus server's rule files configuration and reload.

5. Verification Steps

1. Verify Metrics Endpoint
   From the monitoring host, run:

   curl -s http://<pip-host>:8000/metrics | head -20
   

   You should see lines like # HELP pip_http_requests_total ... and # TYPE pip_http_requests_total counter.

2. Check Prometheus Targets
   In Prometheus UI (http://<prometheus-host>:9090/targets), ensure the pip-api job shows UP.

3. View Dashboard
   Open the imported dashboard in Grafana and verify panels populate with data.

4. Test Alerts (Optional)
   You can temporarily trigger an alert by modifying the threshold (e.g., set HighRequestLatency to > 0) and verify Alertmanager fires.

6. Cleanup / Roll‑needed) later you decide howto monitor within the docker compose stack again, just edit docker-compose.yml and uncomment the monitoring services (prometheus, grafana, alertmanager) and the related volumes, then run:

docker compose -f pip-platform/docker-compose.yml up -d

Make sure the healthcheck in the pip-api service still points to /api/v1/system/health (already updated).

---

This plan is intended to be a one‑time setup guide. Adjust hostnames, ports, authentication, and resource limits to match your production environment.