# Monitoring Integration Plan for Pharmacy Integration Platform (PIP)

This document outlines how to integrate the PIP application with an existing Prometheus/Grafana/Alertmanager stack running on a remote host. The PIP application already exposes Prometheus metrics at `/metrics` and includes OpenTelemetry instrumentation for tracing.

## Prerequisites

1. **Running PIP instance** - The application must be accessible from the monitoring host (via network).
2. **Existing monitoring stack** - Prometheus server, Grafana, and Alertmanager already deployed and reachable.
3. **Network access** - Ensure the monitoring host can scrape the PIP application's metrics endpoint (default port 8000).
4. **Credentials** (if applicable) - Any authentication tokens or basic auth required for scraping.

## 1. Metrics Endpoint

The PIP application exposes Prometheus-formatted metrics at:

```
http://<pip-host>:8000/metrics
```

This endpoint is already mounted in the FastAPI application (see `src/main.py` line 64-65). No further changes are required unless you need to change the path or add authentication.

### Optional: Add Basic Auth to Metrics Endpoint
If your Prometheus server requires authentication, you can wrap the metrics endpoint with middleware. Example:

```python
# In src/main.py after creating the app
from starlette.basic_auth import BasicAuthMiddleware
from src.infrastructure.config.settings import settings

if settings.METRICS_AUTH_USERNAME and settings.METRICS_AUTH_PASSWORD:
    app.add_middleware(
        BasicAuthMiddleware,
        username=settings.METRICS_AUTH_USERNAME,
        password=settings.METRICS_AUTH_PASSWORD,
    )
```

Add the corresponding environment variables to your deployment.

## 2. Prometheus Configuration

Add a scrape job for the PIP service in your Prometheus configuration file (typically `prometheus.yml`).

```yaml
scrape_configs:
  - job_name: 'pip-api'
    static_configs:
      - targets: ['<pip-host>:8000']   # replace with actual host/IP
    # Optional: if you enabled basic auth on /metrics
    # basic_auth:
    #   username: <username>
    #   password: <password>
    # Optional: TLS configuration if using HTTPS
    # scheme: https
    # tls_config:
    #   ca_file: /path/to/ca.pem
    #   cert_file: /path/to/client-cert.pem
    #   key_file: /path/to/client-key.pem
    metric_relabel_configs:
      # Optional: filter or relabel metrics as needed
      - source_labels: [__name__]
        regex: 'pip_.*'
        action: keep
```

After updating the configuration, reload Prometheus:
```bash
curl -X POST http://<prometheus-host>:9090/-/reload
```

## 3. Grafana Dashboards

You can import a pre-built dashboard JSON (provided below) or create your own panels using the exposed metrics.

### Example Dashboard JSON
Save this as `pip-dashboard.json` and import via Grafana UI (`+ Import` → Paste JSON).

```json
{
  "dashboard": {
    "id": null,
    "title": "Pharmacy Integration Platform",
    "timezone": "browser",
    "schemaVersion": 38,
    "version": 1,
    "refresh": "10s",
    "panels": [
      {
        "type": "graph",
        "title": "HTTP Requests Rate",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum by (method, endpoint, http_status) (rate(pip_http_requests_total[5m]))",
            "legendFormat": "{{method}} {{endpoint}} {{http_status}}",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "ops",
            "label": "req/s"
          }
        ],
        "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "Request Duration (seconds)",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum by (le, method, endpoint) (rate(pip_http_request_duration_seconds_bucket[5m])))",
            "legendFormat": "{{method}} {{endpoint}} 95th percentile",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "seconds",
            "label": "seconds"
          }
        ],
        "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "Database Query Duration",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum by (le, operation) (rate(pip_db_query_duration_seconds_bucket[5m])))",
            "legendFormat": "{{operation}} 95th percentile",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "seconds",
            "label": "seconds"
          }
        ],
        "gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "Redis Operations",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum by (operation) (rate(pip_redis_operations_total[5m]))",
            "legendFormat": "{{operation}}",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "ops",
            "label": "ops/s"
          }
        ],
        "gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 }
      },
      {
        "type": "graph",
        "title": "RabbitMQ Messages Published",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum by (exchange, routing_key) (rate(pip_rabbitmq_messages_published_total[5m]))",
            "legendFormat": "{{exchange}} -> {{routing_key}}",
            "refId": "A"
          }
        ],
        "yAxes": [
          {
            "format": "ops",
            "label": "msgs/s"
          }
        ],
        "gridPos": { "x": 0, "y": 16, "w": 24, "h": 8 }
      }
    ],
    "templating": {
      "list": []
    },
    "annotations": {
      "list": []
    }
  },
  "overwrite": true
}
```

### Import Steps
1. In Grafana, click the **+** icon → **Import**.
2. Paste the JSON above or upload the file.
3. Select your Prometheus data source.
4. Click **Import**.

## 4. Alerting Rules (Optional)

Create a rule file for Alertmanager (e.g., `pip_alerts.yml`) and add it to your Prometheus server.

```yaml
groups:
  - name: pip-alerts
    rules:
      - alert: HighRequestLatency
        expr: histogram_quantile(0.95, sum by (le, method, endpoint) (rate(pip_http_request_duration_seconds_bucket[5m]))) > 2
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High request latency on {{ $labels.endpoint }}"
          description: "95th percentile latency is above 2 seconds for {{ $labels.method }} {{ $labels.endpoint }}."

      - alert: APIErrorRateHigh
        expr: sum by (http_status) (rate(pip_http_requests_total{http_status=~"5.."}[5m])) / sum by (http_status) (rate(pip_http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate ({{ $value | printf \"%.2f\" }})"
          description: "More than 5% of requests are returning 5xx errors."

      - alert: RedisDown
        expr: up{job="pip-api"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "PIP application scrape failed"
          description: "Prometheus cannot scrape the PIP application metrics endpoint."

      - alert: DatabaseConnectionFailures
        expr: increase(pip_db_connection_errors_total[5m]) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Database connection errors detected"
          description: "There have been {{ $value }} database connection errors in the last 5 minutes."
```

Add the file to your Prometheus server's rule files configuration and reload.

## 5. Verification Steps

1. **Verify Metrics Endpoint**  
   From the monitoring host, run:
   ```bash
   curl -s http://<pip-host>:8000/metrics | head -20
   ```
   You should see lines like `# HELP pip_http_requests_total ...` and `# TYPE pip_http_requests_total counter`.

2. **Check Prometheus Targets**  
   In Prometheus UI (`http://<prometheus-host>:9090/targets`), ensure the `pip-api` job shows `UP`.

3. **View Dashboard**  
   Open the imported dashboard in Grafana and verify panels populate with data.

4. **Test Alerts (Optional)**  
   You can temporarily trigger an alert by modifying the threshold (e.g., set `HighRequestLatency` to `> 0`) and verify Alertmanager fires.

## 6. Cleanup / Roll‑needed) later you decide howto monitor within the docker compose stack again, just edit `docker-compose.yml` and uncomment the monitoring services (prometheus, grafana, alertmanager) and the related volumes, then run:

```
docker compose -f pip-platform/docker-compose.yml up -d
```

Make sure the healthcheck in the `pip-api` service still points to `/api/v1/system/health` (already updated).

--- 

*This plan is intended to be a one‑time setup guide. Adjust hostnames, ports, authentication, and resource limits to match your production environment.*