Antoni Nuñez Romeu
276 lines
8.9 KiB
Markdown
276 lines
8.9 KiB
Markdown
# Monitoring Integration Plan for Pharmacy Integration Platform (PIP)
|
||
|
||
This document outlines how to integrate the PIP application with an existing Prometheus/Grafana/Alertmanager stack running on a remote host. The PIP application already exposes Prometheus metrics at `/metrics` and includes OpenTelemetry instrumentation for tracing.
|
||
|
||
## Prerequisites
|
||
|
||
1. **Running PIP instance** - The application must be accessible from the monitoring host (via network).
|
||
2. **Existing monitoring stack** - Prometheus server, Grafana, and Alertmanager already deployed and reachable.
|
||
3. **Network access** - Ensure the monitoring host can scrape the PIP application's metrics endpoint (default port 8000).
|
||
4. **Credentials** (if applicable) - Any authentication tokens or basic auth required for scraping.
|
||
|
||
## 1. Metrics Endpoint
|
||
|
||
The PIP application exposes Prometheus-formatted metrics at:
|
||
|
||
```
|
||
http://<pip-host>:8000/metrics
|
||
```
|
||
|
||
This endpoint is already mounted in the FastAPI application (see `src/main.py` line 64-65). No further changes are required unless you need to change the path or add authentication.
|
||
|
||
### Optional: Add Basic Auth to Metrics Endpoint
|
||
If your Prometheus server requires authentication, you can wrap the metrics endpoint with middleware. Example:
|
||
|
||
```python
|
||
# In src/main.py after creating the app
|
||
from starlette.basic_auth import BasicAuthMiddleware
|
||
from src.infrastructure.config.settings import settings
|
||
|
||
if settings.METRICS_AUTH_USERNAME and settings.METRICS_AUTH_PASSWORD:
|
||
app.add_middleware(
|
||
BasicAuthMiddleware,
|
||
username=settings.METRICS_AUTH_USERNAME,
|
||
password=settings.METRICS_AUTH_PASSWORD,
|
||
)
|
||
```
|
||
|
||
Add the corresponding environment variables to your deployment.
|
||
|
||
## 2. Prometheus Configuration
|
||
|
||
Add a scrape job for the PIP service in your Prometheus configuration file (typically `prometheus.yml`).
|
||
|
||
```yaml
|
||
scrape_configs:
|
||
- job_name: 'pip-api'
|
||
static_configs:
|
||
- targets: ['<pip-host>:8000'] # replace with actual host/IP
|
||
# Optional: if you enabled basic auth on /metrics
|
||
# basic_auth:
|
||
# username: <username>
|
||
# password: <password>
|
||
# Optional: TLS configuration if using HTTPS
|
||
# scheme: https
|
||
# tls_config:
|
||
# ca_file: /path/to/ca.pem
|
||
# cert_file: /path/to/client-cert.pem
|
||
# key_file: /path/to/client-key.pem
|
||
metric_relabel_configs:
|
||
# Optional: filter or relabel metrics as needed
|
||
- source_labels: [__name__]
|
||
regex: 'pip_.*'
|
||
action: keep
|
||
```
|
||
|
||
After updating the configuration, reload Prometheus:
|
||
```bash
|
||
curl -X POST http://<prometheus-host>:9090/-/reload
|
||
```
|
||
|
||
## 3. Grafana Dashboards
|
||
|
||
You can import a pre-built dashboard JSON (provided below) or create your own panels using the exposed metrics.
|
||
|
||
### Example Dashboard JSON
|
||
Save this as `pip-dashboard.json` and import via Grafana UI (`+ Import` → Paste JSON).
|
||
|
||
```json
|
||
{
|
||
"dashboard": {
|
||
"id": null,
|
||
"title": "Pharmacy Integration Platform",
|
||
"timezone": "browser",
|
||
"schemaVersion": 38,
|
||
"version": 1,
|
||
"refresh": "10s",
|
||
"panels": [
|
||
{
|
||
"type": "graph",
|
||
"title": "HTTP Requests Rate",
|
||
"datasource": "Prometheus",
|
||
"targets": [
|
||
{
|
||
"expr": "sum by (method, endpoint, http_status) (rate(pip_http_requests_total[5m]))",
|
||
"legendFormat": "{{method}} {{endpoint}} {{http_status}}",
|
||
"refId": "A"
|
||
}
|
||
],
|
||
"yAxes": [
|
||
{
|
||
"format": "ops",
|
||
"label": "req/s"
|
||
}
|
||
],
|
||
"gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
|
||
},
|
||
{
|
||
"type": "graph",
|
||
"title": "Request Duration (seconds)",
|
||
"datasource": "Prometheus",
|
||
"targets": [
|
||
{
|
||
"expr": "histogram_quantile(0.95, sum by (le, method, endpoint) (rate(pip_http_request_duration_seconds_bucket[5m])))",
|
||
"legendFormat": "{{method}} {{endpoint}} 95th percentile",
|
||
"refId": "A"
|
||
}
|
||
],
|
||
"yAxes": [
|
||
{
|
||
"format": "seconds",
|
||
"label": "seconds"
|
||
}
|
||
],
|
||
"gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 }
|
||
},
|
||
{
|
||
"type": "graph",
|
||
"title": "Database Query Duration",
|
||
"datasource": "Prometheus",
|
||
"targets": [
|
||
{
|
||
"expr": "histogram_quantile(0.95, sum by (le, operation) (rate(pip_db_query_duration_seconds_bucket[5m])))",
|
||
"legendFormat": "{{operation}} 95th percentile",
|
||
"refId": "A"
|
||
}
|
||
],
|
||
"yAxes": [
|
||
{
|
||
"format": "seconds",
|
||
"label": "seconds"
|
||
}
|
||
],
|
||
"gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 }
|
||
},
|
||
{
|
||
"type": "graph",
|
||
"title": "Redis Operations",
|
||
"datasource": "Prometheus",
|
||
"targets": [
|
||
{
|
||
"expr": "sum by (operation) (rate(pip_redis_operations_total[5m]))",
|
||
"legendFormat": "{{operation}}",
|
||
"refId": "A"
|
||
}
|
||
],
|
||
"yAxes": [
|
||
{
|
||
"format": "ops",
|
||
"label": "ops/s"
|
||
}
|
||
],
|
||
"gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 }
|
||
},
|
||
{
|
||
"type": "graph",
|
||
"title": "RabbitMQ Messages Published",
|
||
"datasource": "Prometheus",
|
||
"targets": [
|
||
{
|
||
"expr": "sum by (exchange, routing_key) (rate(pip_rabbitmq_messages_published_total[5m]))",
|
||
"legendFormat": "{{exchange}} -> {{routing_key}}",
|
||
"refId": "A"
|
||
}
|
||
],
|
||
"yAxes": [
|
||
{
|
||
"format": "ops",
|
||
"label": "msgs/s"
|
||
}
|
||
],
|
||
"gridPos": { "x": 0, "y": 16, "w": 24, "h": 8 }
|
||
}
|
||
],
|
||
"templating": {
|
||
"list": []
|
||
},
|
||
"annotations": {
|
||
"list": []
|
||
}
|
||
},
|
||
"overwrite": true
|
||
}
|
||
```
|
||
|
||
### Import Steps
|
||
1. In Grafana, click the **+** icon → **Import**.
|
||
2. Paste the JSON above or upload the file.
|
||
3. Select your Prometheus data source.
|
||
4. Click **Import**.
|
||
|
||
## 4. Alerting Rules (Optional)
|
||
|
||
Create a rule file for Alertmanager (e.g., `pip_alerts.yml`) and add it to your Prometheus server.
|
||
|
||
```yaml
|
||
groups:
|
||
- name: pip-alerts
|
||
rules:
|
||
- alert: HighRequestLatency
|
||
expr: histogram_quantile(0.95, sum by (le, method, endpoint) (rate(pip_http_request_duration_seconds_bucket[5m]))) > 2
|
||
for: 2m
|
||
labels:
|
||
severity: warning
|
||
annotations:
|
||
summary: "High request latency on {{ $labels.endpoint }}"
|
||
description: "95th percentile latency is above 2 seconds for {{ $labels.method }} {{ $labels.endpoint }}."
|
||
|
||
- alert: APIErrorRateHigh
|
||
expr: sum by (http_status) (rate(pip_http_requests_total{http_status=~"5.."}[5m])) / sum by (http_status) (rate(pip_http_requests_total[5m])) > 0.05
|
||
for: 5m
|
||
labels:
|
||
severity: critical
|
||
annotations:
|
||
summary: "High error rate ({{ $value | printf \"%.2f\" }})"
|
||
description: "More than 5% of requests are returning 5xx errors."
|
||
|
||
- alert: RedisDown
|
||
expr: up{job="pip-api"} == 0
|
||
for: 1m
|
||
labels:
|
||
severity: critical
|
||
annotations:
|
||
summary: "PIP application scrape failed"
|
||
description: "Prometheus cannot scrape the PIP application metrics endpoint."
|
||
|
||
- alert: DatabaseConnectionFailures
|
||
expr: increase(pip_db_connection_errors_total[5m]) > 0
|
||
for: 2m
|
||
labels:
|
||
severity: warning
|
||
annotations:
|
||
summary: "Database connection errors detected"
|
||
description: "There have been {{ $value }} database connection errors in the last 5 minutes."
|
||
```
|
||
|
||
Add the file to your Prometheus server's rule files configuration and reload.
|
||
|
||
## 5. Verification Steps
|
||
|
||
1. **Verify Metrics Endpoint**
|
||
From the monitoring host, run:
|
||
```bash
|
||
curl -s http://<pip-host>:8000/metrics | head -20
|
||
```
|
||
You should see lines like `# HELP pip_http_requests_total ...` and `# TYPE pip_http_requests_total counter`.
|
||
|
||
2. **Check Prometheus Targets**
|
||
In Prometheus UI (`http://<prometheus-host>:9090/targets`), ensure the `pip-api` job shows `UP`.
|
||
|
||
3. **View Dashboard**
|
||
Open the imported dashboard in Grafana and verify panels populate with data.
|
||
|
||
4. **Test Alerts (Optional)**
|
||
You can temporarily trigger an alert by modifying the threshold (e.g., set `HighRequestLatency` to `> 0`) and verify Alertmanager fires.
|
||
|
||
## 6. Cleanup / Roll‑needed) later you decide howto monitor within the docker compose stack again, just edit `docker-compose.yml` and uncomment the monitoring services (prometheus, grafana, alertmanager) and the related volumes, then run:
|
||
|
||
```
|
||
docker compose -f pip-platform/docker-compose.yml up -d
|
||
```
|
||
|
||
Make sure the healthcheck in the `pip-api` service still points to `/api/v1/system/health` (already updated).
|
||
|
||
---
|
||
|
||
*This plan is intended to be a one‑time setup guide. Adjust hostnames, ports, authentication, and resource limits to match your production environment.* |