feat: complete N8N scrapers for 6 sources and seed script

- Create comprehensive N8N workflow for all 6 parapharmacy sources
- Create webhook-triggered manual scraping workflow
- Add seed script with 20 sample products
- Update n8n/README.md with complete documentation
- Update docs/parapharmacy.md with seeding instructions
- Add seed script to package.json

Sources configured:
- Promofarma
- Pharmarket
- DocMorris
- 1001Farma
- Primor
- MiFarma
This commit is contained in:
Antoni Nuñez Romeu
2026-07-16 13:00:20 +02:00
parent cb564fb170
commit 30f97fe87d
7 changed files with 973 additions and 68 deletions
+116 -41
View File
@@ -1,89 +1,164 @@
# N8N Workflows for FarmaFinder Parapharmacy
## Workflows Disponibles
| Workflow | Trigger | Estado | Descripción |
|----------|---------|--------|-------------|
| Parapharmacy Scraper - All Sources | Cada 3 días a las 2am | Inactivo | Scraping automático de las 6 tiendas |
| Parapharmacy Manual Scraper | POST `/webhook/scrape-all` | Activo | Scraping manual bajo demanda |
## Configuración Inicial
Al iniciar N8N por primera vez, se creará automáticamente una cuenta de administrador:
### 1. Cuenta de Administrador
- **Email**: admin@farmafinder.com (configurable en `.env`)
- **Password**: change-me (configurable en `.env`)
Al iniciar N8N por primera vez, se crea automáticamente:
### Variables de Entorno
| Campo | Valor |
|-------|-------|
| **Email** | admin@farmafinder.com |
| **Password** | change-me |
> **IMPORTANTE**: Cambia la contraseña después del primer login en http://localhost:5678/settings
### 2. Variables de Entorno
```bash
# En el archivo .env de la raíz del proyecto
# En .env
N8N_USER=admin
N8N_PASSWORD=change-me
N8N_EMAIL=admin@farmafinder.com
```
## Workflows Incluidos
### 3. Importar Workflows
### 1. Parapharmacy Scraper (Automático)
Los workflows se importan automáticamente al iniciar el contenedor. Para importar manualmente:
- **Trigger**: Cada 3 días a las 2:00 AM
- **Función**: Scraping automático de Promofarma
- **Estado**: Inactivo por defecto (activar después de configurar)
1. Ir a http://localhost:5678/workflows
2. Hacer clic en "Import from File"
3. Seleccionar el archivo JSON de `n8n/workflows/`
### 2. Parapharmacy Manual Scraper (Webhook)
## Fuentes de Scraping
- **Trigger**: POST a `/webhook/scrape-parapharmacy`
- **Función**: Scraping bajo demanda
- **Estado**: Activo por defecto
| Fuente | URL Base | Estado |
|--------|----------|--------|
| Promofarma | promofarma.com | ✅ Configurado |
| Pharmarket | pharmarket.es | ✅ Configurado |
| DocMorris | docmorris.es | ✅ Configurado |
| 1001Farma | 1001farma.net | ✅ Configurado |
| Primor | primor.eu | ✅ Configurado |
| MiFarma | mifarma.es | ✅ Configurado |
## Uso del Webhook Manual
## Uso
### Scraping Automático
El workflow "Parapharmacy Scraper - All Sources" se ejecuta automáticamente cada 3 días.
**Para activarlo:**
1. Ir a http://localhost:5678/workflows
2. Abrir "Parapharmacy Scraper - All Sources"
3. Activar el toggle "Active"
### Scraping Manual
```bash
# Ejecutar scraping con queries por defecto
curl -X POST http://localhost:5678/webhook/scrape-parapharmacy
# Scraping con queries por defecto (todas las tiendas)
curl -X POST http://localhost:5678/webhook/scrape-all
# Ejecutar scraping con queries específicas
curl -X POST http://localhost:5678/webhook/scrape-parapharmacy \
# Scraping con queries específicas
curl -X POST http://localhost:5678/webhook/scrape-all \
-H "Content-Type: application/json" \
-d '{"queries": ["crema hidratante", "protector solar"]}'
# Scraping solo de fuentes específicas
curl -X POST http://localhost:5678/webhook/scrape-all \
-H "Content-Type: application/json" \
-d '{"sources": ["promofarma", "pharmarket"]}'
```
## Endpoints de la API de Parafarmacia
### Respuesta del Webhook
Los workflows envían los productos scrapeados a:
```json
{
"success": true,
"message": "Scraping completed",
"products": 15
}
```
## Endpoints de la API
Los workflows envían productos a:
```
POST http://parapharmacy-api:3002/api/products/bulk
```
## Poblar Base de Datos
### Datos de Prueba
```bash
# Ejecutar seed script
cd apps/parapharmacy-api
npm run seed
```
Esto inserta 20 productos de prueba en la base de datos.
### Scraping Real
```bash
# Ejecutar scraping manual
curl -X POST http://localhost:5678/webhook/scrape-all
```
## Monitoreo
- **N8N Dashboard**: http://localhost:5678
- **Dashboard N8N**: http://localhost:5678
- **Historial de ejecuciones**: http://localhost:5678/executions
- **Workflows**: http://localhost:5678/workflows
- **Logs**: `docker logs n8n`
## Adding More Sources
Para añadir nuevas fuentes de scraping:
1. Crear un nuevo nodo HTTP Request en el workflow
2. Añadir un nodo Code para extraer los productos
3. Conectar al nodo "Send to Parapharmacy API"
4. Activar el workflow
## Troubleshooting
### N8N muestra /setup
Si N8N muestra la página de configuración, verifica que las variables de entorno estén configuradas:
```bash
N8N_OWNER_EMAIL=admin@farmafinder.com
N8N_OWNER_PASSWORD=change-me
# Verificar variables de entorno
docker-compose exec n8n env | grep N8N
```
### Webhook no funciona
1. Verifica que el workflow esté activo
2. Revisa el historial de ejecuciones en N8N
3. Verifica los logs: `docker logs n8n`
1. Verificar que el workflow esté activo
2. Revisar historial de ejecuciones
3. Verificar logs: `docker logs n8n`
### Scraping falla
1. Verificar que parapharmacy-api esté corriendo
2. Verificar conexión a MongoDB
3. Revisar logs de ejecución en N8N
4. Probar con una sola fuente: `{"sources": ["promofarma"]}`
### Productos no se guardan
1. Verifica que parapharmacy-api esté ejecutándose
2. Revisa los logs de la API: `docker logs parapharmacy-api`
3. Verifica que MongoDB esté conectado
```bash
# Verificar health check
curl http://localhost:3002/api/health
# Verificar MongoDB
docker-compose exec mongodb mongosh --eval "db.adminCommand('ping')"
```
## Adding New Sources
Para añadir una nueva fuente:
1. Crear un nuevo nodo en el workflow
2. Configurar la URL de la fuente
3. Añadir selectores CSS para extraer productos
4. Conectar al nodo "Send to API"
5. Probar con webhook manual
6. Activar el workflow
+134
View File
@@ -0,0 +1,134 @@
{
"name": "Parapharmacy Scraper - All Sources",
"nodes": [
{
"parameters": {
"rule": {
"interval": [
{
"field": "cronExpression",
"expression": "0 2 */3 * *"
}
]
}
},
"id": "schedule",
"name": "Schedule (Every 3 days)",
"type": "n8n-nodes-base.scheduleTrigger",
"typeVersion": 1.2,
"position": [220, 300]
},
{
"parameters": {
"values": {
"string": [
{
"name": "queries",
"value": "crema hidratante,protector solar,leche corporal,champú bebé,aceite bebé,crema pañal,vitaminas,fórmula láctea,paracetamol,ibuprofeno"
}
]
},
"options": {}
},
"id": "set-queries",
"name": "Set Queries",
"type": "n8n-nodes-base.set",
"typeVersion": 3.4,
"position": [440, 300]
},
{
"parameters": {
"fieldToSplitOut": "queries",
"options": {}
},
"id": "split-queries",
"name": "Split Queries",
"type": "n8n-nodes-base.splitOut",
"typeVersion": 1,
"position": [660, 300]
},
{
"parameters": {
"jsCode": "// Define all sources to scrape\nconst query = $input.first().json.queries;\n\nconst sources = [\n {\n name: 'promofarma',\n url: `https://www.promofarma.com/es/search?q=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-card, article[class*=\"product\"]',\n name: 'h3, h2, [class*=\"title\"]',\n price: '[class*=\"price\"], .current-price',\n link: 'a',\n image: 'img'\n }\n },\n {\n name: 'pharmarket',\n url: `https://www.pharmarket.es/catalogsearch/result/?q=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-item, .item.product',\n name: '.product-item-link, .product-name',\n price: '.price',\n link: 'a.product-item-link',\n image: 'img.product-image-photo'\n }\n },\n {\n name: 'docmorris',\n url: `https://www.docmorris.es/search?query=${encodeURIComponent(query)}`,\n selectors: {\n product: '[class*=\"product-card\"], [class*=\"product-item\"]',\n name: '[class*=\"product-name\"], h3',\n price: '[class*=\"price\"]',\n link: 'a',\n image: 'img'\n }\n },\n {\n name: '1001farma',\n url: `https://www.1001farma.net/buscar?s=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-miniature, article.product-miniature',\n name: '.product-title a, h3 a',\n price: '.price',\n link: '.product-title a',\n image: 'img'\n }\n },\n {\n name: 'primor',\n url: `https://www.primor.eu/search?s=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-item, .product-miniature',\n name: '.product-title, h3',\n price: '.price',\n link: 'a.product-title',\n image: 'img'\n }\n },\n {\n name: 'mifarma',\n url: `https://www.mifarma.es/buscador?q=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-card, .product-item',\n name: '.product-name, h3',\n price: '.price',\n link: 'a',\n image: 'img'\n }\n }\n];\n\nreturn sources.map(s => ({ json: { ...s, query } }));"
},
"id": "define-sources",
"name": "Define Sources",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [880, 300]
},
{
"parameters": {
"method": "GET",
"url": "={{ $json.url }}",
"options": {
"timeout": 30000
}
},
"id": "scrape-source",
"name": "Scrape Source",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [1100, 300]
},
{
"parameters": {
"jsCode": "// Extract products based on source\nconst sourceData = $input.first().json;\nconst html = sourceData.data || '';\nconst source = sourceData.name;\n\nconst products = [];\n\n// Common extraction patterns\nconst patterns = {\n promofarma: {\n productRegex: /<div[^>]*class=\"[^\"]*product-card[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>\\s*<\\/div>/gi,\n nameRegex: /<h[23][^>]*>([^<]+)<\\/h[23]>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n },\n pharmarket: {\n productRegex: /<li[^>]*class=\"[^\"]*product-item[^\"]*\"[^>]*>([\\s\\S]*?)<\\/li>/gi,\n nameRegex: /class=\"[^\"]*product-item-link[^\"]*\"[^>]*>([^<]+)<\\/a>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /class=\"[^\"]*product-item-link[^\"]*\"[^>]*href=\"([^\"]+)\"/i,\n imageRegex: /<img[^>]*class=\"[^\"]*product-image[^\"]*\"[^>]*src=\"([^\"]+)\"/i\n },\n docmorris: {\n productRegex: /<div[^>]*class=\"[^\"]*product-card[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>/gi,\n nameRegex: /class=\"[^\"]*product-name[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n },\n '1001farma': {\n productRegex: /<article[^>]*class=\"[^\"]*product-miniature[^\"]*\"[^>]*>([\\s\\S]*?)<\\/article>/gi,\n nameRegex: /class=\"[^\"]*product-title[^\"]*\"[^>]*>\\s*<a[^>]*>([^<]+)<\\/a>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /class=\"[^\"]*product-title[^\"]*\"[^>]*>\\s*<a[^>]*href=\"([^\"]+)\"/i,\n imageRegex: /<img[^>]*class=\"[^\"]*img[^\"]*\"[^>]*src=\"([^\"]+)\"/i\n },\n primor: {\n productRegex: /<div[^>]*class=\"[^\"]*product-item[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>/gi,\n nameRegex: /class=\"[^\"]*product-title[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n },\n mifarma: {\n productRegex: /<div[^>]*class=\"[^\"]*product-card[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>/gi,\n nameRegex: /class=\"[^\"]*product-name[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n }\n};\n\nconst p = patterns[source] || patterns.promofarma;\nlet match;\n\nwhile ((match = p.productRegex.exec(html)) !== null) {\n const card = match[1];\n const name = p.nameRegex.exec(card)?.[1]?.trim();\n const priceStr = p.priceRegex.exec(card)?.[1]?.trim();\n const link = p.linkRegex.exec(card)?.[1];\n const image = p.imageRegex.exec(card)?.[1];\n \n if (name && name.length > 3) {\n const price = parseFloat(priceStr?.replace(/[^\\d.,]/g, '').replace(',', '.')) || 0;\n const baseUrl = {\n promofarma: 'https://www.promofarma.com',\n pharmarket: 'https://www.pharmarket.es',\n docmorris: 'https://www.docmorris.es',\n '1001farma': 'https://www.1001farma.net',\n primor: 'https://www.primor.eu',\n mifarma: 'https://www.mifarma.es'\n }[source];\n \n products.push({\n name: name.substring(0, 200),\n price,\n source_url: link?.startsWith('http') ? link : `${baseUrl}${link}`,\n image_url: image?.startsWith('http') ? image : (image ? `${baseUrl}${image}` : null),\n source,\n source_product_id: link?.match(/[\\/\\?].*?([\\w-]+)$/)?.[1] || `${source}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,\n brand: '',\n category: 'parapharmacy',\n available: true,\n scraped_at: new Date().toISOString()\n });\n }\n}\n\n// Deduplicate by name\nconst seen = new Set();\nconst unique = products.filter(p => {\n const key = p.name.toLowerCase();\n if (seen.has(key)) return false;\n seen.add(key);\n return true;\n});\n\nreturn unique.slice(0, 10).map(p => ({ json: p }));"
},
"id": "extract-products",
"name": "Extract Products",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [1320, 300]
},
{
"parameters": {
"method": "POST",
"url": "http://parapharmacy-api:3002/api/products/bulk",
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ JSON.stringify({ products: $input.all().map(item => item.json) }) }}",
"options": {}
},
"id": "send-to-api",
"name": "Send to API",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [1540, 300]
},
{
"parameters": {},
"id": "done",
"name": "Done",
"type": "n8n-nodes-base.noOp",
"typeVersion": 1,
"position": [1760, 300]
}
],
"connections": {
"Schedule (Every 3 days)": {
"main": [[{ "node": "Set Queries", "type": "main", "index": 0 }]]
},
"Set Queries": {
"main": [[{ "node": "Split Queries", "type": "main", "index": 0 }]]
},
"Split Queries": {
"main": [[{ "node": "Define Sources", "type": "main", "index": 0 }]]
},
"Define Sources": {
"main": [[{ "node": "Scrape Source", "type": "main", "index": 0 }]]
},
"Scrape Source": {
"main": [[{ "node": "Extract Products", "type": "main", "index": 0 }]]
},
"Extract Products": {
"main": [[{ "node": "Send to API", "type": "main", "index": 0 }]]
},
"Send to API": {
"main": [[{ "node": "Done", "type": "main", "index": 0 }]]
}
},
"active": false,
"settings": { "executionOrder": "v1" },
"tags": [{ "name": "parapharmacy" }, { "name": "scraper" }, { "name": "scheduled" }]
}
@@ -0,0 +1,99 @@
{
"name": "Parapharmacy Manual Scraper (All Sources)",
"nodes": [
{
"parameters": {
"httpMethod": "POST",
"path": "scrape-all",
"options": {}
},
"id": "webhook",
"name": "Webhook",
"type": "n8n-nodes-base.webhook",
"typeVersion": 2,
"position": [220, 300],
"webhookId": "scrape-all"
},
{
"parameters": {
"jsCode": "const body = $input.first().json.body || {};\nconst queries = body.queries || 'crema hidratante,protector solar,vitaminas,paracetamol';\nconst sources = body.sources || ['promofarma', 'pharmarket', 'docmorris', '1001farma', 'primor', 'mifarma'];\n\nreturn [{ json: { queries, sources } }];"
},
"id": "parse-input",
"name": "Parse Input",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [440, 300]
},
{
"parameters": {
"jsCode": "const input = $input.first().json;\nconst queries = input.queries.split(',').map(q => q.trim());\nconst sources = input.sources;\n\nconst tasks = [];\nfor (const query of queries) {\n for (const source of sources) {\n const urls = {\n promofarma: `https://www.promofarma.com/es/search?q=${encodeURIComponent(query)}`,\n pharmarket: `https://www.pharmarket.es/catalogsearch/result/?q=${encodeURIComponent(query)}`,\n docmorris: `https://www.docmorris.es/search?query=${encodeURIComponent(query)}`,\n '1001farma': `https://www.1001farma.net/buscar?s=${encodeURIComponent(query)}`,\n primor: `https://www.primor.eu/search?s=${encodeURIComponent(query)}`,\n mifarma: `https://www.mifarma.es/buscador?q=${encodeURIComponent(query)}`\n };\n \n tasks.push({\n query,\n source,\n url: urls[source]\n });\n }\n}\n\nreturn tasks.map(t => ({ json: t }));"
},
"id": "generate-tasks",
"name": "Generate Tasks",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [660, 300]
},
{
"parameters": {
"method": "GET",
"url": "={{ $json.url }}",
"options": {
"timeout": 30000
}
},
"id": "scrape",
"name": "Scrape",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [880, 300]
},
{
"parameters": {
"jsCode": "const data = $input.first().json;\nconst html = data.data || '';\nconst source = data.source;\nconst query = data.query;\n\nconst products = [];\n\n// Generic extraction\nconst cardRegex = /<(?:div|article|li)[^>]*class=\"[^\"]*(?:product|item)[^\"]*\"[^>]*>([\\s\\S]*?)<\\/(?:div|article|li)>/gi;\nconst nameRegex = /<(?:h[23]|a|span)[^>]*class=\"[^\"]*(?:name|title)[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i;\nconst priceRegex = /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i;\nconst linkRegex = /<a[^>]*href=\"(https?:\\/\\/[^\"]+)\"/i;\nconst imageRegex = /<img[^>]*src=\"(https?:\\/\\/[^\"\\.]+\\.(?:jpg|jpeg|png|webp)[^\"]*)\"/i;\n\nlet match;\nwhile ((match = cardRegex.exec(html)) !== null) {\n const card = match[1];\n const name = nameRegex.exec(card)?.[1]?.trim();\n const priceStr = priceRegex.exec(card)?.[1]?.trim();\n const link = linkRegex.exec(card)?.[1];\n const image = imageRegex.exec(card)?.[1];\n \n if (name && name.length > 3 && name.length < 200) {\n const price = parseFloat(priceStr?.replace(/[^\\d.,]/g, '').replace(',', '.')) || 0;\n if (price > 0 && price < 1000) {\n products.push({\n name: name.substring(0, 200),\n price,\n source_url: link || `https://${source}.com`,\n image_url: image || null,\n source,\n source_product_id: `${source}_${Date.now()}_${Math.random().toString(36).substr(2, 6)}`,\n brand: '',\n category: 'parapharmacy',\n available: true,\n scraped_at: new Date().toISOString()\n });\n }\n }\n}\n\n// Deduplicate\nconst seen = new Set();\nconst unique = products.filter(p => {\n const key = `${source}:${p.name.toLowerCase()}`;\n if (seen.has(key)) return false;\n seen.add(key);\n return true;\n});\n\nreturn unique.slice(0, 5).map(p => ({ json: p }));"
},
"id": "extract",
"name": "Extract Products",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [1100, 300]
},
{
"parameters": {
"method": "POST",
"url": "http://parapharmacy-api:3002/api/products/bulk",
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ JSON.stringify({ products: $input.all().map(i => i.json) }) }}",
"options": {}
},
"id": "send-bulk",
"name": "Send to API",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [1320, 300]
},
{
"parameters": {
"respondWith": "json",
"responseBody": "={{ { success: true, message: 'Scraping completed', products: $input.all().length } }}"
},
"id": "respond",
"name": "Respond",
"type": "n8n-nodes-base.respondToWebhook",
"typeVersion": 1.1,
"position": [1540, 300]
}
],
"connections": {
"Webhook": { "main": [[{ "node": "Parse Input", "type": "main", "index": 0 }]] },
"Parse Input": { "main": [[{ "node": "Generate Tasks", "type": "main", "index": 0 }]] },
"Generate Tasks": { "main": [[{ "node": "Scrape", "type": "main", "index": 0 }]] },
"Scrape": { "main": [[{ "node": "Extract Products", "type": "main", "index": 0 }]] },
"Extract Products": { "main": [[{ "node": "Send to API", "type": "main", "index": 0 }]] },
"Send to API": { "main": [[{ "node": "Respond", "type": "main", "index": 0 }]] }
},
"active": true,
"settings": { "executionOrder": "v1" },
"tags": [{ "name": "parapharmacy" }, { "name": "scraper" }, { "name": "webhook" }]
}