feat: complete N8N scrapers for 6 sources and seed script
- Create comprehensive N8N workflow for all 6 parapharmacy sources - Create webhook-triggered manual scraping workflow - Add seed script with 20 sample products - Update n8n/README.md with complete documentation - Update docs/parapharmacy.md with seeding instructions - Add seed script to package.json Sources configured: - Promofarma - Pharmarket - DocMorris - 1001Farma - Primor - MiFarma
This commit is contained in:
+116
-41
@@ -1,89 +1,164 @@
|
||||
# N8N Workflows for FarmaFinder Parapharmacy
|
||||
|
||||
## Workflows Disponibles
|
||||
|
||||
| Workflow | Trigger | Estado | Descripción |
|
||||
|----------|---------|--------|-------------|
|
||||
| Parapharmacy Scraper - All Sources | Cada 3 días a las 2am | Inactivo | Scraping automático de las 6 tiendas |
|
||||
| Parapharmacy Manual Scraper | POST `/webhook/scrape-all` | Activo | Scraping manual bajo demanda |
|
||||
|
||||
## Configuración Inicial
|
||||
|
||||
Al iniciar N8N por primera vez, se creará automáticamente una cuenta de administrador:
|
||||
### 1. Cuenta de Administrador
|
||||
|
||||
- **Email**: admin@farmafinder.com (configurable en `.env`)
|
||||
- **Password**: change-me (configurable en `.env`)
|
||||
Al iniciar N8N por primera vez, se crea automáticamente:
|
||||
|
||||
### Variables de Entorno
|
||||
| Campo | Valor |
|
||||
|-------|-------|
|
||||
| **Email** | admin@farmafinder.com |
|
||||
| **Password** | change-me |
|
||||
|
||||
> **IMPORTANTE**: Cambia la contraseña después del primer login en http://localhost:5678/settings
|
||||
|
||||
### 2. Variables de Entorno
|
||||
|
||||
```bash
|
||||
# En el archivo .env de la raíz del proyecto
|
||||
# En .env
|
||||
N8N_USER=admin
|
||||
N8N_PASSWORD=change-me
|
||||
N8N_EMAIL=admin@farmafinder.com
|
||||
```
|
||||
|
||||
## Workflows Incluidos
|
||||
### 3. Importar Workflows
|
||||
|
||||
### 1. Parapharmacy Scraper (Automático)
|
||||
Los workflows se importan automáticamente al iniciar el contenedor. Para importar manualmente:
|
||||
|
||||
- **Trigger**: Cada 3 días a las 2:00 AM
|
||||
- **Función**: Scraping automático de Promofarma
|
||||
- **Estado**: Inactivo por defecto (activar después de configurar)
|
||||
1. Ir a http://localhost:5678/workflows
|
||||
2. Hacer clic en "Import from File"
|
||||
3. Seleccionar el archivo JSON de `n8n/workflows/`
|
||||
|
||||
### 2. Parapharmacy Manual Scraper (Webhook)
|
||||
## Fuentes de Scraping
|
||||
|
||||
- **Trigger**: POST a `/webhook/scrape-parapharmacy`
|
||||
- **Función**: Scraping bajo demanda
|
||||
- **Estado**: Activo por defecto
|
||||
| Fuente | URL Base | Estado |
|
||||
|--------|----------|--------|
|
||||
| Promofarma | promofarma.com | ✅ Configurado |
|
||||
| Pharmarket | pharmarket.es | ✅ Configurado |
|
||||
| DocMorris | docmorris.es | ✅ Configurado |
|
||||
| 1001Farma | 1001farma.net | ✅ Configurado |
|
||||
| Primor | primor.eu | ✅ Configurado |
|
||||
| MiFarma | mifarma.es | ✅ Configurado |
|
||||
|
||||
## Uso del Webhook Manual
|
||||
## Uso
|
||||
|
||||
### Scraping Automático
|
||||
|
||||
El workflow "Parapharmacy Scraper - All Sources" se ejecuta automáticamente cada 3 días.
|
||||
|
||||
**Para activarlo:**
|
||||
1. Ir a http://localhost:5678/workflows
|
||||
2. Abrir "Parapharmacy Scraper - All Sources"
|
||||
3. Activar el toggle "Active"
|
||||
|
||||
### Scraping Manual
|
||||
|
||||
```bash
|
||||
# Ejecutar scraping con queries por defecto
|
||||
curl -X POST http://localhost:5678/webhook/scrape-parapharmacy
|
||||
# Scraping con queries por defecto (todas las tiendas)
|
||||
curl -X POST http://localhost:5678/webhook/scrape-all
|
||||
|
||||
# Ejecutar scraping con queries específicas
|
||||
curl -X POST http://localhost:5678/webhook/scrape-parapharmacy \
|
||||
# Scraping con queries específicas
|
||||
curl -X POST http://localhost:5678/webhook/scrape-all \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"queries": ["crema hidratante", "protector solar"]}'
|
||||
|
||||
# Scraping solo de fuentes específicas
|
||||
curl -X POST http://localhost:5678/webhook/scrape-all \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"sources": ["promofarma", "pharmarket"]}'
|
||||
```
|
||||
|
||||
## Endpoints de la API de Parafarmacia
|
||||
### Respuesta del Webhook
|
||||
|
||||
Los workflows envían los productos scrapeados a:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Scraping completed",
|
||||
"products": 15
|
||||
}
|
||||
```
|
||||
|
||||
## Endpoints de la API
|
||||
|
||||
Los workflows envían productos a:
|
||||
|
||||
```
|
||||
POST http://parapharmacy-api:3002/api/products/bulk
|
||||
```
|
||||
|
||||
## Poblar Base de Datos
|
||||
|
||||
### Datos de Prueba
|
||||
|
||||
```bash
|
||||
# Ejecutar seed script
|
||||
cd apps/parapharmacy-api
|
||||
npm run seed
|
||||
```
|
||||
|
||||
Esto inserta 20 productos de prueba en la base de datos.
|
||||
|
||||
### Scraping Real
|
||||
|
||||
```bash
|
||||
# Ejecutar scraping manual
|
||||
curl -X POST http://localhost:5678/webhook/scrape-all
|
||||
```
|
||||
|
||||
## Monitoreo
|
||||
|
||||
- **N8N Dashboard**: http://localhost:5678
|
||||
- **Dashboard N8N**: http://localhost:5678
|
||||
- **Historial de ejecuciones**: http://localhost:5678/executions
|
||||
- **Workflows**: http://localhost:5678/workflows
|
||||
- **Logs**: `docker logs n8n`
|
||||
|
||||
## Adding More Sources
|
||||
|
||||
Para añadir nuevas fuentes de scraping:
|
||||
|
||||
1. Crear un nuevo nodo HTTP Request en el workflow
|
||||
2. Añadir un nodo Code para extraer los productos
|
||||
3. Conectar al nodo "Send to Parapharmacy API"
|
||||
4. Activar el workflow
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### N8N muestra /setup
|
||||
|
||||
Si N8N muestra la página de configuración, verifica que las variables de entorno estén configuradas:
|
||||
|
||||
```bash
|
||||
N8N_OWNER_EMAIL=admin@farmafinder.com
|
||||
N8N_OWNER_PASSWORD=change-me
|
||||
# Verificar variables de entorno
|
||||
docker-compose exec n8n env | grep N8N
|
||||
```
|
||||
|
||||
### Webhook no funciona
|
||||
|
||||
1. Verifica que el workflow esté activo
|
||||
2. Revisa el historial de ejecuciones en N8N
|
||||
3. Verifica los logs: `docker logs n8n`
|
||||
1. Verificar que el workflow esté activo
|
||||
2. Revisar historial de ejecuciones
|
||||
3. Verificar logs: `docker logs n8n`
|
||||
|
||||
### Scraping falla
|
||||
|
||||
1. Verificar que parapharmacy-api esté corriendo
|
||||
2. Verificar conexión a MongoDB
|
||||
3. Revisar logs de ejecución en N8N
|
||||
4. Probar con una sola fuente: `{"sources": ["promofarma"]}`
|
||||
|
||||
### Productos no se guardan
|
||||
|
||||
1. Verifica que parapharmacy-api esté ejecutándose
|
||||
2. Revisa los logs de la API: `docker logs parapharmacy-api`
|
||||
3. Verifica que MongoDB esté conectado
|
||||
```bash
|
||||
# Verificar health check
|
||||
curl http://localhost:3002/api/health
|
||||
|
||||
# Verificar MongoDB
|
||||
docker-compose exec mongodb mongosh --eval "db.adminCommand('ping')"
|
||||
```
|
||||
|
||||
## Adding New Sources
|
||||
|
||||
Para añadir una nueva fuente:
|
||||
|
||||
1. Crear un nuevo nodo en el workflow
|
||||
2. Configurar la URL de la fuente
|
||||
3. Añadir selectores CSS para extraer productos
|
||||
4. Conectar al nodo "Send to API"
|
||||
5. Probar con webhook manual
|
||||
6. Activar el workflow
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"name": "Parapharmacy Scraper - All Sources",
|
||||
"nodes": [
|
||||
{
|
||||
"parameters": {
|
||||
"rule": {
|
||||
"interval": [
|
||||
{
|
||||
"field": "cronExpression",
|
||||
"expression": "0 2 */3 * *"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"id": "schedule",
|
||||
"name": "Schedule (Every 3 days)",
|
||||
"type": "n8n-nodes-base.scheduleTrigger",
|
||||
"typeVersion": 1.2,
|
||||
"position": [220, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"values": {
|
||||
"string": [
|
||||
{
|
||||
"name": "queries",
|
||||
"value": "crema hidratante,protector solar,leche corporal,champú bebé,aceite bebé,crema pañal,vitaminas,fórmula láctea,paracetamol,ibuprofeno"
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {}
|
||||
},
|
||||
"id": "set-queries",
|
||||
"name": "Set Queries",
|
||||
"type": "n8n-nodes-base.set",
|
||||
"typeVersion": 3.4,
|
||||
"position": [440, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"fieldToSplitOut": "queries",
|
||||
"options": {}
|
||||
},
|
||||
"id": "split-queries",
|
||||
"name": "Split Queries",
|
||||
"type": "n8n-nodes-base.splitOut",
|
||||
"typeVersion": 1,
|
||||
"position": [660, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "// Define all sources to scrape\nconst query = $input.first().json.queries;\n\nconst sources = [\n {\n name: 'promofarma',\n url: `https://www.promofarma.com/es/search?q=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-card, article[class*=\"product\"]',\n name: 'h3, h2, [class*=\"title\"]',\n price: '[class*=\"price\"], .current-price',\n link: 'a',\n image: 'img'\n }\n },\n {\n name: 'pharmarket',\n url: `https://www.pharmarket.es/catalogsearch/result/?q=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-item, .item.product',\n name: '.product-item-link, .product-name',\n price: '.price',\n link: 'a.product-item-link',\n image: 'img.product-image-photo'\n }\n },\n {\n name: 'docmorris',\n url: `https://www.docmorris.es/search?query=${encodeURIComponent(query)}`,\n selectors: {\n product: '[class*=\"product-card\"], [class*=\"product-item\"]',\n name: '[class*=\"product-name\"], h3',\n price: '[class*=\"price\"]',\n link: 'a',\n image: 'img'\n }\n },\n {\n name: '1001farma',\n url: `https://www.1001farma.net/buscar?s=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-miniature, article.product-miniature',\n name: '.product-title a, h3 a',\n price: '.price',\n link: '.product-title a',\n image: 'img'\n }\n },\n {\n name: 'primor',\n url: `https://www.primor.eu/search?s=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-item, .product-miniature',\n name: '.product-title, h3',\n price: '.price',\n link: 'a.product-title',\n image: 'img'\n }\n },\n {\n name: 'mifarma',\n url: `https://www.mifarma.es/buscador?q=${encodeURIComponent(query)}`,\n selectors: {\n product: '.product-card, .product-item',\n name: '.product-name, h3',\n price: '.price',\n link: 'a',\n image: 'img'\n }\n }\n];\n\nreturn sources.map(s => ({ json: { ...s, query } }));"
|
||||
},
|
||||
"id": "define-sources",
|
||||
"name": "Define Sources",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [880, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "GET",
|
||||
"url": "={{ $json.url }}",
|
||||
"options": {
|
||||
"timeout": 30000
|
||||
}
|
||||
},
|
||||
"id": "scrape-source",
|
||||
"name": "Scrape Source",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [1100, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "// Extract products based on source\nconst sourceData = $input.first().json;\nconst html = sourceData.data || '';\nconst source = sourceData.name;\n\nconst products = [];\n\n// Common extraction patterns\nconst patterns = {\n promofarma: {\n productRegex: /<div[^>]*class=\"[^\"]*product-card[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>\\s*<\\/div>/gi,\n nameRegex: /<h[23][^>]*>([^<]+)<\\/h[23]>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n },\n pharmarket: {\n productRegex: /<li[^>]*class=\"[^\"]*product-item[^\"]*\"[^>]*>([\\s\\S]*?)<\\/li>/gi,\n nameRegex: /class=\"[^\"]*product-item-link[^\"]*\"[^>]*>([^<]+)<\\/a>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /class=\"[^\"]*product-item-link[^\"]*\"[^>]*href=\"([^\"]+)\"/i,\n imageRegex: /<img[^>]*class=\"[^\"]*product-image[^\"]*\"[^>]*src=\"([^\"]+)\"/i\n },\n docmorris: {\n productRegex: /<div[^>]*class=\"[^\"]*product-card[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>/gi,\n nameRegex: /class=\"[^\"]*product-name[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n },\n '1001farma': {\n productRegex: /<article[^>]*class=\"[^\"]*product-miniature[^\"]*\"[^>]*>([\\s\\S]*?)<\\/article>/gi,\n nameRegex: /class=\"[^\"]*product-title[^\"]*\"[^>]*>\\s*<a[^>]*>([^<]+)<\\/a>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /class=\"[^\"]*product-title[^\"]*\"[^>]*>\\s*<a[^>]*href=\"([^\"]+)\"/i,\n imageRegex: /<img[^>]*class=\"[^\"]*img[^\"]*\"[^>]*src=\"([^\"]+)\"/i\n },\n primor: {\n productRegex: /<div[^>]*class=\"[^\"]*product-item[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>/gi,\n nameRegex: /class=\"[^\"]*product-title[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n },\n mifarma: {\n productRegex: /<div[^>]*class=\"[^\"]*product-card[^\"]*\"[^>]*>([\\s\\S]*?)<\\/div>/gi,\n nameRegex: /class=\"[^\"]*product-name[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i,\n priceRegex: /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i,\n linkRegex: /<a[^>]*href=\"([^\"]+)\"[^>]*>/i,\n imageRegex: /<img[^>]*src=\"([^\"]+)\"[^>]*>/i\n }\n};\n\nconst p = patterns[source] || patterns.promofarma;\nlet match;\n\nwhile ((match = p.productRegex.exec(html)) !== null) {\n const card = match[1];\n const name = p.nameRegex.exec(card)?.[1]?.trim();\n const priceStr = p.priceRegex.exec(card)?.[1]?.trim();\n const link = p.linkRegex.exec(card)?.[1];\n const image = p.imageRegex.exec(card)?.[1];\n \n if (name && name.length > 3) {\n const price = parseFloat(priceStr?.replace(/[^\\d.,]/g, '').replace(',', '.')) || 0;\n const baseUrl = {\n promofarma: 'https://www.promofarma.com',\n pharmarket: 'https://www.pharmarket.es',\n docmorris: 'https://www.docmorris.es',\n '1001farma': 'https://www.1001farma.net',\n primor: 'https://www.primor.eu',\n mifarma: 'https://www.mifarma.es'\n }[source];\n \n products.push({\n name: name.substring(0, 200),\n price,\n source_url: link?.startsWith('http') ? link : `${baseUrl}${link}`,\n image_url: image?.startsWith('http') ? image : (image ? `${baseUrl}${image}` : null),\n source,\n source_product_id: link?.match(/[\\/\\?].*?([\\w-]+)$/)?.[1] || `${source}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,\n brand: '',\n category: 'parapharmacy',\n available: true,\n scraped_at: new Date().toISOString()\n });\n }\n}\n\n// Deduplicate by name\nconst seen = new Set();\nconst unique = products.filter(p => {\n const key = p.name.toLowerCase();\n if (seen.has(key)) return false;\n seen.add(key);\n return true;\n});\n\nreturn unique.slice(0, 10).map(p => ({ json: p }));"
|
||||
},
|
||||
"id": "extract-products",
|
||||
"name": "Extract Products",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [1320, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "http://parapharmacy-api:3002/api/products/bulk",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ JSON.stringify({ products: $input.all().map(item => item.json) }) }}",
|
||||
"options": {}
|
||||
},
|
||||
"id": "send-to-api",
|
||||
"name": "Send to API",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [1540, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {},
|
||||
"id": "done",
|
||||
"name": "Done",
|
||||
"type": "n8n-nodes-base.noOp",
|
||||
"typeVersion": 1,
|
||||
"position": [1760, 300]
|
||||
}
|
||||
],
|
||||
"connections": {
|
||||
"Schedule (Every 3 days)": {
|
||||
"main": [[{ "node": "Set Queries", "type": "main", "index": 0 }]]
|
||||
},
|
||||
"Set Queries": {
|
||||
"main": [[{ "node": "Split Queries", "type": "main", "index": 0 }]]
|
||||
},
|
||||
"Split Queries": {
|
||||
"main": [[{ "node": "Define Sources", "type": "main", "index": 0 }]]
|
||||
},
|
||||
"Define Sources": {
|
||||
"main": [[{ "node": "Scrape Source", "type": "main", "index": 0 }]]
|
||||
},
|
||||
"Scrape Source": {
|
||||
"main": [[{ "node": "Extract Products", "type": "main", "index": 0 }]]
|
||||
},
|
||||
"Extract Products": {
|
||||
"main": [[{ "node": "Send to API", "type": "main", "index": 0 }]]
|
||||
},
|
||||
"Send to API": {
|
||||
"main": [[{ "node": "Done", "type": "main", "index": 0 }]]
|
||||
}
|
||||
},
|
||||
"active": false,
|
||||
"settings": { "executionOrder": "v1" },
|
||||
"tags": [{ "name": "parapharmacy" }, { "name": "scraper" }, { "name": "scheduled" }]
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"name": "Parapharmacy Manual Scraper (All Sources)",
|
||||
"nodes": [
|
||||
{
|
||||
"parameters": {
|
||||
"httpMethod": "POST",
|
||||
"path": "scrape-all",
|
||||
"options": {}
|
||||
},
|
||||
"id": "webhook",
|
||||
"name": "Webhook",
|
||||
"type": "n8n-nodes-base.webhook",
|
||||
"typeVersion": 2,
|
||||
"position": [220, 300],
|
||||
"webhookId": "scrape-all"
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "const body = $input.first().json.body || {};\nconst queries = body.queries || 'crema hidratante,protector solar,vitaminas,paracetamol';\nconst sources = body.sources || ['promofarma', 'pharmarket', 'docmorris', '1001farma', 'primor', 'mifarma'];\n\nreturn [{ json: { queries, sources } }];"
|
||||
},
|
||||
"id": "parse-input",
|
||||
"name": "Parse Input",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [440, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "const input = $input.first().json;\nconst queries = input.queries.split(',').map(q => q.trim());\nconst sources = input.sources;\n\nconst tasks = [];\nfor (const query of queries) {\n for (const source of sources) {\n const urls = {\n promofarma: `https://www.promofarma.com/es/search?q=${encodeURIComponent(query)}`,\n pharmarket: `https://www.pharmarket.es/catalogsearch/result/?q=${encodeURIComponent(query)}`,\n docmorris: `https://www.docmorris.es/search?query=${encodeURIComponent(query)}`,\n '1001farma': `https://www.1001farma.net/buscar?s=${encodeURIComponent(query)}`,\n primor: `https://www.primor.eu/search?s=${encodeURIComponent(query)}`,\n mifarma: `https://www.mifarma.es/buscador?q=${encodeURIComponent(query)}`\n };\n \n tasks.push({\n query,\n source,\n url: urls[source]\n });\n }\n}\n\nreturn tasks.map(t => ({ json: t }));"
|
||||
},
|
||||
"id": "generate-tasks",
|
||||
"name": "Generate Tasks",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [660, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "GET",
|
||||
"url": "={{ $json.url }}",
|
||||
"options": {
|
||||
"timeout": 30000
|
||||
}
|
||||
},
|
||||
"id": "scrape",
|
||||
"name": "Scrape",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [880, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "const data = $input.first().json;\nconst html = data.data || '';\nconst source = data.source;\nconst query = data.query;\n\nconst products = [];\n\n// Generic extraction\nconst cardRegex = /<(?:div|article|li)[^>]*class=\"[^\"]*(?:product|item)[^\"]*\"[^>]*>([\\s\\S]*?)<\\/(?:div|article|li)>/gi;\nconst nameRegex = /<(?:h[23]|a|span)[^>]*class=\"[^\"]*(?:name|title)[^\"]*\"[^>]*>([^<]+)<\\/[^>]+>/i;\nconst priceRegex = /class=\"[^\"]*price[^\"]*\"[^>]*>([^<]*\\d+[.,]\\d+[^<]*)<\\/[^>]+>/i;\nconst linkRegex = /<a[^>]*href=\"(https?:\\/\\/[^\"]+)\"/i;\nconst imageRegex = /<img[^>]*src=\"(https?:\\/\\/[^\"\\.]+\\.(?:jpg|jpeg|png|webp)[^\"]*)\"/i;\n\nlet match;\nwhile ((match = cardRegex.exec(html)) !== null) {\n const card = match[1];\n const name = nameRegex.exec(card)?.[1]?.trim();\n const priceStr = priceRegex.exec(card)?.[1]?.trim();\n const link = linkRegex.exec(card)?.[1];\n const image = imageRegex.exec(card)?.[1];\n \n if (name && name.length > 3 && name.length < 200) {\n const price = parseFloat(priceStr?.replace(/[^\\d.,]/g, '').replace(',', '.')) || 0;\n if (price > 0 && price < 1000) {\n products.push({\n name: name.substring(0, 200),\n price,\n source_url: link || `https://${source}.com`,\n image_url: image || null,\n source,\n source_product_id: `${source}_${Date.now()}_${Math.random().toString(36).substr(2, 6)}`,\n brand: '',\n category: 'parapharmacy',\n available: true,\n scraped_at: new Date().toISOString()\n });\n }\n }\n}\n\n// Deduplicate\nconst seen = new Set();\nconst unique = products.filter(p => {\n const key = `${source}:${p.name.toLowerCase()}`;\n if (seen.has(key)) return false;\n seen.add(key);\n return true;\n});\n\nreturn unique.slice(0, 5).map(p => ({ json: p }));"
|
||||
},
|
||||
"id": "extract",
|
||||
"name": "Extract Products",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [1100, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "http://parapharmacy-api:3002/api/products/bulk",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ JSON.stringify({ products: $input.all().map(i => i.json) }) }}",
|
||||
"options": {}
|
||||
},
|
||||
"id": "send-bulk",
|
||||
"name": "Send to API",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [1320, 300]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"respondWith": "json",
|
||||
"responseBody": "={{ { success: true, message: 'Scraping completed', products: $input.all().length } }}"
|
||||
},
|
||||
"id": "respond",
|
||||
"name": "Respond",
|
||||
"type": "n8n-nodes-base.respondToWebhook",
|
||||
"typeVersion": 1.1,
|
||||
"position": [1540, 300]
|
||||
}
|
||||
],
|
||||
"connections": {
|
||||
"Webhook": { "main": [[{ "node": "Parse Input", "type": "main", "index": 0 }]] },
|
||||
"Parse Input": { "main": [[{ "node": "Generate Tasks", "type": "main", "index": 0 }]] },
|
||||
"Generate Tasks": { "main": [[{ "node": "Scrape", "type": "main", "index": 0 }]] },
|
||||
"Scrape": { "main": [[{ "node": "Extract Products", "type": "main", "index": 0 }]] },
|
||||
"Extract Products": { "main": [[{ "node": "Send to API", "type": "main", "index": 0 }]] },
|
||||
"Send to API": { "main": [[{ "node": "Respond", "type": "main", "index": 0 }]] }
|
||||
},
|
||||
"active": true,
|
||||
"settings": { "executionOrder": "v1" },
|
||||
"tags": [{ "name": "parapharmacy" }, { "name": "scraper" }, { "name": "webhook" }]
|
||||
}
|
||||
Reference in New Issue
Block a user