# CMA CGM ETA Scraping - Issue Analysis & Solution

## Problem Summary

The CMA CGM ETA data is not appearing in the dashboard because:

1. **DataDome Anti-Bot Protection**: CMA CGM uses DataDome, an advanced anti-bot system that detects and blocks Playwright/Puppeteer automation
2. **ScrapingBee 403 Error**: The fallback service (ScrapingBee) is returning 403 Forbidden, indicating account/credit issues

## What We've Done

✅ Updated scraper to detect DataDome and use ScrapingBee as fallback
✅ Added comprehensive error handling and logging  
✅ Fixed API response to include `eta_raw` field in dashboard
✅ Improved carrier name matching for "cma-cgm" variants

## The Root Cause

When Playwright tries to access CMA's tracking page:
```
[CMA] 📄 Page content length: 1566 bytes  ← Too small, indicates error page
[CMA] ⚠️ DataDome/Anti-bot protection detected!
[CMA-ScrapingBee] ❌ 403 Forbidden - Check ScrapingBee account
```

## Solutions to Try

### Option 1: Verify Container Exists (Quickest)
1. Visit manually: https://www.cma-cgm.com/ebusiness/tracking/detail/CMAU0863618
2. If you see tracking data → container exists but needs special handling
3. If you see 404 or error → container number is wrong or not trackable

### Option 2: Fix ScrapingBee Account (Recommended)
1. Go to https://www.scrapingbee.com/dashboard/
2. Check account status and credits
3. If low on credits → add more or upgrade plan
4. Restart the scraper once credits are restored

### Option 3: Use Alternative Carrier Method
If CMA remains blocked:
1. Try extracting ETA from CMA API directly (if available)
2. Use paid proxy service specifically designed for CMA
3. Implement manual tracking URL submission by admin

## Testing the Fix

Run this to diagnose the issue:
```bash
npx tsx scripts/diagnose-cma.ts
```

Then test the scraper:
```bash
npx tsx scripts/test-cma-scraper.ts
```

## Next Steps

1. **Check ScrapingBee credits** - This is likely the issue
2. **Verify container exists manually** - Make sure CMAU0863618 is valid
3. **Once fixed, run**: The automatic ETA scraper will update the database
4. **ETA will appear** in the dashboard within 5 minutes

## Technical Details

The scraper now uses this priority:
1. **Playwright direct access** (fast, free) → Blocked by DataDome
2. **ScrapingBee fallback** (paid, handles anti-bot) → 403 error
3. **Manual entry** (last resort) → Requires admin intervention

## Monitoring

Check the logs from the scraper:
```
[CMA] ✅ Extracted ETA from JSON: 24-APR-2026
[CMA] Updated container CMAU0863618: success
```

The ETA will be stored in the database as:
- `container_shipment.eta` → Formatted date (2026-04-24)
- `container_shipment.eta_raw` → Raw value (24-APR-2026)

Both will appear in the dashboard's "ETA ACTUALIZADA" column.