# HMM Shipping Tracker Scraper Guide

## Overview

The HMM scraper extracts ETA (Estimated Time of Arrival) data from HMM's Track & Trace website for containers.

**Container for testing:** `HDMU5599362`
**Expected ETA:** `2026-05-16 15:00` (from Discharging Port)

## Issues & Solutions

### Problem: HTTP/2 Connection Error

HMM's website uses HTTP/2 protocol with strict security measures that block:
- Direct axios requests (timeout)
- Standard Playwright connections (ERR_HTTP2_PROTOCOL_ERROR)

### Solutions (in order of preference)

#### Option 1: Using ScrapingBee Service ⭐ (Recommended)

ScrapingBee bypasses HTTP/2 and anti-bot protections:

1. **Get API key:**
   - Sign up at https://www.scrapingbee.com
   - Get your free API key

2. **Configure:**
   ```bash
   # Add to your .env file:
   SCRAPINGBEE_API_KEY=your_api_key_here
   ```

3. **Run the scraper:**
   ```bash
   npx tsx scripts/test-hmm-scraper.ts
   ```

#### Option 2: Manual HTML Extraction

If you can access the HMM website manually (through a VPN, proxy, or different network):

1. **Get the tracking page:**
   - Visit: https://www.hmm21.com/e-service/general/trackNTrace/TrackNTrace.do
   - Enter container number: `HDMU5599362`
   - Press "Retrieve" button
   - Wait for results

2. **Save the HTML:**
   - Press `F12` to open Developer Tools
   - Right-click on the page > "Inspect"
   - Right-click on the HTML element > "Copy" > "Copy element"
   - Save to file: `hmm-results.html`

3. **Extract ETA from saved HTML:**
   ```bash
   npx tsx scripts/demo-extract-from-html.ts hmm-results.html HMM
   ```

4. **Update database manually:**
   ```sql
   UPDATE container_shipment 
   SET eta = '2026-05-16 15:00'
   WHERE container_number = 'HDMU5599362';
   ```

#### Option 3: Using VPN/Proxy

If the issue is regional blocking:
1. Connect through a VPN in a different region
2. Then run: `npx tsx scripts/test-hmm-scraper.ts`

## Implementation Details

### Functions in `src/lib/shipping-scraper.ts`

#### `scrapHMM(containerNumber: string, trackingUrl?: string): Promise<string>`
- Main scraping function
- Returns: ETA string or "No Encontrado" or "Error"
- Tries: ScrapingBee → Playwright → Fallback

#### `extractHMMEta(html: string): string | null`
- Extracts ETA from HTML content
- Looks for "Discharging Port" in multiple formats
- Returns: Date string (YYYY-MM-DD HH:MM) or null

#### `extractETAFromHTML(html: string, carrier: string): {...}`
- **Exported function** for processing saved HTML
- Parameters:
  - `html`: Full HTML content
  - `carrier`: "HMM", "Hapag-Lloyd", or "CMA"
- Returns:
  ```typescript
  {
    eta: string | null,
    success: boolean,
    message: string
  }
  ```

### ETA Extraction Patterns

The scraper looks for dates in this priority order:

1. **"Discharging Port" + DateTime:**
   ```
   Discharging Port ...
   ... 2026-05-16 15:00
   ```

2. **"Discharging Port" + Date:**
   ```
   Discharging Port ... 2026-05-16
   ```

3. **Last DateTime in page:**
   - Fallback if no Discharging Port found
   - Format: YYYY-MM-DD HH:MM

## Testing Scripts

### Test 1: Extraction Logic (No Network)
```bash
npx tsx scripts/test-hmm-extraction.ts
```
✅ Fast test that validates regex patterns work

### Test 2: Full Scraper (Requires Network)
```bash
npx tsx scripts/test-hmm-scraper.ts
```
- First create container in database via dashboard
- Container must have:
  - Number: HDMU5599362
  - Carrier: HMM
  - Tracking URL: https://www.hmm21.com/e-service/general/trackNTrace/TrackNTrace.do

### Test 3: Extract from Saved HTML
```bash
npx tsx scripts/demo-extract-from-html.ts <html-file> HMM
```

### Debug 4: Connectivity Check
```bash
npx tsx scripts/debug-hmm-connectivity.ts
```
- Tests axios and Playwright connections
- Shows which methods are available

## Database Schema

The scraper updates these tables:

### `container_shipment`
- `id`: UUID
- `container_number`: string (e.g., "HDMU5599362")
- `carrier`: string (e.g., "HMM")
- `eta`: timestamp (e.g., "2026-05-16 15:00")
- `eta_raw`: string (raw extracted text)
- `vessel_tracking_url`: string

### `container_tracking_history`
- Records all ETA scrape attempts
- `scrape_status`: "success" | "not_found" | "error"
- `eta_raw_text`: Raw extracted text
- `eta_scraped`: Formatted ETA

## Troubleshooting

| Issue | Solution |
|-------|----------|
| "No Encontrado" | Results page didn't load or ETA not found in page |
| "Error" | Exception during scraping (network error, parse error) |
| HTTP/2 error | Configure SCRAPINGBEE_API_KEY or use manual HTML extraction |
| Playwright timeout | Server is slow - wait 30+ seconds or try different method |
| Input field not filled | Page structure changed - check HMM website |

## Future Improvements

- [ ] Add support for HMM API if they provide one
- [ ] Add proxy rotation for resilience  
- [ ] Add caching to reduce scrape attempts
- [ ] Monitor success rate and alert on failures
- [ ] Add container validation before scraping

## References

- HMM Track & Trace: https://www.hmm21.com/e-service/general/trackNTrace/TrackNTrace.do
- ScrapingBee: https://www.scrapingbee.com
- Container numbering standards: ISO 6346