> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alana.shopping/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Enrichment

> End-to-end guide for enriching product data through the Bronze → Silver → Gold pipeline, including Bright Data web scraping and SERP analysis.

## Overview

Data enrichment is the process of taking raw product records and progressively improving their quality through the pipeline stages. This guide covers the full enrichment flow — from initial import through Bronze, Silver normalization, and Gold scoring — plus advanced enrichment via Bright Data web scraping.

***

## Step 1 — Import (Bronze)

The first step is getting your data into the system. All imports land in the **Bronze** layer: raw, unmodified, and idempotent.

### Import methods

| Method                                                                  | Best for                                     |
| ----------------------------------------------------------------------- | -------------------------------------------- |
| CSV/Excel upload                                                        | Existing supplier spreadsheets               |
| URL import                                                              | Scraping product pages directly from the web |
| API (`POST /api/workspace/{workspaceId}/catalogs/{catalogId}/products`) | Programmatic feeds                           |
| MCP feed                                                                | Real-time catalog sync from external systems |

After import, products are visible in your catalog with `pipeline_stage: "bronze"`.

***

## Step 2 — Silver (Normalize)

Silver cleans and normalizes your raw data: standardizes casing, validates image URLs, detects duplicates, and maps fields to the Alana schema.

### Run Silver via UI

1. Open your catalog
2. Select the products you want to normalize (or use **Select All**)
3. Click **Batch Actions** → **Normalize**
4. A progress indicator shows normalized / total
5. When complete, review the results panel: fields mapped, duplicates found, broken URLs

### Run Silver via API

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://app.alana.shopping/api/workspace/WORKSPACE_ID/catalogs/CATALOG_ID/batch/silver" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"scope": "all"}'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch(
    `https://app.alana.shopping/api/workspace/${workspaceId}/catalogs/${catalogId}/batch/silver`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ scope: 'all' }),
    }
  );
  const result = await response.json();
  console.log(`Normalized ${result.processed} products in ${result.duration_ms}ms`);
  ```

  ```python Python theme={null}
  import requests

  response = requests.post(
      f"https://app.alana.shopping/api/workspace/{workspace_id}/catalogs/{catalog_id}/batch/silver",
      headers={"Authorization": f"Bearer {api_key}"},
      json={"scope": "all"}
  )
  result = response.json()
  print(f"Normalized {result['processed']} products in {result['duration_ms']}ms")
  ```
</CodeGroup>

### Silver results

After Silver runs, each product record contains:

* `fieldsNormalized` — count of fields that were transformed
* `duplicateOf` — product ID if a duplicate was detected
* `urlsValidated` — count of image/media URLs checked
* `pipeline_stage: "silver"`

***

## Step 3 — Gold (Score & Analyze)

Gold produces an optimization score (0–100) and a gap list — the fields that, if filled, would most increase the score.

### Run Gold via UI

1. Select products in the catalog
2. Click **Batch Actions** → **Analyze**
3. A progress indicator shows analyzed / total
4. When complete, each product shows its score badge and gap highlights

### Run Gold via API

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://app.alana.shopping/api/workspace/WORKSPACE_ID/catalogs/CATALOG_ID/batch/gold" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"scope": "all"}'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch(
    `https://app.alana.shopping/api/workspace/${workspaceId}/catalogs/${catalogId}/batch/gold`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ scope: 'all' }),
    }
  );
  const { results, catalogSummary } = await response.json();
  console.log(`Average score: ${catalogSummary.avgScore}`);
  ```

  ```python Python theme={null}
  import requests

  response = requests.post(
      f"https://app.alana.shopping/api/workspace/{workspace_id}/catalogs/{catalog_id}/batch/gold",
      headers={"Authorization": f"Bearer {api_key}"},
      json={"scope": "all"}
  )
  data = response.json()
  print(f"Average score: {data['catalogSummary']['avgScore']}")
  ```
</CodeGroup>

***

## Step 4 — Review scores in Canvas

After Gold, open the **Canvas** to review and act on enrichment results:

1. Sort products by score (ascending) to find the lowest-quality items
2. For each product, the **Gaps** panel shows exactly which fields are missing
3. Use the inline editor to fill gaps directly in Canvas
4. Re-run Gold on edited products to update the score

***

## Step 5 — Bright Data enrichment

For products with incomplete data, Alana integrates with **Bright Data** to enrich via web scraping, SERP analysis, and dataset presets.

### URL import with web scraping

When you import via a product URL, Bright Data's Web Scraper extracts structured data directly from the product page.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://app.alana.shopping/api/workspace/WORKSPACE_ID/url-import" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/product/blue-running-shoes",
      "catalogId": "CATALOG_ID",
      "method": "web_scraper"
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch(
    `https://app.alana.shopping/api/workspace/${workspaceId}/url-import`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        url: 'https://example.com/product/blue-running-shoes',
        catalogId,
        method: 'web_scraper',
      }),
    }
  );
  const { jobId } = await response.json();
  ```

  ```python Python theme={null}
  import requests

  response = requests.post(
      f"https://app.alana.shopping/api/workspace/{workspace_id}/url-import",
      headers={"Authorization": f"Bearer {api_key}"},
      json={
          "url": "https://example.com/product/blue-running-shoes",
          "catalogId": catalog_id,
          "method": "web_scraper",
      }
  )
  job = response.json()
  print(f"Job ID: {job['jobId']}")
  ```
</CodeGroup>

### SERP analysis for SEO

Use Bright Data's SERP API to discover high-value keywords for your products. Results are used to optimize titles, descriptions, and meta fields during Gold scoring.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://app.alana.shopping/api/workspace/WORKSPACE_ID/catalogs/CATALOG_ID/enrich/serp" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "productIds": ["prod_123", "prod_456"],
      "locale": "en-US"
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch(
    `https://app.alana.shopping/api/workspace/${workspaceId}/catalogs/${catalogId}/enrich/serp`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        productIds: ['prod_123', 'prod_456'],
        locale: 'en-US',
      }),
    }
  );
  ```

  ```python Python theme={null}
  import requests

  response = requests.post(
      f"https://app.alana.shopping/api/workspace/{workspace_id}/catalogs/{catalog_id}/enrich/serp",
      headers={"Authorization": f"Bearer {api_key}"},
      json={
          "productIds": ["prod_123", "prod_456"],
          "locale": "en-US",
      }
  )
  ```
</CodeGroup>

### Dataset presets for competitive analysis

Bright Data dataset presets pull structured competitor data for categories like electronics, apparel, and home goods — useful for benchmarking your catalog against market leaders.

Available presets:

* `electronics_specs` — technical specifications from major retailers
* `apparel_sizing` — size charts and fit data
* `grocery_nutrition` — nutritional information and ingredients
* `home_goods_dimensions` — physical dimensions and materials

***

## Full enrichment flow

```mermaid theme={null}
graph LR
    A["Raw URL / File"] --> B["Bronze Ingest"]
    B --> C["Silver Normalize\n(Normalize button)"]
    C --> D["Gold Analyze\n(Analyze button)"]
    D --> E["Review Scores\nin Canvas"]
    E --> F{Gaps?}
    F -- yes --> G["Fill gaps\n(inline edit / Bright Data)"]
    G --> D
    F -- no --> H["Publish / Distribute"]
```

***

## Best practices

<AccordionGroup>
  <Accordion title="Always run Silver before Gold">
    Gold scoring relies on normalized data. Running Gold on raw Bronze data produces artificially low scores because fields like brand and category are not linked yet.
  </Accordion>

  <Accordion title="Use SERP enrichment for top-priority products">
    SERP analysis has a cost per query. Run it on your highest-traffic SKUs or new product launches, not bulk catalogs.
  </Accordion>

  <Accordion title="Prioritize gaps by score impact">
    The gaps list is ordered by impact. Filling the first gap will increase the score more than filling the last. Focus on the top 2–3 gaps per product.
  </Accordion>

  <Accordion title="Re-run Gold after every edit session">
    Scores are not live — they reflect the last time Gold ran. Re-run Gold after filling gaps to get current scores.
  </Accordion>
</AccordionGroup>