# Site Intelligence

> Run bounded, robots-respecting website crawls distilled into typed company intelligence, and store contract-only site-audit plans.

Canonical guide: <https://engine.neraperla.com/docs/site-intelligence>

Raw Markdown: <https://engine.neraperla.com/docs/raw/site-intelligence.md>

OpenAPI: <https://engine.neraperla.com/v1/openapi.json>

Site Intelligence has two Preview surfaces: an executable crawl-and-extract run
that turns a company website into one typed intelligence report, and the
contract-only Site Audit Specification resource that freezes a future readiness
audit without performing any work.

## Run a site-intelligence crawl

Preview operation `create_site_intelligence_job_v1_site_intelligence_post`
accepts a target website plus crawl options and returns an asynchronous job.
The platform performs a bounded, robots-respecting crawl of the website's own
HTTPS origin — never leaving that origin, never executing JavaScript, and never
sending credentials or cookies — then a platform-managed model distills the
fetched pages into one structured report. Page content is treated as untrusted
evidence: instructions embedded in pages are ignored, and every populated
section cites the crawled `source_urls` that evidence it.

Request fields:

- `website_url` (required): the target site. The crawl is pinned to this URL's
  HTTPS origin; a `www.` variant of the same host is treated as the same site.
- `max_pages` (default 8, range 1-25): maximum HTML pages fetched, including
  the start page.
- `max_depth` (default 2, range 1-3): maximum link depth from the start page.
  Depth 1 fetches only the start page.
- `include_paths`: up to 20 path prefixes (each starting with `/`) crawled
  with priority before other discovered links.
- `exclude_paths`: up to 20 path prefixes that are never fetched.
- `focus_areas`: sections to prioritize. Allowed values are `overview`,
  `products`, `pricing`, `customers`, `technology`, `team`, `hiring`,
  `contact`, and `content`. Omitting the field (or null) means every section.

```bash
curl --fail-with-body --silent --show-error \
  --request POST "$BLACKPEARL_API_BASE_URL/v1/site-intelligence" \
  --header "Authorization: Bearer $BLACKPEARL_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Idempotency-Key: site-intel-acme-1" \
  --data '{
    "website_url": "https://acme.example",
    "max_pages": 8,
    "max_depth": 2,
    "include_paths": ["/pricing"],
    "focus_areas": ["overview", "products", "pricing"]
  }'
```

The request is accepted with HTTP 202 and a typed `site_intelligence` job.
Send `Idempotency-Key` so an ambiguous retry replays the original response for
24 hours.

### Poll the job

Poll `get_job_v1_jobs__job_id__get` until the job reaches a terminal state
(`succeeded`, `failed`, or `canceled`). The job reports the ordered stages
`prepare`, `crawl`, `extract`, and `finalize`, and its `result` stays null
until the run succeeds. Disconnecting the client does not stop the run; the
crawl and extraction continue and the result is retained for the standard
30-day job window, after which `GET /v1/jobs/{job_id}` returns HTTP 410 with
`job_expired`. Runs are bounded by server-side crawl and extraction deadlines;
a run that cannot fetch the start page fails with a stable error message
rather than fabricating a report.

```bash
curl --fail-with-body --silent --show-error \
  --request GET "$BLACKPEARL_API_BASE_URL/v1/jobs/$JOB_ID" \
  --header "Authorization: Bearer $BLACKPEARL_API_KEY"
```

### The typed report

A succeeded job's `result` contains:

- `crawl`: what actually happened — pages crawled and discovered, bytes
  received, depth reached, duration, and a bounded sample of skipped URLs with
  stable reasons (`robots_disallowed`, `excluded_path`, `page_limit_reached`,
  `depth_limit_reached`, `byte_limit_reached`, `fetch_failed`, `not_html`).
- `pages`: every fetched page with its URL, title, meta description, HTTP
  status, link depth, and visible word count.
- `intelligence`: the structured report — `company` (identity, locations,
  social links, published contact channels), `products`, `pricing` (model,
  plans, free tier and trial signals), `customers` (segments, named customers,
  case studies), `technology` (detected technologies and integrations),
  `team` (leadership and hiring signals), `content` (publishing activity), and
  `positioning` (value propositions, differentiators, competitors mentioned) —
  plus an overall `confidence` grade and explicit `limitations`. Scalar fields
  are null and collections empty when the site provides no evidence.

Job `usage.cost_usd` reports the amount the run was charged. Crawls honor
`robots.txt` disallow rules for the platform's identified crawler user agent,
and a site that blocks all crawlers fails rather than being fetched anyway.
Crawling a website does not prove ownership or permission; callers remain
responsible for targeting sites they are authorized to analyze.

## Store a contract-only site-audit plan

**Specification preview** — this surface accepts and stores typed plans; execution is not yet available. Storing an audit plan never audits a site; only the site-intelligence crawl above performs work.

Site Audit Specifications remain Preview, contract-only resources. They freeze
an authorized HTTPS target and allow-listed check profile for a future
readiness audit. The gateway does not resolve DNS, fetch pages, execute
scripts, scan infrastructure, or evaluate the site for these resources.

Preview operation `create_site_audit_specification_v1_site_audit_specifications_post` requires a caller key, target, check profile, expected policy version, and auditable reason. Target validation rejects credentials, fragments, unsupported schemes, unsafe addressing, and scope outside the declared origin.

```bash
curl --fail-with-body --silent --show-error \
  --request POST "$BLACKPEARL_API_BASE_URL/v1/site-audit-specifications" \
  --header "Authorization: Bearer $BLACKPEARL_API_KEY" \
  --header "Content-Type: application/json" \
  --data @site-audit-specification.json
```

Preview operation `list_site_audit_specifications_v1_site_audit_specifications_get` lists project/environment specifications with cursor pagination. Preview operation `get_site_audit_specification_v1_site_audit_specifications__specification_id__get` reads one.

Every specification response states that execution is unavailable and reports zero fetch, evaluation, and finding counts. A stored plan does not prove site ownership or permission; callers remain responsible for explicit authorization and a future executor must revalidate the target at execution time.
