Site Intelligence
Run bounded, robots-respecting website crawls distilled into typed company intelligence, and store contract-only site-audit plans.
Site Intelligence has two Preview surfaces: an executable crawl-and-extract run that turns a company website into one typed intelligence report, and the contract-only Site Audit Specification resource that freezes a future readiness audit without performing any work.
Run a site-intelligence crawl#
Preview operation create_site_intelligence_job_v1_site_intelligence_post
accepts a target website plus crawl options and returns an asynchronous job.
The platform performs a bounded, robots-respecting crawl of the website's own
HTTPS origin — never leaving that origin, never executing JavaScript, and never
sending credentials or cookies — then a platform-managed model distills the
fetched pages into one structured report. Page content is treated as untrusted
evidence: instructions embedded in pages are ignored, and every populated
section cites the crawled source_urls that evidence it.
Request fields:
website_url(required): the target site. The crawl is pinned to this URL's HTTPS origin; awww.variant of the same host is treated as the same site.max_pages(default 8, range 1-25): maximum HTML pages fetched, including the start page.max_depth(default 2, range 1-3): maximum link depth from the start page. Depth 1 fetches only the start page.include_paths: up to 20 path prefixes (each starting with/) crawled with priority before other discovered links.exclude_paths: up to 20 path prefixes that are never fetched.focus_areas: sections to prioritize. Allowed values areoverview,products,pricing,customers,technology,team,hiring,contact, andcontent. Omitting the field (or null) means every section.
curl --fail-with-body --silent --show-error \
--request POST "$BLACKPEARL_API_BASE_URL/v1/site-intelligence" \
--header "Authorization: Bearer $BLACKPEARL_API_KEY" \
--header "Content-Type: application/json" \
--header "Idempotency-Key: site-intel-acme-1" \
--data '{
"website_url": "https://acme.example",
"max_pages": 8,
"max_depth": 2,
"include_paths": ["/pricing"],
"focus_areas": ["overview", "products", "pricing"]
}'The request is accepted with HTTP 202 and a typed site_intelligence job.
Send Idempotency-Key so an ambiguous retry replays the original response for
24 hours.
Poll the job#
Poll get_job_v1_jobs__job_id__get until the job reaches a terminal state
(succeeded, failed, or canceled). The job reports the ordered stages
prepare, crawl, extract, and finalize, and its result stays null
until the run succeeds. Disconnecting the client does not stop the run; the
crawl and extraction continue and the result is retained for the standard
30-day job window, after which GET /v1/jobs/{job_id} returns HTTP 410 with
job_expired. Runs are bounded by server-side crawl and extraction deadlines;
a run that cannot fetch the start page fails with a stable error message
rather than fabricating a report.
curl --fail-with-body --silent --show-error \
--request GET "$BLACKPEARL_API_BASE_URL/v1/jobs/$JOB_ID" \
--header "Authorization: Bearer $BLACKPEARL_API_KEY"The typed report#
A succeeded job's result contains:
crawl: what actually happened — pages crawled and discovered, bytes received, depth reached, duration, and a bounded sample of skipped URLs with stable reasons (robots_disallowed,excluded_path,page_limit_reached,depth_limit_reached,byte_limit_reached,fetch_failed,not_html).pages: every fetched page with its URL, title, meta description, HTTP status, link depth, and visible word count.intelligence: the structured report —company(identity, locations, social links, published contact channels),products,pricing(model, plans, free tier and trial signals),customers(segments, named customers, case studies),technology(detected technologies and integrations),team(leadership and hiring signals),content(publishing activity), andpositioning(value propositions, differentiators, competitors mentioned) — plus an overallconfidencegrade and explicitlimitations. Scalar fields are null and collections empty when the site provides no evidence.
Job usage.cost_usd reports the amount the run was charged. Crawls honor
robots.txt disallow rules for the platform's identified crawler user agent,
and a site that blocks all crawlers fails rather than being fetched anyway.
Crawling a website does not prove ownership or permission; callers remain
responsible for targeting sites they are authorized to analyze.
Store a contract-only site-audit plan#
Specification preview — this surface accepts and stores typed plans; execution is not yet available. Storing an audit plan never audits a site; only the site-intelligence crawl above performs work.
Site Audit Specifications remain Preview, contract-only resources. They freeze an authorized HTTPS target and allow-listed check profile for a future readiness audit. The gateway does not resolve DNS, fetch pages, execute scripts, scan infrastructure, or evaluate the site for these resources.
Preview operation create_site_audit_specification_v1_site_audit_specifications_post requires a caller key, target, check profile, expected policy version, and auditable reason. Target validation rejects credentials, fragments, unsupported schemes, unsafe addressing, and scope outside the declared origin.
curl --fail-with-body --silent --show-error \
--request POST "$BLACKPEARL_API_BASE_URL/v1/site-audit-specifications" \
--header "Authorization: Bearer $BLACKPEARL_API_KEY" \
--header "Content-Type: application/json" \
--data @site-audit-specification.jsonPreview operation list_site_audit_specifications_v1_site_audit_specifications_get lists project/environment specifications with cursor pagination. Preview operation get_site_audit_specification_v1_site_audit_specifications__specification_id__get reads one.
Every specification response states that execution is unavailable and reports zero fetch, evaluation, and finding counts. A stored plan does not prove site ownership or permission; callers remain responsible for explicit authorization and a future executor must revalidate the target at execution time.