# The web for the model

> Searching and reading public pages from inside your perimeter: engines, allow and block lists, content types, stop words — and what cannot be configured.

The task, the code and the knowledge base are what the team has. The web is what it does not have:
a library’s documentation, an analysis of an error, a standard, someone else’s README. The “Web” integration
gives the model two tools — **search** and **read** — and runs them not on our
backend but in your connector, under your administrator’s control.

<Callout type="warn">
**Read before installing.** The model names the addresses, and they go to the chosen search
engine and to the sites — into their logs. A page that was read goes into the model’s context, and a site
may contain text aimed at the model rather than at a person; the allow and block lists
narrow this risk but do not remove it. DuckDuckGo works through an unofficial endpoint and
under load may answer with an “are you a robot” check — then switch the engine.
The internal network is always closed, but the model will read everything the lists allow on the public
internet.
</Callout>

## Why through the connector

The environment the model works in has its own page loading. We have **turned it off**
everywhere — both in “Ask” and in the development stages. The reason is simple: none of your lists
applies on our backend, while on the connector every one of them does. Without the integration installed
the model has no web at all; with it — exactly as much as the administrator allowed.

## Tools

| Tool | What it does |
|---|---|
| `web_search` | searches the chosen engine; returns titles, addresses and excerpts. Results from sites on the block list (or outside the allow list) are not shown, and the answer says how many were hidden |
| `web_fetch` | reads a page by its address. HTML arrives as text with headings, lists, tables and link addresses; JSON — formatted; the page is returned **in full**, without truncation |

Both tools only read, so they are available wherever the model is given connector
tools: in “Ask” and in deep research — both in a project with code and in a
[space](/en/docs/quickstart/space). The development stages of tasks do not get connector tools,
and they have no page loading of their own either. The model searches first, then
reads a suitable result — as a person would.

_Illustration: A question whose answer is only on the web: the model found the page with the search tool, read it and answered from it._

## Engines

| Engine | Key | What matters |
|---|---|---|
| **DuckDuckGo** (default) | not needed | works right after installation, but through an unofficial HTML endpoint: under load it answers with an “are you a robot” check, and the integration says so plainly rather than “nothing found” |
| **SearXNG** | not needed | your own metasearch inside the perimeter — stable and private; its `settings.yml` must have the `json` format enabled (`search.formats`), otherwise it answers 403, and the integration names this setting |
| **Brave Search API** | needed | a commercial API with a free tier; the key is stored in the connector and goes only to Brave, never to sites |

The engine is an ordinary configured connector host: it has its own address, its own paths and its own key.
Sites are another matter.

## Where the model may go

The rule for sites is called **web**, and it differs from all other connector rules:
the model names the address, not you. So instead of an address you have lists.

**Allow list.** If it is filled in, the model goes only to these sites, and the block list does not
apply. **Block list** applies when the allow list is empty: everything except what is listed.
Both lists empty — the open public internet.

The entry format is the same for both, the same as for the search tools of the large providers:

| Entry | What it covers |
|---|---|
| `example.com` | the site and all its subdomains (`docs.example.com` too) |
| `docs.example.com` | only this subdomain — not the parent and not its siblings |
| `example.com/blog` | everything under this path: `/blog`, `/blog/post-1`, … |
| `example.com/*/articles` | `*` is exactly one path segment |

No scheme (`https://` in an entry is an error the form will point out), no port, no `*` in
the domain itself (subdomains are included anyway), and ASCII only: a letter from another alphabet
that looks like a Latin one makes a different domain, and an entry with it is rejected, so that the list does not
promise what it does not do.

**Content types.** What may be handed to the model, by the response’s `Content-Type` header:
`text/html`, `application/json`, `text/*`. Empty — any. It is checked before the response
body is read.

**Stop words.** A search query or an address containing any of these words is rejected on the spot, without
a single request going out; case does not matter.

_Illustration: The setup form: the engine as a choice, lists comma-separated, content types and stop words._

## What cannot be configured

**The internal network is always closed.** Before every request the connector resolves the site’s name and
refuses if even one of the addresses is not public: loopback, private ranges,
link-local (where cloud metadata lives), CGNAT. The connection is made to exactly the
address that passed the check, so the name cannot “change its mind” between the check and
the connection. This rule cannot be turned off by a list or a setting: the connector’s ordinary connections
are allowed private networks — that is their point — but the web is not.

**Every redirect is checked.** The connector does not follow a `302` by itself:
the integration makes each hop a separate request, and a public site that sends
the reader to an address inside your network runs into the same refusal.

**Credentials do not go to sites.** A request under the web rule cannot carry any of
your keys — even if the model asks. The Brave key goes only to Brave.

**Only `http` and `https`.** Anything else is not a web address.

Refusals reach the model with a reason, and the model tells the person; every call, including
a rejected one, is in the connection’s log and in the platform’s accounting, as with any connector
tool.

## Setup

1. Install the “Web” integration from the [catalogue](/en/docs/connector/marketplace) on
   the connector’s local page. Before installation you can see what it will get: the public internet
   within your lists, the engine addresses, a key only for Brave.
2. Create a connection to a development project. The default engine is DuckDuckGo, and the engine addresses
   are already filled in; for SearXNG enter your instance’s address, for Brave — the key.
3. Fill in the lists. A sensible starting point for a team is an allow list of the documentation of
   your technologies and standards; the open internet is for teams that need exactly that.

From the command line it is the same: `connector install web`, then
`connector instance add web --config engine=duckduckgo --config deny=pastebin.com`.

<Callout type="info">
The integration needs connector version **0.7 or newer**: the web rule and the public-address check
appeared there. An old connector refuses to install the integration and names both
versions.
</Callout>

## What the integration does not do

- It does not go to the intranet or to localhost — neither by a list nor at the model’s request.
- It does not read PDFs or images: in this version — HTML, text and JSON.
- It does not cache pages or index sites in advance: what an answer needed is read,
  and that is what you are billed for.
- It does not replace your firewall. The lists are a boundary at the integration level; the network boundary
  remains yours, see [“Isolation”](/en/docs/connector/isolation).