The web for the model
Searching and reading public pages from inside your perimeter: engines, allow and block lists, content types, stop words — and what cannot be configured.
The task, the code and the knowledge base are what the team has. The web is what it does not have: a library’s documentation, an analysis of an error, a standard, someone else’s README. The “Web” integration gives the model two tools — search and read — and runs them not on our backend but in your connector, under your administrator’s control.
Read before installing. The model names the addresses, and they go to the chosen search engine and to the sites — into their logs. A page that was read goes into the model’s context, and a site may contain text aimed at the model rather than at a person; the allow and block lists narrow this risk but do not remove it. DuckDuckGo works through an unofficial endpoint and under load may answer with an “are you a robot” check — then switch the engine. The internal network is always closed, but the model will read everything the lists allow on the public internet.
Why through the connector
The environment the model works in has its own page loading. We have turned it off everywhere — both in “Ask” and in the development stages. The reason is simple: none of your lists applies on our backend, while on the connector every one of them does. Without the integration installed the model has no web at all; with it — exactly as much as the administrator allowed.
Tools
| Tool | What it does |
|---|---|
web_search | searches the chosen engine; returns titles, addresses and excerpts. Results from sites on the block list (or outside the allow list) are not shown, and the answer says how many were hidden |
web_fetch | reads a page by its address. HTML arrives as text with headings, lists, tables and link addresses; JSON — formatted; the page is returned in full, without truncation |
Both tools only read, so they are available wherever the model is given connector tools: in “Ask” and in deep research — both in a project with code and in a space. The development stages of tasks do not get connector tools, and they have no page loading of their own either. The model searches first, then reads a suitable result — as a person would.
Engines
| Engine | Key | What matters |
|---|---|---|
| DuckDuckGo (default) | not needed | works right after installation, but through an unofficial HTML endpoint: under load it answers with an “are you a robot” check, and the integration says so plainly rather than “nothing found” |
| SearXNG | not needed | your own metasearch inside the perimeter — stable and private; its settings.yml must have the json format enabled (search.formats), otherwise it answers 403, and the integration names this setting |
| Brave Search API | needed | a commercial API with a free tier; the key is stored in the connector and goes only to Brave, never to sites |
The engine is an ordinary configured connector host: it has its own address, its own paths and its own key. Sites are another matter.
Where the model may go
The rule for sites is called web, and it differs from all other connector rules: the model names the address, not you. So instead of an address you have lists.
Allow list. If it is filled in, the model goes only to these sites, and the block list does not apply. Block list applies when the allow list is empty: everything except what is listed. Both lists empty — the open public internet.
The entry format is the same for both, the same as for the search tools of the large providers:
| Entry | What it covers |
|---|---|
example.com | the site and all its subdomains (docs.example.com too) |
docs.example.com | only this subdomain — not the parent and not its siblings |
example.com/blog | everything under this path: /blog, /blog/post-1, … |
example.com/*/articles | * is exactly one path segment |
No scheme (https:// in an entry is an error the form will point out), no port, no * in
the domain itself (subdomains are included anyway), and ASCII only: a letter from another alphabet
that looks like a Latin one makes a different domain, and an entry with it is rejected, so that the list does not
promise what it does not do.
Content types. What may be handed to the model, by the response’s Content-Type header:
text/html, application/json, text/*. Empty — any. It is checked before the response
body is read.
Stop words. A search query or an address containing any of these words is rejected on the spot, without a single request going out; case does not matter.
What cannot be configured
The internal network is always closed. Before every request the connector resolves the site’s name and refuses if even one of the addresses is not public: loopback, private ranges, link-local (where cloud metadata lives), CGNAT. The connection is made to exactly the address that passed the check, so the name cannot “change its mind” between the check and the connection. This rule cannot be turned off by a list or a setting: the connector’s ordinary connections are allowed private networks — that is their point — but the web is not.
Every redirect is checked. The connector does not follow a 302 by itself:
the integration makes each hop a separate request, and a public site that sends
the reader to an address inside your network runs into the same refusal.
Credentials do not go to sites. A request under the web rule cannot carry any of your keys — even if the model asks. The Brave key goes only to Brave.
Only http and https. Anything else is not a web address.
Refusals reach the model with a reason, and the model tells the person; every call, including a rejected one, is in the connection’s log and in the platform’s accounting, as with any connector tool.
Setup
- Install the “Web” integration from the catalogue on the connector’s local page. Before installation you can see what it will get: the public internet within your lists, the engine addresses, a key only for Brave.
- Create a connection to a development project. The default engine is DuckDuckGo, and the engine addresses are already filled in; for SearXNG enter your instance’s address, for Brave — the key.
- Fill in the lists. A sensible starting point for a team is an allow list of the documentation of your technologies and standards; the open internet is for teams that need exactly that.
From the command line it is the same: connector install web, then
connector instance add web --config engine=duckduckgo --config deny=pastebin.com.
The integration needs connector version 0.7 or newer: the web rule and the public-address check appeared there. An old connector refuses to install the integration and names both versions.
What the integration does not do
- It does not go to the intranet or to localhost — neither by a list nor at the model’s request.
- It does not read PDFs or images: in this version — HTML, text and JSON.
- It does not cache pages or index sites in advance: what an answer needed is read, and that is what you are billed for.
- It does not replace your firewall. The lists are a boundary at the integration level; the network boundary remains yours, see “Isolation”.