What it is and what it is for
Should your agent know what is on your website? A website source reads your public HTTPS pages. Each page becomes an internal article in the selected knowledge base. Public agents in the website chat and on the phone use this knowledge too. Crawling is a paid feature. Every run is charged in advance in blocks of 100 pages, scheduled runs included.
How it works
- Open Knowledge Base AI → Website sources and click Add source.
- On the General tab, enter a name, the Start URL (with https://) and the Knowledge base. Choose the Refresh interval, from every 6 hours to monthly, and Max. pages.
- On the Crawler tab, choose Auto, Fast or Full. Full uses a browser for pages that need JavaScript.
- On the Scope tab, limit paths or exclude areas if needed.
- Click Save. The first crawl starts a few minutes later.
- Check the result under Knowledge Base AI → Indexed pages.
Good to know
- All fetched pages count, including unchanged ones. A started block is not refunded.
- If your credit runs out, the run stops. Pages already indexed stay. Once you top up, the source runs on schedule again.
- HTTPS only, same host only. robots.txt is respected by default.
- Two runs with Crawl now must be at least 15 minutes apart.
- Add anything that is not on your website as a separate article.
- Knowledge Base AI is available from the Plus plan.