Knowledge sources

Import and refresh a website or sitemap

Turn public help pages into searchable knowledge articles and recover from sync problems.

Use Website / sitemap in Knowledge to import public HTTPS content. A sitemap is the best starting point when you want a whole help center; a single article URL works for a smaller source.

Choose the crawl scope

Enter the URL, limit the allowed path to the help content you want, and decide whether to follow links. If the sitemap already lists all articles, leave link following off.

The page limit accepts 1–1,000 pages and defaults to 25. Set it high enough for the selected scope. A limit of 25 is not a way to publish the first 25 pages of a larger discovered site; narrow the scope if you only want a subset. Large sites may also need to be split into smaller sources because text and download limits apply separately.

The site’s robots rules must allow fetching, and pages must be available without login, cookies, or a JavaScript challenge.

Follow progress and refresh

Open Show sync and ingestion jobs to see progress. After a successful import, each usable page becomes an article with its own title and public link. Teammates can insert those links from the composer’s slash menu.

Choose manual, daily, or weekly resync, or use Resync now for an immediate refresh. Source health → Load page health shows page-level status.

Understand partial success

Partially successful means the crawl completed but some individual pages failed. Successful pages are usable, and previously usable content is kept for pages that failed to refresh.

Use Retry failed URLs when available to retry those pages without fetching the successful ones again. For an interrupted crawl, expired retry data, or failed initial setup, use a full resync or Replace / Configure resync instead.

Only a complete full refresh without failures removes pages that disappeared from the site. Failed or cancelled refreshes don’t wipe the last successful source.

Fix common failures

  • 404: the page is missing. Fix the link or remove it from the source sitemap.
  • Robots blocked: update the site’s crawl rules if you own it, or ask its owner to allow the public help pages.
  • 401, 403, or browser challenge: make robots.txt, the sitemap, and articles readable by a normal HTTPS request. A page working in your signed-in browser doesn’t prove a crawler can read it.
  • Scope or page limit: choose a smaller sitemap or narrower path, or increase the limit within the supported range.

If the first import failed before saving a usable sync configuration, re-enter its URL and options through Replace or Configure resync.

Browse all topics →

Still need a hand? Ask our team.