# How to build a chatbot from your website URL

URL: https://instantknowledgebot.com/guides/build-chatbot-from-website-url  
Last updated: 2026-10-21

> To build a chatbot from your website URL, enter your site's address in InstantKnowledgeBot and choose Single page, Whole site or Sitemap URL. The main text of each page is imported (menus, footers and cookie banners are left out), robots.txt is respected, and every page becomes a document you can review or remove. Imports take up to 10 pages on Free, 50 on Starter, 200 on Pro and 500 on Agency; paid plans can re-import the site weekly. Then test the bot and embed it with one line of code.

## Key takeaways

- Enter your website's address and pick Single page, Whole site or Sitemap URL; no copying, saving or uploading needed.
- Each import takes up to 10 pages on Free, 50 on Starter, 200 on Pro and 500 on Agency, within the 5 million characters each bot can hold.
- robots.txt is respected, and login, cart, account and search pages, duplicates and other websites are skipped.
- Every imported page is listed with its title and URL, so you can remove the ones the bot shouldn't use.
- Use Refresh from website after you change your site; paid plans (and the free trial) can re-import it automatically every week.

You can **build a chatbot from your website URL** in a few minutes: enter your site's address, choose how much of it to import, check the pages that came in, test the bot and embed it. Your existing pages (services, prices, policies, FAQ) become the bot's knowledge, and it answers visitors' questions from them in their own words.

## Step 1: Which import mode should you choose?

When you create a bot, step 2 (**Knowledge**) has a **Learn from a website** box. The same box is in every bot's **Knowledge** tab. Enter your address, then pick what to import:

| Mode | What it imports | Best for |
|---|---|---|
| **Whole site** | The page you enter, then the pages found by following links across the same site, then the pages in your sitemap | Most small business sites |
| **Sitemap URL** | Every page listed in the sitemap you enter (sitemap indexes are followed) | Larger sites with a clean sitemap |
| **Single page** | Exactly one URL; a PDF URL is imported as a PDF | Your FAQ, pricing or returns page |

`www.` and the bare domain count as the same site, and pages on other websites are never followed. A subdomain such as `blog.example.com` is a separate site, so import it on its own.

## Step 2: How many pages can you import?

Each import takes up to a set number of pages, depending on your plan:

| Plan | Pages per import | Weekly auto-refresh |
|---|---|---|
| Free | 10 | No (refresh by hand) |
| Starter | 50 | Yes |
| Pro (and the 14-day trial) | 200 | Yes |
| Agency | 500 | Yes |

All the knowledge in one bot, including uploaded files and pasted text, is capped at **5 million characters**. If your site is larger than your limit, use **Sitemap URL** with a sitemap of your most useful pages, or import the important sections one at a time as **Single page** imports. On a new account the 14-day Pro trial uses the Pro limit (200), including for the import you make while creating your first bot. Current limits are always on the [pricing page](/pricing).

## Step 3: What happens during the import?

The import runs in the background with a progress card showing pages added, fetched and skipped, and a **Cancel** button (cancelling discards the pages fetched so far). You can keep setting up the bot while it runs. Each bot runs one import at a time.

For each page, InstantKnowledgeBot:

- **checks robots.txt** first and skips anything it disallows (it identifies itself as `InstantKnowledgeBot` and honors `Crawl-delay`);
- **keeps the main text** and leaves out menus, headers, footers, sidebars, cookie banners and popups;
- **skips pages a bot shouldn't learn from** among the links and sitemap entries it finds: login, sign-up, cart, checkout, account, admin and search pages, and links with filter or sort parameters;
- **skips duplicates**, such as the same page with tracking parameters, or two pages with identical text;
- **imports linked PDFs** too, if you tick **Include linked PDFs** (up to 50 MB each; each counts as one page).

When it finishes you see how many pages were added and why the others were skipped.

## Step 4: How do you review the imported pages?

Every imported page becomes its own document in the **Knowledge** tab, named after the page title and tagged **web**, with a link to its URL. Go through the list and remove pages that would confuse the bot: old announcements, discontinued products, job ads, or anything with outdated prices. Fewer, current pages beat many stale ones. More tips: [preparing documents for an AI chatbot](/learn/preparing-documents-for-ai-chatbot).

You can add PDFs, Word files or pasted text to the same bot, for details that aren't on your website, such as an internal price list or a longer policy document.

## Step 5: How do you test and embed the bot?

Open the **Test** tab and ask the questions your customers ask, worded the way they would word them. Then try a question your site doesn't answer: the bot should say it doesn't have that information. Set the fallback message in **Settings**, for example your support email.

When you're happy, open **Embed & share** and paste the one-line snippet before `</body>`:

```html
<script src="https://instantknowledgebot.com/widget.js" data-bot="YOUR_BOT_ID" defer></script>
```

Platform guides: [WordPress](/guides/add-chatbot-to-wordpress), [Shopify](/guides/add-chatbot-to-shopify), [Wix](/guides/add-chatbot-to-wix), [Squarespace](/guides/add-chatbot-to-squarespace), [Webflow](/guides/add-chatbot-to-webflow) or [plain HTML](/guides/add-chatbot-to-html-website).

## Step 6: How do you keep the bot in sync with your site?

Each website you import gets a row in the **Knowledge** tab with its page count and last sync time:

- **Refresh from website** re-imports the site with the same mode and options, at your current plan's limit, and replaces that site's pages when it finishes. If the refresh fails or you cancel it, the old pages are kept.
- **Weekly** switches on an automatic re-import every week. It's included on Starter, Pro and Agency, and during the free trial; if the account moves to Free (for example when the trial ends), weekly refreshes stop.
- **Remove website** (the bin icon) deletes the website and all of its pages.

There's no real-time sync: after an important change, such as new prices, click **Refresh from website** rather than waiting for the weekly run. If you removed a single page from the list, a refresh brings it back, so remove it from your site or your sitemap too.

## What can't the website import read?

- **Pages that need JavaScript** to show their text. The import reads the HTML your server sends and doesn't run scripts, so these pages come back empty. Paste or upload that content instead.
- **Pages behind a login** or paywall, and anything robots.txt blocks.
- **Images, videos and charts.** Describe important visuals in text.
- **Other people's websites** you don't have the right to use. Import your own content.

## What does it cost?

The free plan includes one bot, 100 hosted AI messages a month and website imports of up to 10 pages; connect your own AI key for unlimited messages. Paid plans add more pages per import, weekly auto-refresh and more bots, and Pro and Agency add lead capture; see [pricing](/pricing). On a new account, your first bot also starts a 14-day Pro trial, no card needed, with 500 hosted AI messages. Only want to use your FAQ page? Follow [how to create a chatbot from your website FAQ](/guides/create-chatbot-from-website-faq).

## Frequently asked questions

**Can I build a chatbot from my website URL?**

Yes. When you create a bot (or later, in its Knowledge tab), enter your website's address and choose Single page, Whole site or Sitemap URL. InstantKnowledgeBot imports the main text of each page, suggests a bot name, greeting and starter questions from it, and you embed the finished bot with one line of code. You can add files or pasted text to the same bot.

**How many pages can it import from my website?**

Up to 10 pages per import on Free, 50 on Starter, 200 on Pro and 500 on Agency (the 14-day Pro trial uses the Pro limit, including for the import made while you create your first bot). Each linked PDF counts as one page. All knowledge in a bot, including uploaded files, is capped at 5 million characters, so an import stops early if the bot is full.

**Does it respect robots.txt, and which pages are skipped?**

Yes, robots.txt is respected in every mode, including single pages, and its Crawl-delay is honored; the importer identifies itself as InstantKnowledgeBot. Among the pages it finds through links or a sitemap, the import also skips login, sign-up, cart, checkout, account, admin and search pages, links with filter or sort parameters, images and other files, pages on other websites, duplicate pages, and pages over 2 MB. When the import finishes you see how many pages were skipped and why.

**How do I keep the bot updated when my website changes?**

Click Refresh from website on the site's row in the Knowledge tab. It re-imports the pages with the same settings and replaces the old ones. On paid plans, including the free trial, you can also switch on a weekly automatic refresh. There's no real-time sync, so refresh by hand after an important change.

**Can I import a sitemap or just one page?**

Both. Single page imports exactly one URL (a PDF URL is imported as a PDF). Sitemap URL imports the pages listed in your sitemap, including sitemap indexes. Whole site starts from the address you enter, follows your site's own links and then its sitemap, staying on the same domain.

**Can it read PDFs linked on my site?**

Yes. Tick Include linked PDFs and PDFs linked from your pages are imported too, up to 50 MB each. Scanned, image-only PDFs need a text layer (OCR) first.

**Can it read pages that need JavaScript or a login?**

No. The import reads the HTML your server sends, without running JavaScript or signing in. Pages that only show their text after scripts run come back empty; copy and paste that content or upload it as a file instead.