What changed
AWS has introduced a managed web crawler for Amazon Bedrock Knowledge Bases, enabling users to prepare and ingest web content into their knowledge bases without building or hosting their own crawler. The Amazon Bedrock Web Crawler data source connector allows users to provide seed URLs or sitemaps, and it crawls within the scope and rate limits set by the user. Scheduled syncs keep the knowledge base current as pages change, and the connector can wait for content that loads with JavaScript.
What this means in practice
The Amazon Bedrock Web Crawler is the starting point for most public website content, as it respects robots.txt and captures only added, updated, and deleted content after the first full crawl. For more control over what reaches the knowledge base, users can use Crawl4AI, an open source Python crawler, to prepare content with structured Markdown, bounded extraction, and custom metadata. The prepared content is written to Amazon S3, which the knowledge base ingests through its Amazon S3 data source. This workflow allows users to validate retrieval quality and start ingestion into a managed knowledge base.
Limitations and availability
The workflow described is for batch ingestion of known web properties, not real-time web access at inference time. The Crawl4AI path works with both managed and customer-managed knowledge bases, as they both support Amazon S3 data sources. The workflow processes public pages only and does not implement authenticated sessions. For sites that require sign-in, users should evaluate Amazon Bedrock Web Crawler authentication options first, which include basic, form-based, and SAML authentication. Extending the sample with authentication requires session handling, credentials in AWS Secrets Manager, and a security review.
