Most teams use a segmented approach. Allow crawlers that may drive discovery, and rate-limit or block aggressive training scrapers that consume bandwidth without business value.