Most organizations find that a blanket block is too aggressive because some AI bots power new search engines that might eventually drive discovery. A better strategy is to use data to identify the most aggressive training crawlers that offer no value and rate-limit or block those while allowing others.