# Scrape\-LE

Analyse robots\.txt content and report whether a path may be crawled\.

Canonical: https://claudelist.com/listings/mcp-reg-io-github-nolindnaidoo-scrape-le-deb0b0f9ad

Category: MCP Servers

Source: community · nolindnaidoo

Source checked: 2026-10-14

Website: https://www.claudemcp.org/mcp/registry/io-github-nolindnaidoo-scrape-le-deb0b0f9ade9.html

Publisher description: Analyse robots.txt content and report whether a path may be crawled.

## Install or set up

1. Review the publisher repository and npm package `scrape-le-mcp@2.4.0` before running it. The MCP Registry declares this package as a stdio server.
2. Install Node.js, then run the command below in your Claude Code project. Check the publisher README for any account or configuration requirements.
3. Restart Claude Code and use `/mcp` to confirm the connection and review its tools and permissions.

```sh
claude mcp add mcp-io-github-nolindnaidoo-scrape-le-deb0b0 -- npx -y scrape-le-mcp@2.4.0
```

Installation guide: https://github.com/nolindnaidoo/scrape-le