About Sokjan

Sokjan is a focused web crawler used for private research, search experimentation, and language-model tooling. It operates from a self-hosted lab environment and is not a mass-indexing or advertising crawler.

User-Agent

Sokjan Crawler/1.0 (+https://sokjan.net/)

Crawl & Respect Policy

Sokjan obeys robots.txt, honours crawl directives, and is tuned for low request rates.

To block:

User-agent: Sokjan Crawler Disallow: /

To allow gently:

User-agent: Sokjan Crawler Crawl-delay: 10
Back to Home