Search results
63 packages found
A specification compliant robots.txt parser with wildcard (*) matching support.
Parser for robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Generate a robots.txt for Astro
A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Middleware for Express to add a robots.txt
A TypeScript-based project for easy serving of static files with support for live reloading, compression, and typed requests.
- TypeScript
- static file serving
- live reload
- compression
- typed requests
- HTTP server
- SSL
- cors
- express middleware
- proxy
- sitemap
- feeds
- robots.txt
- compression (gzip, deflate, brotli)
A proper robots.txt parser and combinator that works with eulalie
Vite plugin for generating robots.txt for production and development builds
It uses the user-agents.org xml file for detecting bots.
The grunt task for html-snapshots
Express middleware for generating your robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Deney all robots express middleware
An Eleventy plugin to generate a robots.txt file for your static site