Robots.txt Generator
A ready robots.txt with your rules for search engine crawlers and the sitemap address.
…
How to do it
- 1Add the text
Paste or type the text or amount here.
- 2See the result
The result appears right away as you type.
- 3Copy it
Copy the result and use it anywhere. Your text goes nowhere.
About this tool
Choose the crawler, list the paths you do not want crawled and any paths to allow, and add your sitemap address. Copy the result into a file named robots.txt in the root of your site. Remember that robots.txt is a request to crawlers, not a way to hide private pages.
Frequently asked questions
Where does robots.txt go?
In the root of your website, so it is available at yourdomain.com/robots.txt.
Does it hide pages from search?
It asks crawlers not to visit them, but pages can still appear if linked elsewhere. Use noindex or a password for private pages.
More tools you may need
Bijoy to UnicodeTurn text in the Bijoy (SutonnyMJ) font into Unicode Bangla.QR Code Generator (with Logo and Colours)QR codes for a link, WiFi, WhatsApp, phone number or contact card, in your own colours and logo.CV MakerFill in a form and get a clean CV as a PDF, in Bangla or English.Invoice MakerList your items and get an invoice PDF with the totals worked out for you.Age CalculatorFind your age in years, months and days from a birth date. Also the gap between two dates.Taka in WordsWrite any Taka amount in words, in Bangla and English.