About JoinPostlyBot, our website crawler
JoinPostlyBot is the crawler behind the Site Audit in Join Postly Rank, our SEO toolkit. It identifies itself with this user agent:
JoinPostlyBot/1.0 (+https://joinpostly.com/bot)Why it visits
A Join Postly user started a Site Audit of a website, usually their own. The bot reads that site's pages to check for SEO problems such as broken links, missing titles or slow redirects.
It follows robots.txt
Before crawling, JoinPostlyBot reads /robots.txt. It obeys the rules for “JoinPostlyBot”, or the “*” group when there is none, including Allow, Disallow and wildcards.
It's polite
One request at a time, at least one second apart. A Crawl-delay in robots.txt is respected (up to 10 seconds). Only one audit of a website runs at a time, and each audit stops at a fixed page limit.
What it reads
Only HTML pages, sitemaps and robots.txt, at most 2 MB per page. It doesn't run JavaScript, submit forms, log in or download files, and it never visits private network addresses.
How to block or slow it down
Add a group for JoinPostlyBot to the robots.txt file at the root of your site. To block it everywhere:
User-agent: JoinPostlyBot
Disallow: /To keep it out of one folder and ask for 5 seconds between requests:
User-agent: JoinPostlyBot
Disallow: /private/
Crawl-delay: 5JoinPostlyBot reads robots.txt again at least once a day, so changes apply to the next audit. If robots.txt answers with a server error, the bot doesn't crawl the site at all.
Questions or a problem with the bot?
Email info@joinpostly.com with your domain and the time of the visits and we'll look into it. You can also read our privacy policy.