Robots.txt
Indexing
By default, indexation is disabled. You can enable it from your app.config.ts:
export default defineAppConfig({
seo: {
indexable: true,
},
})
You can also use the SITE_INDEXABLE environment variable, which is handy when you want a staging
deployment to stay out of search results while production is indexed:
SITE_INDEXABLE=true
When both are set, app.config.ts wins. Leave seo.indexable out entirely to rely on the
environment variable alone.
When the site is not indexable, the generated robots.txt disallows everything. When it is, all
robots are allowed to crawl the blog and the sitemap.xml is advertised.
app.config.ts !!
It is what the sitemap.xml advertised here is built from.AI crawlers
Blocking AI crawlers is a dedicated option, since it is the most common thing people want to change here. See AI crawlers:
export default defineAppConfig({
seo: {
ai: {
allowCrawlers: false,
},
},
})
Advanced configuration
Bloggrify uses the @nuxtjs/robots
module to generate the robots.txt file. Anything the module supports can be set in the robots key
of your own nuxt.config.ts.
For example, to disallow Yandex from crawling your blog:
export default defineNuxtConfig({
extends: ['@bloggrify/core'],
robots: {
groups: [
{
userAgent: ['Yandex'],
disallow: ['/'],
},
],
},
})
Or to keep a section private:
export default defineNuxtConfig({
extends: ['@bloggrify/core'],
robots: {
disallow: ['/drafts'],
},
})
robots.txt is a request, not an access control mechanism. Well-behaved crawlers honour it, but
nothing stops the others. Do not use it to hide anything sensitive: a disallowed path is still
public, and listing it in robots.txt actually advertises it.Before version 2.0.0
By default, the robots.txt allows all robots to crawl the website. It's possible to modify the robots.txt file in the app.config.ts file.
For example if you don't want to block yandex:
robots: [
{
UserAgent: "Yandex",
Disallow: ["/"],
},
],
The default configuration allows all robots to crawl the website.