Automatic language redirection causes Googlebot to index in wrong languages

[wearebandb]

We have implemented an automatic language detection in our store, which redirects the user to his preferred language (browser language) when he arrives at the store.

The problem is, that the Googlebot has English set as language, which causes that all pages get indexed only in English.

What's the best practice here?

[OliverSo]

Hi 👋,

to ensure Googlebot indexes all language versions of your site correctly, you should avoid redirects for bots, use hreflang tags, create language-specific sitemaps, use canonical tags, and have language-specific URLs.

Detect if the user-agent is a search engine bot and serve the default or requested language without redirecting. Implement hreflang tags to specify language versions of your pages:

<link rel="alternate" href="https://www.example.com/en" hreflang="en">
<link rel="alternate" href="https://www.example.com/fr" hreflang="fr">

Submit separate sitemaps for each language in Google Search Console. Use canonical tags to indicate the preferred version of a page to prevent duplicate content issues:

<link rel="canonical" href="https://www.example.com/en">

Ensure each language version has a unique URL (e.g., example.com/en, example.com/fr).

Example code for bot detection:

function isSearchEngineBot() {
    const bots = ['googlebot', 'bingbot', 'slurp', 'duckduckbot', 'baiduspider', 'yandexbot', 'sogou'];
    const userAgent = navigator.userAgent.toLowerCase();
    return bots.some(bot => userAgent.includes(bot));
}

if (!isSearchEngineBot()) {
    // Perform automatic language redirection
} else {
    // Serve the default or requested language version without redirection
}

By following these steps, you can ensure proper indexing of all language versions by search engines while providing a seamless experience for users.

I hope this helps!