Django internationalization problems with fr-ca, es-us

Question

Working on internationalizing, mainly the urls right now, a site and everything seems to be working with the exception of Canadian French (fr-ca) and US Spanish (es-us).

I'm using Django 1.4

settings.py has django.middleware.locale.LocaleMiddleware as an installed middleware

my list of language are:

ugettext = lambda s: s
LANGUAGES = (
    ('en',    ugettext('English')),
    ('en-gb', ugettext('English United Kingdom')),
    ('es',    ugettext('Spanish')),
    ('es-us', ugettext('Spanish United States')),
    ('fr',    ugettext('French')),
    ('fr-ca', ugettext('French Canada')),
)  

urls.py:

from django.conf.urls.defaults import patterns, include
from django.conf.urls.i18n import i18n_patterns

urlpatterns = patterns('myapp',
    (r'^example/?$',            'main.views.example'),
    (r'^$',                     'main.views.index'),

    (r'',                       include('myapp.main.urls')),
)

urlpatterns += i18n_patterns('myapp.main.views',  
    (r'^example/?$',            'example'),
    (r'^example_1/?$',          'example1'),
    (r'^example_2/?$',          'example2'),
    (r'^$',                     'index'),
)

I've run

# django-admin.py makemessages -l en
# django-admin.py makemessages -l en-GB
# django-admin.py makemessages -l fr
# django-admin.py makemessages -l fr_CA
# django-admin.py makemessages -l es
# django-admin.py makemessages -l es_US
#
# django-admin.py compilemessages

This is what I get:

|  Accept-Language Header              | Response Language  | Expected Language  |
|--------------------------------------|--------------------|--------------------|
|  fr-ca;q=0.9, fr;q=0.8               |       fr           |       fr-ca        |
|  fr-ca;q=0.9                         |       fr           |       fr-ca        |
|                                      |                    |                    |
|  es-us;q=0.9, es-mx;q=0.8, es;q=0.7  |       es           |       es-us        |
|  es-us;q=0.9                         |       es           |       es-us        |
|                                      |                    |                    |
|  en-gb;q=0.8, en-us;q=0.7, en;q=0.6  |       en-gb        |       en-gb        |
|  en-us;q=0.9                         |       en           |       en           |
|                                      |                    |                    |
|  fr-ca;q=0.8, en-gb;q=0.7            |       fr           |       fr-ca        |
|  fr-ca;q=0.7, en-gb;q=0.8            |       en-gb        |       en-gb        |
|--------------------------------------|--------------------|--------------------|

If I place the locale codes in the URL, then I get the correct locale response:

http://localhost:8000/fr-ca/       I get fr-ca text back
http://localhost:8000/fr/          I get fr text back
http://localhost:8000/en/          I get en text back
http://localhost:8000/en-gb/       I get en-gb text back
http://localhost:8000/es-us/       I get es-us text back
http://localhost:8000/es/          I get es text back
http://localhost:8000/             I get en text back _(Default in settings.py)_

en-gb works as expected, and in the last request is returned instead of the higher precedence fr-ca.

Am I just missing something, or do fr-ca/es-us just not work with Django?

---

(** I also sent in the default django_language cookie, and everything works as expected, so it looks like I'm only experiencing the issue when using the Accept-Language header. **)

---

UPDATE - The short and sweet answer was already provided by @ilvar

Awright, I finally dug down into Django source (Since it's awesomely open source, and all)

The file [django/utils/translation/trans_real.py][1] is where I found, verified, the answer.

Here's a snippet of the function get_language_from_request, line 350:

...
for path in all_locale_paths():
    if os.path.exists(os.path.join(path, dirname, 'LC_MESSAGES', 'django.mo')):
        _accepted[normalized] = lang
        return lang  
...  

The path is set to the conf/locale directory of the Django install (in site-packages) and dirname is the locale name from the request, in my case fr-ca. So, if the Django install doesn't contain the locale from the request, which it doesn't have fr-ca, it will revert to one that is, fr.

Just above the snippet the language is checked for within the URL first, if you're using i18n routes, and then within the language cookie. If it exists in either then that language code is returned immediately, so it doesn't matter if the language doesn't exist within the Django install.

Handling it differently for 2/3 scenarios is what threw me off, but makes some sense since the browser could send anything, but if you've set a cookie|url then at that point it's your issue.

Thanks @ilvar for the answer, even though I questioned it ;)
All I did was just copy the fr to fr_CA, within Django's locales, and voila...

Answer 1

Django fully supports only locales which it is shipped with. But you can manually add necessary locales (copying them from es or fr, for example) in Django's library path.