Client-side translations

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Client-side translations

by Nore » Thu Aug 24, 2017 16:08

As a new feature in the 0.5 branch, we now have client-side translations! That means that you can have a mod on a server, with texts translated by each client according to their locale. Translations are documented here.

Following is a quick tutorial on how to translate your mod:
  • Add the following line at the beginning of your mod:
    Code: Select all
    local S = minetest.get_translator(minetest.get_current_modname())
  • For each string you want to translate, add a call to S around it, like this:
    Code: Select all
    S("string to translate")
  • For each language you want your mod to be translated in, create a file named "[yourmodname].[language].tr" in the "locale/" folder of your mod, with the following contents:
    Code: Select all
    # textdomain: [your mod name here]
    string1=translation1
    string2=translation2
    ...

Finally, if you want your mod to still be compatible with older versions of Minetest, you can add this at the beginning of your mod:

Code: Select all
if not minetest.translate then
   function minetest.translate(textdomain, str, ...)
      local arg = {n=select('#', ...), ...}
      return str:gsub("@(.)", function(matched)
         local c = string.byte(matched)
         if string.byte("1") <= c and c <= string.byte("9") then
            return arg[c - string.byte("0")]
         else
            return matched
         end
      end)
   end

   function core.get_translator(textdomain)
      return function(str, ...) return core.translate(textdomain or "", str, ...) end
   end
end
 

User avatar
ExeterDad
Member
 
Posts: 1714
Joined: Sun Jun 01, 2014 20:00
Location: New Hampshire U.S.A
In-game: ExeterDad

Re: Client-side translations

by ExeterDad » Thu Aug 24, 2017 18:18

This is awesome! We are so in need of translations. Uhhhhh and translators. :)
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Thu Aug 24, 2017 19:57

Wow, finally. Too bad it’s not the common gettext stuff but hey … :)

Will it work in custom functions like …

Code: Select all
mymod_translation('string to be translated')

… in the actual mod code and …

Code: Select all
mymod_translation = function (str)
    return S(str)
end

… being in the init file?

Edit:

If I understand the documentation correctly using …

Code: Select all
mymod_translate = function (str)
    return minetest.translate(minetest.get_current_modname(), str)
end

… would be possible to achieve what I described. Am I right?

Please say that is correct :)
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Thu Aug 24, 2017 20:55

If you never use arguments, yes, it will work. Besides, you could do the even simpler
Code: Select all
mymod_translate = minetest.get_translator(minetest.get_current_modname())
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Thu Aug 24, 2017 22:03

Nore wrote:If you never use arguments, yes, it will work. Besides, you could do the even simpler
Code: Select all
mymod_translate = minetest.get_translator(minetest.get_current_modname())

Good, good :)

I will use arguments, but I have them already implemented, all I need is a function that translates a simple string but doing other fancy stuff with it (like returning both the string itself and the translated string and stuff).

I have just written a (not error proof and absolutely not optimized) conversion script in Lua that takes a PO file and creates a TR file in the same directory :)

repo directory, raw
 

User avatar
Wuzzy
Member
 
Posts: 3166
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

My review

by Wuzzy » Thu Aug 24, 2017 22:56

Yes! I have been waiting for this for a very long time, now it's finally here. Thank you very much, you are my hero! :D
Now I have taken some time to test and review this.

Overall, it seems to work. I have found 2 issues with the documentation and a major issue with the translated strings themselves.

One crucial missing feature at the moment are tools for automatic string extraction and updating. These are extremely important, I definitely do not want to extract every string manually, let alone update them.

In the long run, the switch to Gettext should be done for various reasons.

TEST RESULTS
I have performed a couple of simple functional tests:

- Simple pure ASCII strings: PASSED
- Translated string with Unicode chars: PASSED
- Original string with lots of weird Unicode chars in it: PASSED
- String with 1 parameter in it: PASSED
- String with equals sign: PASSED
- String with at sign: PASSED
- String with newline: PASSED, but misleading documentation (see below)
- Nested translated strings: PASSED
- String with 9 parameters in it, but the translation had them in reverse order: PASSED
- Performing string operations on translated strings: FAILED (see below)


+ Testing mod source code



DOCUMENTATION
Documentation is mostly fine. I found 2 minor issues.

Issue 1:
Documentation made me think that for a newline I have literally type “@\n” into the text. I later realized that I must replace “\n” with a *real* newline character. This is pretty confusing and also makes the translation file look a bit messy.

Issue 2:
For instance, suppose we want to translate "@1 Wool" with "@1" being replaced by the translation of "Red".
We can do the following:

local S = minetest.get_translator()
S("@1 Wool", S("Red"))

This is a bad example and should be replaced. This example encourages a poor coding practise. It makes an invalid assumption about languages, namely that the word “Red” will always have the exact same translation in all contexts. This assumption already fails when the word “red” needs to be inflected. Here's are some translations into German:

red = rot
red carpet = roter Teppich
red wool = rote Wolle
red car = rotes Auto

Therefore, the safest way to translate item names is to simply always put the full name into minetest.translate, even if it means more typing for you.

I suggest to use a different example which uses e.g. a number, which is much more common.

Read https://www.gnu.org/software/gettext/ma ... ng-Strings for some good principles to follow. Although this does not use Gettext (yet), the principles in this document very much apply to this translation system as well.


OTHER COMMENTS
The file name suffix “.tr” is a poor choise IMO. This suffix is already used by Qt translation files and those have a very different format (XML-based).
What about “.mtf” (for “Minetest Translation File”)?

STRING OPERATIONS FAIL
I have made a troubling observation. Apparently you can no longer trust basic operations on strings returned by minetest.translate.

Playing around with luacmd:

/lua print(#"Hello")
5
/lua S = minetest.get_translator()
/lua b=S("Hello")
/lua print(b)
Hello
/lua print(#b)
9


In other words, the “#” operator returns the incorrect length for “Hello” after it went through the translator. WTF?

Looking further into this, I found why this is happening:
Code: Select all
S = minetest.get_translator("test_translate") -- I created a test mod with this name
VAR = S("Hello @1", S("World"))
VARTAB = {}
for i=1, #VAR do
        local char = string.sub(VAR, i, i)
        table.insert(VARTAB, char)
end     
REALVAR = table.concat(VARTAB, ":")

I had the feeling that the print operator is lying to me. So I want to look at each character. This code basically steps through each character in VAR and prints it out, so I can reveal the true string, but separated with colons.
Removing the colons then revealed the true string:
Code: Select all
(T@test_translate)Hello F(T@test_translate)WorldEEE


This is troubling. This proofs that the “#” is not useful on translated strings, and a lot of other string operations like string.sub, string.gsub, etc. apparently operate on the strange string above, instead of just “Hello World”.
This will probably mess up a lot of assumptions modders can usually make about strings in Lua. Scary.
Is this intentional? Are there other operations I cannot do on “translated” strings (e.g. strings returned by minetest.translate)?
Nothing of this is documented.


GETTEXT
Any yes, I agree that the switch to Gettext should be eventually made. There are many reasons, like a mature and WORKING toolchain for string extraction and updating (VERY important), a ton of 3rd party tools, translators are used to it, Minetest uses it for the engine, potential Weblate support, plural support, and, and, and …
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 05:23

Since the translations look differently on different clients, you can't expect anything on the strings. You need to treat them as black-box strings where the only things you can do on them is to use concatenation or things like minetest.colorize.
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 06:53

Sorry, but this is shit :(

So I'm forced to the extremely limited variables system instead of being able to use my flexible own variables system.

This should totally be fixed! At least the most common string functions should work.
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 07:02

There is no way functions such as string length can be made to work as you would expect them to. Consider a simple string such as "Hello", once translated. What is its length? Is it 5? Is is 6? Is it 7? Is it yet another value? It all depends on the language you translate that string in. However, this translation is done on the client (that's the whole point of client-side translations, mind you), so you can't get a result that would be consistent will all clients at once on the server.
Besides, suppose mod a translates this string in some manner, and mod b in another way. If the returned string was simply "Hello", how could the translation code on the client know which translation to use. Actually, how could it even know it is a translated string and not some untranslated string?
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 09:21

Then make it possible to use named variables at least and do not crash when variables and provided values do not match in amount.

Var @1 and @2 should never crash even if only @1 is provided. @2 then should simply be returned as a literal @2.

And allow @var to be used like this (am I the only one who thinks that named variables for translations are an absolute must-have?):

Code: Select all
S('Var: @var, and foo: @foo', {
    var = 'vaue of var',
    foo = 'value of foo'
})

And don't tell me this is impossible. I do something similar just by using gsub on a string. :)

Do it while it's still fresh and not widely used before it will be impossible to change because "a change would break something".
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 09:33

Var @1 and @2 should never crash even if only @1 is provided. @2 then should simply be returned as a literal @2.

It is deliberate to produce an error in that case. If you want a literal @2, use @@2 instead.

Concerning named variables, let me explain why it is much more problematic:
We need the message to appear untranslated when removing escape sequences. Thus, it means that the argument escapes are reconstructed *by the client*, without any more information (and that is why it is required to give the arguments in order). Hence, having named arguments would complicate this much more, since the client would somehow need to know the name (yes, it can be done, but I reckon it is quite a lot of complexity for not much added value). Besides, how do you know where the end of the argument name is and where the text begins? (ie, to give a very bad example, what would you do with "@1s"?)

If you feel this is so important, you are free to code it and submit a pull request, however.
 

User avatar
Wuzzy
Member
 
Posts: 3166
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: Client-side translations

by Wuzzy » Fri Aug 25, 2017 10:22

So I'm forced to the extremely limited variables system instead of being able to use my flexible own variables system.

What are your required use cases?

Since the translations look differently on different clients, you can't expect anything on the strings. You need to treat them as black-box strings where the only things you can do on them is to use concatenation or things like minetest.colorize.

This must totally be added to the documentation, because this is important. It will create so much confusion otherwise. Modders WILL expect to do normal operations. It should be explained in the docs how the system works (roughly) so they understand that you can't just do string.len, etc.

Actually, I don't really think this is serious because it is rarely a good idea to do serious string operations on translated strings. It's like saying you know better how to translate strings than the translator. I can't recall when I have ever needed something like this and I find it hard to find any compelling use case for this.

So, to summarize: The only legal things are minetest.colorize and concatenation. This seems fair enough. But this must go into the docs.

Then make it possible to use named variables at least

Sorry, I don't see the point and it seems way too complicated. I also don't see any benefit, as this is just a different syntax, but no actual new feature. The number system works just fine, it doesn't need “fixing”.

The only problem there might be is that it is currently limited to 9 variables. But a string with 10 or more variables is excessive and probably needs a rewrite/split anyway. I think this “problem” can be ignored.

(am I the only one who thinks that named variables for translations are an absolute must-have?):

It seems to be the case. :P

and do not crash when variables and provided values do not match in amount.

Oh, thanks for noticing! This is a real bug. Nore, you totally underestimated this bug!
I just tested this, here is what happens:
When a string gets translated with invalid parameters (e.g. “You have @1 apples and @2 bananas.=Sie haven @3 Äpfel und @5 Bananen.”), and a Minetest client attempts to show this faulty string, it crashes with a segfault. So there definitely should be some failsafe here, e.g. when the client encounters invalid parameters, fail gracefully by using an error string and pushing an error log message or whatever.
This bug is critical because if the server sends a faulty translation, then it could make all clients with the faulty language crash.
Last edited by Wuzzy on Fri Aug 25, 2017 10:34, edited 5 times in total.
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 10:27

Oh, I didn't notice that either. As for string concatenation: it is still useful, for formspecs for example (so that you can do a formspec with the text translated).
 

User avatar
Wuzzy
Member
 
Posts: 3166
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: Client-side translations

by Wuzzy » Fri Aug 25, 2017 10:35

Yeah, I took that back about string concatenation. You just need to be careful to not try to do the job of the translator.
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 11:08

Wuzzy wrote:What are your required use cases?

a: Sending "The value: @foo" and a table {foo = "the value"} in,and getting the translated "Der Wert: the value" out.

b: Sending "An Item made of +x from +y" in, and getting "Ein Gegenstand aus +x von +y" out being able to use result:gsub('*.', {['*x'] = 'whatever', ['+y'] = 'something'}) with the return value of the translation value.

c: Not having to write the following to get a translation of a string with a line break:

Code: Select all
Here is
a line break = Hier ist
ein Zeilenumbruch

But instead being able to simply use a literal \n (or @n as an alias).

Wuzzy wrote:Actually, I don't really think this is serious because it is rarely a good idea to do serious string operations on translated strings. It's like saying you know better how to translate strings than the translator.

As long as the translator has limited possibilities to actually translate the string the result has to be post-processed.

Wuzzy wrote:
Then make it possible to use named variables at least

Sorry, I don't see the point and it seems way too complicated. I also don't see any benefit, as this is just a different syntax, but no actual new feature. The number system works just fine, it doesn't need “fixing”.

As said, there is ABSOLUTELY NOTHING that is complicated in using properly named variables instead of generic non-semantic numbers.

Can I at least insert "@5@2@8@9@4@4@4@4" and get what I expect (having @N replaced with the corresponding value, multiple times, mixed)?

Wuzzy wrote:The only problem there might be is that it is currently limited to 9 variables. But a string with 10 or more variables is excessive and probably needs a rewrite/split anyway. I think this “problem” can be ignored.

I STRONGLY disagree with that. Since @ is special anyways, why only match for @[0-9] instead of matching for @[0-9]* or even @[0-9a-z]* to have named variables?
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 11:19

Linuxdirk wrote:
Wuzzy wrote:What are your required use cases?

a: Sending "The value: @foo" and a table {foo = "the value"} in,and getting the translated "Der Wert: the value" out.

You can do that with numbered escape sequences.

Linuxdirk wrote:b: Sending "An Item made of +x from +y" in, and getting "Ein Gegenstand aus +x von +y" out being able to use result:gsub('*.', {['*x'] = 'whatever', ['+y'] = 'something'}) with the return value of the translation value.

This as well, I don't see why you would want to use gsub.

Linuxdirk wrote:c: Not having to write the following to get a translation of a string with a line break:

Code: Select all
Here is
a line break = Hier ist
ein Zeilenumbruch

But instead being able to simply use a literal \n (or @n as an alias).


This could be done, however.

Linuxdirk wrote:
Wuzzy wrote:Actually, I don't really think this is serious because it is rarely a good idea to do serious string operations on translated strings. It's like saying you know better how to translate strings than the translator.

As long as the translator has limited possibilities to actually translate the string the result has to be post-processed.

Wuzzy wrote:
Then make it possible to use named variables at least

Sorry, I don't see the point and it seems way too complicated. I also don't see any benefit, as this is just a different syntax, but no actual new feature. The number system works just fine, it doesn't need “fixing”.

As said, there is ABSOLUTELY NOTHING that is complicated in using properly named variables instead of generic non-semantic numbers.

Can I at least insert "@5@2@8@9@4@4@4@4" and get what I expect (having @N replaced with the corresponding value, multiple times, mixed)?

Wuzzy wrote:The only problem there might be is that it is currently limited to 9 variables. But a string with 10 or more variables is excessive and probably needs a rewrite/split anyway. I think this “problem” can be ignored.

I STRONGLY disagree with that. Since @ is special anyways, why only match for @[0-9] instead of matching for @[0-9]* or even @[0-9a-z]* to have named variables?

I already explained in a previous post *why* this was not possible without making the code quite a bit more complex. If you aren't satisfied with the current version, as I said, please feel free to make a pull request.
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 11:26

Nore wrote:You can do that with numbered escape sequences.

Minimal working example please.

Nore wrote:I don't see why you would want to use gsub.

As a workaround for named variables so translators do not have to fiddle around with a bunch of non-semantic numbers.

Nore wrote:
Linuxdirk wrote:c: Not having to write the following to get a translation of a string with a line break:

Code: Select all
Here is
a line break = Hier ist
ein Zeilenumbruch

But instead being able to simply use a literal \n (or @n as an alias).

This could be done, however.

Yeah, then please do that. No-one in the right mind wants unescaped line breaks in a key-value store :)

Nore wrote:I already explained in a previous post *why* this was not possible without making the code quite a bit more complex.

Translations ARE complex. What's the problem with writing code that is "a bit more complex"? Why weren't named variables and stuff planned from the beginning on?
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 11:30

Linuxdirk wrote:
Nore wrote:You can do that with numbered escape sequences.

Minimal working example please.

Code: Select all
S("The value: @1", "the value")


Linuxdirk wrote:
Nore wrote:I already explained in a previous post *why* this was not possible without making the code quite a bit more complex.

Translations ARE complex. What'the problem with writing code that is "a bit more complex"? Why weren't named variables and stuff planned from the beginning on?


  • You seem to be the only one caring about named variables.
  • Feel free to code it, if it's so simple.
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 11:36

Nore wrote:
Code: Select all
S("The value: @1", "the value")

And how do translators know what @1 is?

Nore wrote:
  • You seem to be the only one caring about named variables.
  • Feel free to code it, if it's so simple.

As said before: a simple gsub can do that. There is nothing new to code at all. Simply use whats already there.

https://repl.it/KZdp/1
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 11:39

I'm still waiting for you to show me how these translations can be different for each client according to their locale. Your code doesn't do that.
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 12:14

Nore wrote:I'm still waiting for you to show me how these translations can be different for each client according to their locale. Your code doesn't do that.

It doesn't do that because it's already a step further. The translated strings are there. I's only about how to handle variables.

But I see. It's only another rushed project by a core dev who wants to implement something the dev likes no matter how handy it is in real-life usage and that needs a mod wrapping it into something useful.

I can deal with that. Just keep it unhandy, and limited, and non-standard like it is. Just please make it properly escape newlines and make it not crashing when there are less values provided than there variable are in the code and vice versa.
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 13:02

Linuxdirk wrote:But I see. It's only another rushed project by a core dev who wants to implement something the dev likes no matter how handy it is in real-life usage and that needs a mod wrapping it into something useful.

If you think that, then you haven't tried to understand at all how client-side translations can even work. Feel free to explain me *in detail* how you even want to do client-side translations over which string operations as you would explain them to. As for named variables: yes it can be done, no it won't be done unless you provide the code for it yourself, or enough different people (ie not just you) require it.
 

User avatar
rubenwardy
Moderator
 
Posts: 5452
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: Client-side translations

by rubenwardy » Fri Aug 25, 2017 13:49

You expect 100% dedication from everyone else, yet fail to contribute or even put 10% in yourself. You could be helpful like Fixer, but the complainer only complains I guess.
 

User avatar
Linuxdirk
Member
 
Posts: 1576
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Client-side translations

by Linuxdirk » Fri Aug 25, 2017 14:16

Nore wrote:As for named variables: yes it can be done, no it won't be done unless you provide the code for it yourself
rubenwardy wrote:You expect 100% dedication from everyone else, yet fail to contribute or even put 10% in yourself.

Here's a wrapping function that allows using named variables: https://repl.it/KZr4

It takes the string with named variables and a table with named entries and converts it so it can be used by S().
 

Nore
Developer
 
Posts: 492
Joined: Wed Nov 28, 2012 11:35
GitHub: Ekdohibs

Re: Client-side translations

by Nore » Fri Aug 25, 2017 14:21

Linuxdirk wrote:
Nore wrote:As for named variables: yes it can be done, no it won't be done unless you provide the code for it yourself
rubenwardy wrote:You expect 100% dedication from everyone else, yet fail to contribute or even put 10% in yourself.

Here's a wrapping function that allows using named variables: https://repl.it/KZr4

It takes the string with named variables and a table with named entries and converts it so it can be used by S().


This is not a solution: strings in the translation file still need to use the numerical arguments.
 

Next

Return to News



Who is online

Users browsing this forum: No registered users and 3 guests