localizing in Russian without unicode

[Pocus]

Hi,

is it possible? I'm using D3D fonts by the way, but I would need true type font which has cyrillic chars AND are not unicode.

[dmantione]

Sure, you need a font which supports ISO 8859-5 encoding.

http://www.google.com/search?client=safari&rls=en-us&q=iso8859-5+truetype&ie=UTF-8&oe=UTF-8

... gives many results.

[Pocus]

ok, but if I fetch some TTF without unicode, with Cyrillic, how do I proceed? Does the localizing person has to type in cyrillic, or in western alphabet, and then Direct3D will display the right chars by mapping the western chars to the cyrillic chars?

i'm confused.

[dmantione]

ISO 8859-5 has the Cyrillic characters in [#160..#255], so there is no conflict with Western characters. Afaik Windows has separate functions for 8- and 16-bit characters. If you use the 8-bit functions, and the system locale is set to the right values, Windows should automatically return input and output in ISO 8859-5.

[savage]

Can I ask why you do not want to use Unicode?

There are some easy to use, drop in replacements of Delphi's components called TNT Controls which are free and support Unicode, if that is what is worrying you. I have used TNT Controls in 2 projects now and they work wonderfully.

[Clootie]

What do you mean by "I'm using D3D fonts by the way"? Can you describe it in more details (as there are a lot of realsations of font rendering in D3D)?

Standart cyrillic charset for Windows is win-1251. This is single byte charset with cyrillic chars encoded in upper (128-255) part of table. So, localizing person could type them in cyrillic, save text without formatting and this text can be pasted in Delphi IDE (but will look strange on your system).

[Pocus]

Can I ask why you do not want to use Unicode?

There are some easy to use, drop in replacements of Delphi's components called TNT Controls which are free and support Unicode, if that is what is worrying you. I have used TNT Controls in 2 projects now and they work wonderfully.

because I'm not using controls. All my text is manipulated with the string variable type and TStringList class, and is then rendered with D3D. I don't feel like adapting the whole app to use widestring variable type. Also I would need a unicode compliant TStringList.

[Pocus]

What do you mean by "I'm using D3D fonts by the way"? Can you describe it in more details (as there are a lot of realsations of font rendering in D3D)?



this one (made by you :) ):

//-----------------------------------------------------------------------------
// Name: DrawText()
// Desc: Draws 2D text
//-----------------------------------------------------------------------------
function CD3DFont.DrawText(x, y: Single; dwColor: DWORD;
strText: PChar; dwFlags: DWORD): HRESULT;



Standart cyrillic charset for Windows is win-1251. This is single byte charset with cyrillic chars encoded in upper (128-255) part of table. So, localizing person could type them in cyrillic, save text without formatting and this text can be pasted in Delphi IDE (but will look strange on your system).

Ok, so I get a true type font with charset 1251 and use it in my app, creating the 3DFont with:

CD3DFont.Create(strFontName: PChar; dwHeight, dwFlags: DWORD);

(your code too)

The localizing person type cyrillic, but save it in plain ANSI and not unicode.

And that's all?

A question by the way, XLS is not able to save cyrillic except in unicode, do you know a workaround? (My localization file is a XLS file, saved in CSV).

[dmantione]

And that's all?

A question by the way, XLS is not able to save cyrillic except in unicode, do you know a workaround? (My localization file is a XLS file, saved in CSV).

That is all.

Regarding Excel, is your Windows system configured for the right code page? (Open the Control Panel -> regional settings -> "Set default..".

[Clootie]

Yea, it looks good. But you will need to verify it with real font you've got, cos I'm unsure about behaviour in case if Windows system locale is set to charset different from cyrillic.

But actually looking at source code I see:
function CD3DFont.PaintAlphabet(hDC: Windows.HDC; bMeasureOnly: Boolean{$IFDEF SUPPORTS_DEFAULTPARAMS} = False{$ENDIF}): HRESULT;
...
// For each character, draw text on the DC and advance the current position
for c := #32 to #126 do
begin
...

So you'll have a problem here - at least you need to change #126 to #255.

Later you'll have couple of places with something like (that should be updated too):
if ((Ord(c)-32) < 0) or (Ord(c)-32 >= 128-32) then Continue;


So actually I would recommend you to render text using ID3DXFont from recent D3DX9 libraries with unicode strings.


As for XLS - if your file contains localization strings for different laguages (common western subset, cyrillic, arabic, etc.) - I don't thinks it's possible so save it correctly with one command. If it contains only single language when best route IMHO is to set system locale to match exposted language before saving file as CVS (then Excel should flatten file to single-byte char sequence if possible)[/pascal]

[Pocus]

I'm stuck in using Dx8 for now, I'm using Omega for Dx8. I would like to go to Asphyre, but I don't have quite the time to do the conversion.

I can change in the D3DFont.pas the occurences of 126 to 255, if this is the only problem I can encounter, its not a big deal.

For now my regional settings are set to my country, France.

Can I ask you where to find charset 1251 true types font? A search on the web returns various pages on how to "russify" an app, but its far from being clear which fonts are not unicode and contains the cyrillic chars in part 127+.

Sorry to ask that, this is not I don't want to do the search, but I just need 2 files, one arial and one smallfonts, but in 1251 format and it should be ok. If you have them under the hand, I would really appreciate a forward.

Thanks you all for your input.

[Clootie]

I'm not aware where to find pure cyrillic truetype fonts. For combined fonts look here: http://www.alanwood.net/unicode/fonts.html.
But I can send you cyrillic FON (raster font) from mine system if you wish (mail me directly).
Other way to experiment is to use combined font and set windows locale to russian.

[dmantione]

As far as I know most standard fonts shipped with Windows can do Cyrillic. You don't need to search for a specific Cyrillic truetype font, but one that contains Cyrillic glyphs and a codepage 1251 table.

[Clootie]

And that's the problem for him: if Windows have different locale and he tries to render text containing chars from upper 128 elements of single byte table - he will definitely get wrong output with standart fonts (even if one contains cyrillic glyths)...

[Pocus]

I will mail you, if raster fonts can be used instead of TTF, I don't really care :wink:

Thanks all for your patient answers.

[Clootie]

Another note -
In d3dfont.pas you can modify:
pFont := CreateFont(nHeight, 0, 0, 0, dwBold, dwItalic,
0, 0, DEFAULT_CHARSET, OUT_DEFAULT_PRECIS,
CLIP_DEFAULT_PRECIS, ANTIALIASED_QUALITY,
VARIABLE_PITCH, @m_strFontName);
And replace DEFAULT_CHARSET with RUSSIAN_CHARSET (or some other). This should allow you to use cyrillic glyths even then windows locale set to different one.
So can just update your windows copy with "extended fonts" via language control panel with combined unicode fonts and still draw single byte ASCII sequences.

[Pocus]

I got your font Clootie.

Made a serie of tests. Nothing work.


I tried with or without changing DEFAULT_CHARSET to RUSSIAN_CHARSET.

I tried with or without changing my local setting from French to Russian.

With Lucida Sans Unicode or with your font, either I get my text in western character (my strings are really western characters but I thought they would be remaped to their equivalent index in the cyrillic charset), or I do not get any text.


-----------------

Just to be sure I understand the logic: a russian person localizing text can save cyrillic chars in ANSI format (using a 1251 charset). This appears as Cyrillic to him, but for an english windows, it will be displayed as western letters.

This work like this: if the english person uses a 1251 charset table, a mapping is done, for example character 230:

"f" => "Ф", # xE6 230 x424 1060 xD0A4 CYRILLIC CAPITAL LETTER EF

is displayed in the text file (english windows) as a "f". it is displayed as a "Ф" in russian windows.

it is supposed to be shown as "Ф", index 230, with an 1251 charset table.

Well, I'm right?

It don't work for now though :(

[dmantione]

First, compare

http://en.wikipedia.org/wiki/Code_page_1252

... and ...

http://en.wikipedia.org/wiki/Code_page_1251

The Ф can be found at position D6, not E6 in codepage 1251. At position D6 in codepage 1252, you will see an ?ñ. So, the Ф will appear in a machine configured for a western font as ?ñ, not f, there exists no automatic transliteration.

[Clootie]

dmantione is 100% right - there will be no automatic transliteration.

Example of text file you'll probably get from translator: http://www.frei.ru/golos/chat/100798.txt (this seems to be a log from a chat program)

[EAST]

For Enigma we did translations into 6 languages including Russian, Polish and Hungarian.

In all cases we rasterized a cyrillic font set mapped to a Russian (or Polish, Hungarian) keyboard.

All text for the game was in a localization file and indexed. We used native language speakers (from fansites no less) to localize line by line. It was largely voluntary, some cost, but much better results than the localization company did with the original translations.

We had 10000+ lines of text and it worked well.

[Pocus]

dmantione is 100% right - there will be no automatic transliteration.

Example of text file you'll probably get from translator: http://www.frei.ru/golos/chat/100798.txt (this seems to be a log from a chat program)

understood, this is transliterated using the position of the chars, so char 235 will remains char 235, on a single byte charset. This is was EAST has used if I understand well.

[Pocus]

UPDATE

I don't know what happened, perhaps I forgot to rebuild (but now it works with just a standard compiling!).

If I use RUSSIAN_CHARSET to create a font, the upper chars that I use (?©?†?® in french) are in Cyrillic!

Basically it seems like I have just to slightly modify InitDeviceObjects in D3DFont to have the possibility to display Cyrillic. I will explore that, but if this really work like it seems, ITS PRETTY DARN COOL :D

Thanks all for your lengthy explainations and patience...