arabic language support

[javasymbrew]

Hi All,
My question is related to J2ME. Its my understanding that the support for a language is dependent on the system (mobile). If the mobile(handset) does not support a language then the software running on the handset would not be able to support it. Now my question is does the software also need to support the language. Meaning i have a browser that i need to run in arabic language .. in addition to the support from the handset , does the browser itself has to do something to support the arabic language. Also I have noticed a strange thing that on Nokia- N85 where the mnicroedition.encoding is ISO-8859-1 (which according to the docs does not support Arabic) i can see the arabic fonts clearly but on other handset ZTE where the encoding is UTF-8 i see all boxes.. Any suggestions .. pointers ???
Thanks to all,
Aj

[grahamhughes]

OK... these are several questions.

First: Java always works in Unicode. "microedition.encoding" just tells how how certain methods work. For example:

byte[] bytes = "some text".getBytes();
String s = new String(bytes);

These two methods use the platform default encoding when converting between chars and bytes.

Not every device will have every character from the unicode set. For example, most handsets will not have Chinese characters (unless, fairly obviously, they're made for the Chinese market).

Some applications include their own font, to avoid differences between different devices. Games especially do this.

Some devices may have Arabic characters, but might lack support for correct rendering (right to left, and correctly joining characters).

An application that relies on certain user interface components needs the device to be set in the same language. For example, some softkey labels are provided by the device ("Options"). Otherwise, it is not necessary for the device to support the same language.

Hope that helps.

Graham.

[javasymbrew]

Thanks for the response..
"First: Java always works in Unicode. "microedition.encoding" just tells how how certain methods work"
Two questions: 1. What does the "microedition.encoding" tells us? is it the default char encoding used by the JVM or is it the encoding returned by the native OS to the JVM
2. Is there a way to find out all the encodings supported by a handset in the JVM..
The reason i ma asking this q is because on Nokia n-85 the encoding that is returned by the microedition.encoding is ISO-8859-1 (i suspect is the default one) but still the Nokia N-85 is able to handle the Arabic fonts while the other ZTE handset that i used returned the encoding type as UTF-8 and still the Arabic fonts are displayed as boxes there ...

Thanks,
Aj

[grahamhughes]

"Encodings" are only connected to how certain API methods convert between bytes and characters. They are not related to what characters the VM or host operating system can support.

"Encodings" have nothing to do with what languages the phone does or can support.

There is no way to find out whether or not a particular handset can display Arabic characters.

Graham.

[bhanuchandar.k]

Hi ,
I think you can do one thing take some of the Arabic character and craw that using g.drawstring() or drawchars() if it renders it then it is having the Arabic font . If it displays the box like characters or junk characters then it doesn't support the language. This is just to check the language support is there or not .

[javasymbrew]

"Encodings" are only connected to how certain API methods convert between bytes and characters. They are not related to what characters the VM or host operating system can support.

"Encodings" have nothing to do with what languages the phone does or can support.

There is no way to find out whether or not a particular handset can display Arabic characters.

Graham.

I guess the Windows comes with a font folder to map different fonts to encodings and same would be the case with other operating systems.. i was expecting something similar with either JVM or Symbian OS..

[javasymbrew]

"Encodings" are only connected to how certain API methods convert between bytes and characters. They are not related to what characters the VM or host operating system can support.

"Encodings" have nothing to do with what languages the phone does or can support.

There is no way to find out whether or not a particular handset can display Arabic characters.

Graham.

Well if what you are saying is right , how is that the same implementation of the api works different on different devices.. i expect the same behavior on each device when i use the api -> byte[] bytearr = somestr.getbytes(); String utfstr = new String(bytearr,"UTF-8"); My understanding is that unless the VM supports characters for a given language there is no way to display it unless you embed it in a picture.
Thanks,
Aj

[grahamhughes]

how is that the same implementation of the api works different on different devices.. i expect the same behavior on each device when i use the api ->

byte[] bytearr = somestr.getBytes();

String utfstr = new String(bytearr,"UTF-8");

That second line:

String utfstr = new String(bytearr,"UTF-8");

Will behave the same on all devices that support UTF-8 as an encoding. That should include all MIDP-2 devices. (UTF-8 was not a requirement for MIDP-1.)

The first line, however:

byte[] bytearr = somestr.getBytes();

This will behave differently on different devices. This is because no encoding has been specified, and so the platform default encoding will be used. This encoding can be read from:

System.getProperty("microedition.encoding");

Remember that this encoding is the default encoding. It is not necessarily the only encoding supported by the device. One device might support several different encodings for converting between characters and bytes.

My understanding is that unless the VM supports characters for a given language there is no way to display it unless you embed it in a picture.

Yes, it must support the characters. By that, I mean it must have an image in its font corresponding to those characters.

But: this is nothing to do with the encoding. A device can use ISO-8859-1 as its default encoding, and still be able to display Russian, Greek and Arabic.

Within Java, characters ("char") are 16 bit. They are Unicode characters. Fonts are Unicode fonts. (However, just because they are Unicode fonts does not mean they will contain every character. Typically, Chinese, Japanese and Korean characters are left out, to save space - except, obviously, on phones sold in those countries.) Just as in Windows, any character that is not present in the font will display as a small square (or similar, depending on the device).

So long as you are working with characters, there is no "encoding". All characters are Unicode.

Encodings only appear when you want to convert characters into bytes, or bytes into characters. When you do this, you are squeezing 16 bit data into 8 bit units (or the reverse). There are many different ways to represent characters in 8 bit units. The "encoding" tells the Java API which representation to use.

There are several different techniques for converting characters to bytes:

1. One byte per character. Each character is encoded as a single byte. Since a char is 16 bits (with 65536 different values), and a byte is 8 bits (with 256 values), not all Unicode characters can be converted. We must choose which 256 characters we are going to use. Any character in the String that cannot be encoded in the set we choose, will be lost (or will become "?" or something).

Some examples:

ISO-8859-1:

 !"#$%&'()*+,-./0123456789:;<=>?
@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_
`abcdefghijklmnopqrstuvwxyz{|}~
 ¡¢£¤¥¦§¨©ª«¬-­®¯°±²³´µ¶·¸¹º»¼½¾¿
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß
àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ

ISO-8859-2:

 !"#$%&'()*+,-./0123456789:;<=>?
@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_
`abcdefghijklmnopqrstuvwxyz{|}~
 Ą˘Ł¤ĽŚ§¨ŠŞŤŹ­-ŽŻ°ą˛ł´ľśˇ¸šşťź˝žż
ŔÁÂĂÄĹĆÇČÉĘËĚÍÎĎĐŃŇÓÔŐÖ×ŘŮÚŰÜÝŢß
ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙

ISO-8859-5:

 !"#$%&'()*+,-./0123456789:;<=>?
@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_
`abcdefghijklmnopqrstuvwxyz{|}~
ЁЂЃЄЅІЇЈЉЊЋЌ­-ЎЏАБВГДЕЖЗИЙКЛМНОПР
СТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопр
стуфхцчшщъыьэюя№ёђѓєѕіїјљњћќ§ўџ

A look-up table like these is used to convert the characters into bytes (or the bytes into characters, if we're converting the other way). Note that this table is just a piece of data in the Java API. It is not part of the font.

2. Two bytes per character. Since a char is 16 bits, we can just split it into two bytes. We can order the bytes most-significant first, or least-significant first (commonly known as "big-endian" or "little-endian"). This format represents all unicode characters. It is often hard to view on non-Unicode systems, giving a characteristic "double-spaced" look.

3. Variable bytes per character. Some encodings use a different number of bytes to encode different characters. Typically, ASCII characters (0 - 127) are encoded in a single byte, while other characters require two or three. UTF-8 is an example of this, and allows all 65536 characters to be encoded. Chinese and Japanese specific encodings also often follow this pattern (but without encoding all unicode characters). Examples are GB-2312 and BIG5.

The encoding is simply a choice of which of these schemes to use. Usually, chars are converted to bytes in order to exchange data with some other system (like a server), and the choice of encoding will depend on the whatever is at the other end of the connection.

The "platform default encoding" is just the encoding that will be used automatically if you don't specify one.

Fonts and encodings are completely unrelated.

Graham.