Regression? UTF-8 problem in 1.1.5, not present in 1.1.3

[sx1freak]

Hello,

with this script I can import vcards in an utf-8 file (with umlauts etc.) with PyS60 1.1.3pre (Series 60 1.2):

import re
import contacts
import codecs
import graphics
import e32
import appuifw

expremptyline = re.compile(r"\r\n\r\n", re.IGNORECASE & re.MULTILINE)
exprswitchtel = re.compile(r"TEL\;TYPE=", re.IGNORECASE & re.MULTILINE)

def exportforseries60(vcard):
 return exprswitchtel.sub(r"TEL;", vcard)

def readfile(filename):
 file = codecs.open(filename,"r","utf8")
 inhalt = file.read()
 file.close()
 return inhalt

def writetext(s,clear = True,x = 2, y = 12):
 if clear: draw.clear()
 draw.text((x,y),s,0x008000)

def carryon(event):
 global lockjc
 lockjc.signal()

def quit():
 global lockjc
 lockjc.signal() #so it won't be locked in any case
 appuifw.app.exit_key_handler=None #don't know if this is necessary
 appuifw.app.set_exit()
 
appuifw.app.title=u"Import VCards"
canvas=appuifw.Canvas(event_callback=carryon)
appuifw.app.body=canvas
draw=graphics.Draw(canvas)

lockjc=e32.Ao_lock()
appuifw.app.exit_key_handler=quit

vcardstrings = expremptyline.split(readfile(r"e:\python\addressbook.vcf")) #last line is an empty newline (ist Kontact schuld)
writetext(u"%s VCards aus Datei geöffnet"%(len(vcardstrings)-1))

contactdb = contacts.open()
allenamen = contactdb.keys()
laengeallenamen = len(allenamen)
writetext(u"Es sind %s Kontakte im Telefonbuch vorhanden."%(laengeallenamen),False,2,25)
lockjc.wait()
for i in range(laengeallenamen):
 writetext(u"Lösche Kontakt Nr.: %s/%s"%(i+1, laengeallenamen))
 e32.ao_yield()
 del contactdb[allenamen[i]] #Missing here is the compact method to get the db-file small again (see other post of mine)

#appuifw.note(u"Alle %s Kontakte gelöscht..."%(laengeallenamen),"info")
#contactdb.compact()
print "Alle gelöscht"

laengevcardstring = len(vcardstrings)
for g in range(laengevcardstring):
 writetext(u"Importiere VCard Nr.: %s/%s"%(g, laengevcardstring-1))
 contactdb.import_vcards(exportforseries60(vcardstrings[g]))
 e32.ao_yield()

allenamen = contactdb.keys()
laengeallenamen = len(allenamen)
writetext(u"Komprimiere Telefonbuch...",False,2,25)
#contactdb.compact()
contactdb.close()
appuifw.note(u"%s Kontakte importiert!"%(laengeallenamen),"info")

This is not possible anymore with 1.1.5 (with the exact same vcf-file), the vcard_import method fails with

Traceback (most recent call last):
  File "E:\System\Apps\Python\my\syncandrunclient.py", line 154, in run
    execfile(target, g)
  File "e:\python\vcardimportseries60.py", line 66, in ?
    contactdb.import_vcards(exportforseries60(vcardstrings[g]))
  File "E:\system\libs\contacts.py", line 157, in import_vcards
    return [self[x] for x in self._db.import_vcards(unicode(vcards),vcard_flags)]
UnicodeError: ASCII encoding error: ordinal not in range(128)

I think the relevant parts are the method where I read the file via codecs.open(f,"r","utf8") and the method to import the vcard in the database.
Is there any change between 1.1.3pre and 1.1.5 concerning the handling of unicode data?

Thanks,

Joost

[jplauril]

The default character set was changed back from 'latin1' back to 'ascii' between 1.1.3 and 1.1.5, but the reason for the breakage you see is actually a bug in contacts.py. Changing unicode(vcards) to vcards on contacts.py line 157 should fix this.

[sx1freak]

Hello,

sorry, but that doesn't fix the problem. The new error is:

Alle gelöscht
Traceback (most recent call last):
  File "E:\System\Apps\Python\my\syncandrunclient.py", line 154, in run
    execfile(target, g)
  File "e:\python\vcardimportseries60.py", line 65, in ?
    #contactdb.import_vcards(exportforseries60(vcardstrings[g]))
  File "E:\system\libs\contacts.py", line 157, in import_vcards
    return [self[x] for x in self._db.import_vcards(vcards,vcard_flags)]
UnicodeError: ASCII encoding error: ordinal not in range(128)

Hoping for your next suggestion...
Bye,
Joost

[sx1freak]

Hi,

I tend to think that this problem is related to the latin1 -> ascii switch, as I could import äöüß in 1.1.3, but no € sign. And as I understand it, äöüß are covered by latin1, but not the € sign, and none of them is covered by ascii. So it seems to me that I never imported in unicode, but always in latin1 (which perhaps does not work anymore due to the switch to ascii).

Bye,

Joost

[Nightscape]

I have the same problem when opening files with Non-Ascii characters like "茅", "胈" and so on. With unicode conversion i get the same error message, without all UI elements show only rectangles instead of letters.
I also tried converting the text file to unicode and opening them with different codec types from the codecs module with the same result.
Thanks a lot

[bpitk]

I don't have experience with vCard.
But ,when I read or write Thai-char in Unicode,I use this code.
It worked fine for me.

n=appuifw.query(u'Name of file E:\\','text',nm)
if n is not None:
f=codecs.open(("E:\\"+n),mode='w',encoding='utf_8')
f.write(t.get())
f.close()
appuifw.note(u"save file E:\\"+n,'info')

or

n=appuifw.query(u'Name of file E:\\','text',nm)
if n is not None:
f=codecs.open(("E:\\"+n),mode='r',encoding='utf_8')
t.clear()
t.add(f.read()+u'')
f.close()

or (from screen to file)

n=appuifw.query(u'Name of file E:\\','text',nm)
if n is not None:
f=open(("E:\\"+n),mode='w')
f.write(t.get().encode('utf8').replace("\xE2\x80\xA9","\x0D\x0A"))
f.close()
appuifw.note(u"save file E:\\"+n,'info')

May help...
You may try my simple Unicode Editor
http://www.isiam.info/dl/knotepad.zip

[korakotc]

Glad to know that there is another Thai python programmer here.

[sx1freak]

Cool, thanks for the code! I will try this, of course, sadly I'm quite busy with an exam right now.

Bye,

Joost