[announce] s60pyrex alpha

[simo.salminen]

Pyrex lets you write code that mixes Python and C data types any way you want, and compiles it into a C extension for Python. For more info, see this page: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/.

The s60pyrex does not modify pyrex in any way - only the building process is bit different.

This package contains example that shows how to build pyrex extensions: http://pymbian.sourceforge.net/misc/...0.20051119.zip (pyrex 0.9.3 required).

Before building, you have to modify python-port-s60\symbian_python\pyconfig.h in your SDK directory. Find this line: #undef BAD_STATIC_FORWARD, and change it to #define.

Other notes:
This software uses dlls with static writable data. Officially this is not supported by Symbian, but it does work. See this link for more info: http://pymbian.sourceforge.net/misc/statichack/INFO

S60/Python also uses this technique. Because of this, it is important that the address spaces don't collide. S60/Python uses address 0x33000000 (this can be find out by running petran.exe python222.dll | grep "Data link addr"). The provided s60pyrex example uses address 0x30000000, so they won't collide (unless you make a extensions that uses 48 MB of RAM, which is not really possible). So, if you make extensions, you have to make sure that they each have non-colliding address spaces. Otherwise you will get KErrBadLibraryEntryPoint errors when loading them.



If that didn't make any sense, don't worry. I'll make new release with better doc in near future.

[korakotc]

Pyrex is probably the easiest choice for developing python extension.
I have been wishing for s60pyrex for a long time. Thanks again, Simo.

A little problem is that pyrex is sometimes difficult to setup (at least for me).
I tried 2-3 times till succeed for plain pyrex.
For s60pyrex, I'd better wait for the coming new release.

[simo.salminen]

Here is a new release of s60pyrex. Included is hopefully understandable documentation and c++ wrapping example: http://pymbian.sourceforge.net/misc/...0.20051206.zip

The README.txt:

= Overview =
s60pyrex is s60/python integration of the pyrex package.

This package has following goals:
 * compiling standard pyrex modules to .pyds loadable by s60/python
 * enabling easy symbian c++ api wrapping 

The s60pyrex does not itself contain any software. This package only
contains information in this README.txt and in the form of examples.

= Install =

Following packages need to be installed:
 * python2.3 for the PC (should work with 2.2/2.4)
 * pyrex 0.9.3 (or better)
 * s60 c++ sdk (2nd ed fp2 is good)
 * s60/python sdk (1.2 or better)

After installation, go to directory:
c:\Symbian\8.0a\S60_2nd_FP2\python-port-s60\symbian_python\ (assuming
2nd ed fp2 sdk), and modify file pyconfig.h. Change line "#undef
BAD_STATIC_FORWARD" to "#define BAD_STATIC_FORWARD".

After this, the examples shipped in s60pyrex should compile. Go to 
s60pyrex\examples\wrapping. Open the build_pyrex_ext.make file, and
edit the pyrexc.py path (default is c:\utils\python23\scripts\). Then
run the build commands.
> bldmake bldfiles 
> abld build wins udeb   # emulator build
> abld build armi urel   # phone build

After this you can test the extension with the 'test_wrap.py' script.
In emulator, copy the file to: "C:\Symbian\8.0a\S60_2nd_FP2\epoc32\
release\wins\udeb\z\system\APPS\python\". To test in phone, copy the
file "C:\Symbian\8.0a\S60_2nd_FP2\epoc32\release\armi\urel\uiutils.pyd" and
the file 'test_wrap.py' to the phone.

Explanation of different files in examples\wrapping\:

source files:
bld.inf : top-level build file. defines makefiles and target platforms
build_pyrex_ext.make : does pyrex compilation (.pyx->.cpp)
wrap.mmp : compiles and links sources to .pyd

test_wrap.py : python test script for uiutils
uiutils.pyx : uiutils pyrex module. wraps note() function
notewrap.h, notewrap.cpp : declaration and definition of note() function

dllentry.cpp : dll entry function. performs dll data initialisation.

movetophone.bat : s60/python filetransfer synchronization script

generated files:
ABLD.BAT : generated by 'bldmake bldfiles'
uiutils.cpp : generated after 'abld build' from build_pyrex_ext.make

= Constructing pyrex extension for s60/python = 

Pyrex can be used to wrap existing C code quite nicely. However, it
was not designed to wrap C++ code. Basicly what we need is a c'ish
adaption layer between pyrex and symbian c++. 

Here is an overview how things look:

|| layer name            || notes	           ||
-----------------------------------------------------
|| python code (.py)     || caller		   ||
|| pyrex (.pyx)          || pyrex		   ||
|| adaption layer (.cpp) || c'ish interface to c++ ||
|| symbian (.cpp/.lib)   || pure c++/symbian	   ||

Adaption layer does following:
 * convert from python objects to c++
 * convert return values from c++ to python objects
 * handle exceptions, and propagate them as python exceptions
 * does Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS calls when making 
   blocking request

Here is a good example of adaption layer from s60/python api doc
(this example is found in the examples\wrapping directory):

{{{
01 extern "C" PyObject * 
02 note(PyObject* /*self*/, PyObject *args) 
03 { 
04   TInt error = KErrNone; 
05   int l_tx, l_ty; 
06   char *b_tx, *b_ty; 
07 
08   if (!PyArg_ParseTuple(args, "u#s#", &b_tx, &l_tx, &b_ty, &l_ty))
09   return NULL; 
10 
11   TPtrC8 stype((TUint8*)b_ty, l_ty); 
12   TPtrC note_text((TUint16 *)b_tx, l_tx); 
13   CAknResourceNoteDialog* dlg = NULL; 
14 
15   if (stype.Compare(KErrorNoteType) == 0) 
16     dlg = new CAknErrorNote(ETrue); 
17   else if (stype.Compare(KInfoNoteType) == 0) 
18     dlg = new CAknInformationNote(ETrue); 
19   else if (stype.Compare(KConfNoteType) == 0) 
20     dlg = new CAknConfirmationNote(ETrue); 
21   else { 
22     PyErr_BadArgument(); 
23     return NULL; 
24   } 
25 
26   if (dlg == NULL) 
27     return PyErr_NoMemory(); 
28 
29   Py_BEGIN_ALLOW_THREADS 
30   TRAP(error, dlg->ExecuteLD(note_text)); 
31   Py_END_ALLOW_THREADS 
32 
33   if (error != KErrNone) 
34     return SPyErr_SetFromSymbianOSErr(error); 
35   else { 
36     Py_INCREF(Py_None); 
37     return Py_None; 
38   } 
39 }
}}}

Notice how the interface is c-like, no c++ language features used.

Notice also following:
 * Conversion from Python data types, usage of built-in data types at 
extension interface, usage of Unicode strings (lines 8-12) 
 * Maintenance of the reference counts (line 36) 
 * Passing of exceptions between C/C++ code and Python (line 34)
 * Releasing the interpreter lock while performing a blocking call to a 
service outside the interpreter (lines 29, 31) 

At the Pyrex layer, things look following:
{{{
cdef extern from "notewrap.h":
    cdef object note(object args)

def showinfo(message):
    note((message, 'info'))
}}}

And the python code:
{{{
import uiutils
uiutils.showinfo(u"hello")
}}}


= Static writable data on dll's =

Pyrex uses static writable data in the generated C source
files. Officially Symbian does not support writable static data in
dll's nor in applications, only in .exe's.

This is actually not true. The instructions how to enable writable
static data in dll's can be found here
(http://pymbian.sourceforge.net/misc/statichack/INFO). Static writable
data on applications can be achieved by using ECompXL.

The problem with the "dll statichack" is that one has to allocate and
administrate a memory address for the writable data. The address space
is allocated from pool of 256 slots. Each slot is minumum 1 MB. The
address must be a multiple of 1 MB. Basicly what this limitation means
that different dll's in same process must not use conflicting
addresses for writable static data.

The "EPOCDATALINKADDRESS" in .mmp file defines where dll writable static
data is relocated at dll load time. See the examples for the usage.

In summary, the requirements for the address are following:
 - the address must be a multiple of 1 MB and must be between 0x30000000 and
   0x3ff0000, inclusive
 - data of two DLLs used by one programme cannot overlap

Note that python222.dll uses 0x33000000. Run following command to be sure:
{{{
petran.exe <phonebinary>\python222.dll | grep "Data link addr"
}}}
where <phonebinary> is where you have copied the .dll from the phone. The
python222.dll currently uses less than 1 MB memory, so one can use the
address '0x33100000' (this info is based only on emperical tests, so beware).

If you specify conflicting memory addresses, the python dll loader will
fail with Symbian error KErrBadLibraryEntryPoint.

It is good idea to have a textfile/sheet to manage the used addresses.

If two dll collide, and no source is available, it should be possible
to monkeypatch the address manually. I have not researched this
further.

= Contact =
Simo Salminen, ssalmine@users.sf.net

[korakotc]

Thank you! Simo.

[AlfredoRU]

Hello!

Please, could you show us how to modify the build process in order
to build dlls with static writable data.

TLS is not suitable for multithreaded programs and EPOCSTAT approach has perfomance problems.

Thank You very much

NOTE: Is it possible to use this hack with standard Nokia Series 60 SDK?

[simo.salminen]

Hello,

here is example: http://pymbian.sourceforge.net/misc/statichack/

All you need to do is add these lines to .mmp

EPOCALLOWDLLDATA
EPOCCALLDLLENTRYPOINTS
EPOCDATALINKADDRESS 0x????????

and modify the E32Dll() entrypoint as shown in spammodule.cpp.

See the s60pyrex documentation for more info (especially about the EPOCDATALINKADDRESS, if it collides with other dlls in the process the dll wont load).

Yes, it is possible using standard sdk.

[ebtschi]

Hello,

the examples don't compile correctly when using Pyrex 0.9.4.
Everything will work fine if you use Pyrex 0.9.3.

ebtschi

[ballagas]

I have been trying this out on Symbian 3rd Edition SDK and I've been having some problems getting it to compile. I've changed /epoc32/system/include/Python/pyconfig.h to #define BAD_STATIC_FORWARD but I still get the following errors using the GCCE compiler:

error: '__pyx_f' was declared 'extern' and later 'static'
error: previous declaration of '__pyx_f'

Any hints?