Ability to change the default heap size

[tote_b5]

Hi all,

Is there any way to have influence on the heap size of the host (Python) application whose context my script is running in? I guess there is no way of doing that in case of the Python Shell, unless you re-build it. But that would be an undesirable option, since you would need to ship the re-compiled component along with your own, right? I checked it out if Ensymble could help me, but found that although it does support generating stand-alone applications, it does not allow us changing the heap size.

Background: I'd like to take photos with my N95's built-in 5MP camera. I've noticed, however, that doing that might require up to ~800k (with maximum JPEG quality) from free RAM, i.e. the host application's own heap. That along with the fact that Python Shell uses the default heap size (1MB) suggests me that that might be the reason why I'm getting (-4) KErrNoMemory errors every now and then even after taking a single shot.

Any ideas, suggestions, as to how I could solve this problem in Python?

Thanks in advance,

Tote

[jethro.fn]

Is there any way to have influence on the heap size of the host (Python) application whose context my script is running in?
...

Any ideas, suggestions, as to how I could solve this problem in Python?

Heap size is set during compilation and stored in the EXE header. Ensymble includes an EXE stub with each PyS60 application to start the Python interpreter. By recompiling the EXE stub the heap size can be selected freely. Ensymble will then need to be patched with the new EXE stub. The PyS60 interpreter SIS can remain unchanged.

Ensymble could provide a command line option to set the heap size. After all, the EXE stub header is already modified to set the application UID and capability bit mask. If there's enough interest I can release a new version of Ensymble with this feature.

[tote_b5]

Heap size is set during compilation and stored in the EXE header. Ensymble includes an EXE stub with each PyS60 application to start the Python interpreter. By recompiling the EXE stub the heap size can be selected freely. Ensymble will then need to be patched with the new EXE stub. The PyS60 interpreter SIS can remain unchanged.

Ensymble could provide a command line option to set the heap size. After all, the EXE stub header is already modified to set the application UID and capability bit mask. If there's enough interest I can release a new version of Ensymble with this feature.

Okay, that sounds like a viable option. Could you give me some pointers as to how I could recompile the exe stub? Please note that I'm a bit uncertain regarding what an exe stub really is. I have checked out Ensymble source carefully and seen that you use the stub in question. However, how that stub is made is still a big question mark to me as well as how I could have influence on the content of it.

And, of course, you can count on me in needing the new version of Ensymble already supporting heap size changing. But since I expect that to take some time, I expect that I'll need to hack the stub by myself first.

Tote

[jethro.fn]

Could you give me some pointers as to how I could recompile the exe stub? Please note that I'm a bit uncertain regarding what an exe stub really is.

You need to download PyS60 source (pys60-1.4.2_src.zip) from Sourceforge. Inside it there is a src\py2sis\templates_eka2 directory. There are several files with names ending in .template. Py2sis from Nokia uses these files to compile an EXE stub for the application. Ensymble does not require a working Symbian C++ toolchain but uses a precompiled EXE stub instead and patches it when required.

You need to apply the patch pytest.cpp.template.diff from Ensymble source package to the appropriate file using the Patch command line tool. Then you also need to replace every ${{...}} portion in every file ending in .template with an actual value. You'll need to invent some values, for example 12345678 for the UID. Finally, remove the .template suffix from all filenames and compile. You can take a look at the py2sis program source to figure out how it modifies the templates and compiles the final binary.

Ensymble stores the EXE stub as a base-64-encoded binary inside cmd_py2sis.py (variable exestubdata). Replace it with the compiled binary and run install.sh (or look what it does and do it by hand if you're not running Linux/Mac OS X/other Unix-like system) to generate a new ensymble.py.

[tote_b5]

Thanks for the hints, I finally managed to change the heap size for stand-alone PyS60 applications. Now I can take large photos (biggest 1.4MB in size) without any memory problems.

Please note that I noticed another problem with Ensymble: it was not able to cope with my base64 encoded private key in which it was NOT explicitly mentioned whether it was an RSA- or DSA-encoded key. In other words, it either expected

-----BEGIN DSA PRIVATE KEY-----

or

-----BEGIN RSA PRIVATE KEY-----

boundary, whereas mine was simply

-----BEGIN PRIVATE KEY-----

.

I'm not sure if the last format should always be defaulted to RSA, however, since mine was RSA-encoded (got from TrustCenter, btw) I adjusted cryptutil.py script so that it works the same way in the last two cases.

Thanks again,

Tote

[jethro.fn]

... whereas mine was simply

-----BEGIN PRIVATE KEY-----

.

I'm not sure if the last format should always be defaulted to RSA, however, since mine was RSA-encoded (got from TrustCenter, btw)

Thanks for the tip. I've never seen that kind of key myself, but I mostly use recent versions of OpenSSL anyway. This feature will be added to the upcoming release of Ensymble. Heap size adjustment works already, but I've yet to update the documentation.

[sstoica]

Hello guys,

I've run into the same problem. I need to make use of large strings streamed over a socket, but after I stream 1 or 2 of such strings I get "Memory Error". I get the error either when trying to decode the strings, either when reading from the socket (the error doesn't appear in the same spot every time) so I'm guessing to increase the heap size.

Can you please post a more detailed tutorial on how to do this?

Thanks

[tote_b5]

Hi,

Well, I always got KErrNoMemory (aka out-of-memory) errors and that was a clear indication that I needed more. The max heap size in PyS60 applications is 1MB by default - is it really not enough for you? On a resource-constrained device it's always recommended to keep as few data in memory as possible - do you persist (or do whatever else) your data once it's not needed anymore?

A generic "Memory error" seems suspicious to me, it might have something to do with small heap or might not.

Finally, as to your problem of not understanding fully what Jethro wrote: what is your problem actually? What do you not understand?

Cheers,

Tote

[sstoica]

Hi,

Well, I always got KErrNoMemory (aka out-of-memory) errors and that was a clear indication that I needed more. The max heap size in PyS60 applications is 1MB by default - is it really not enough for you? On a resource-constrained device it's always recommended to keep as few data in memory as possible - do you persist (or do whatever else) your data once it's not needed anymore?

A generic "Memory error" seems suspicious to me, it might have something to do with small heap or might not.

Finally, as to your problem of not understanding fully what Jethro wrote: what is your problem actually? What do you not understand?

Cheers,

Tote

I am trying to draw a big 3d scene using openGL, so my vertices and indices list that I am using are quite big. I must stream them over a socket from the pc, so when I serialize the two lists (for a medium-sized object) I get strings of 750k length.
So, since I can't import the gc module to call the gc manualy,
once I recieve the strings an decode them, I make sure to delete all the unnecesary data (del variable). Even so, I get the generic error: "Memory Error" which I can't figure out. So I am guessing to increase the heapsize...
I should mention that so far I have done all my testing only on the 3rd MR Emulator!

You need to download PyS60 source (pys60-1.4.2_src.zip) from Sourceforge. Inside it there is a src\py2sis\templates_eka2 directory. There are several files with names ending in .template. Py2sis from Nokia uses these files to compile an EXE stub for the application. Ensymble does not require a working Symbian C++ toolchain but uses a precompiled EXE stub instead and patches it when required.

i've done this.

You need to apply the patch pytest.cpp.template.diff from Ensymble source package to the appropriate file using the Patch command line tool.

i am not sure how to do this.

Then you also need to replace every ${{...}} portion in every file ending in .template with an actual value. You'll need to invent some values, for example 12345678 for the UID.

i've done this. replaced all
${{PY2SIS_APPNAME}} with ${{str3D}}
and
${{PY2SIS_UID}} with ${{0x01234567}}
and
${{PY2SIS_CAPS}} with ${{LocalServices Location NetworkServices PowerMgmt ProtServ ReadDeviceData ReadUserData SurroundingsDD SwEvent TrustedUI UserEnvironment WriteDeviceData WriteUserData}}

not sure if it's correct and how to replace this block:
${{if PY2SIS_AUTOSTART>0
START RESOURCE ${{PY2SIS_UID[2:]}}.rss
TARGETPATH \Resource\Apps
END
}}

Finally, remove the .template suffix from all filenames and compile. You can take a look at the py2sis program source to figure out how it modifies the templates and compiles the final binary.

not sure what to compile and what :| i'm sorry if am too much of a noob...

Ensymble stores the EXE stub as a base-64-encoded binary inside cmd_py2sis.py (variable exestubdata).

i can't find this variable insite cmd_py2sis.py

Replace it with the compiled binary and run install.sh (or look what it does and do it by hand if you're not running Linux/Mac OS X/other Unix-like system) to generate a new ensymble.py.

how do i do this on windows?


thanks a LOT for all your help

[jtullis]

Completely out of my depth here, but just wanted to check you'd seen the miso module which apparently let's you do stuff with heap sizes etc. Sorry if this is of no use!

http://pdis.hiit.fi/pdis/download/miso/

http://pdis.hiit.fi/pdis/download/mi...-1.92-api.html

[sstoica]

Completely out of my depth here, but just wanted to check you'd seen the miso module which apparently let's you do stuff with heap sizes etc. Sorry if this is of no use!

http://pdis.hiit.fi/pdis/download/miso/

http://pdis.hiit.fi/pdis/download/mi...-1.92-api.html

Hey,
Thanks for the answer. I have seen this module, don't know how to install it for the emulator but anyways from what I see you can't increase the heapsize of the app. This can only be done by recompiling pys60 or when you build your sis file out of the .py files.
I will try to use it to see at least how the memory is used and if the limited size of the heap is the problem for the generic error "Memory Error".

[tote_b5]

Patching the .cpp file should be straightforward if you're familiar with any source control management tool, such as Subversion. If not, even then it's not too difficult: check out what's the difference between \epoc32\tools\py2sis\template2_eka2\pytest.cpp and ensymble\exestub\pytest.cpp. Latter contains information for the merger tool (such as Patch, I assume, that I've never used): e.g. - means that you should remove that line, + means that you should add it, and so on.

When replacing "variables" that are enclosed between {{ and }}, please keep in mind that you should remove these signs, too. That is, a line specifying e.g. capabilities must look like this:
CAPABILITY NetworkServices Location

What to compile? Well, if you're familiar with the build process in Symbian, then it should be okay. Otherwise, you should get familiar with it.

Base-64 encoding: yeah, there's a variable in the python file in question, named execstubdata. It contains characters only from a pre-defined 6-bit dictionary (hence 2^6=64). You should take the entire content of generated .exe, which is in binary. Convert that to Base-64 form - I used Total Commander for that purpose, for example. Copy the generated Base-64 output in place of this variable and that's it.

Finally, obviously you don't need to run an .sh file on Windows (you could do that with some tricks, but why to bother?). The point is to invoke squeeze.py script in the right way, i.e. feeding it the right parameters in the right form. It should not be difficult if you're a bit familiar with Unix script shell.

Tote

[sstoica]

Thank you for the pointers. I will try it right away and give feedback.
Thanks again !

[jethro.fn]

Thank you for the pointers. I will try it right away and give feedback.
Thanks again !

What version of Python are you running on a PC? I can create a pre-release version of Ensymble with the heap size adjustment feature. I've been busy with other things lately and developing Ensymble has been a lower priority activity for me... The heap size adjustment feature is implemented in Ensymble already but testing and documentation is still not done, and there are other features I would like to implement in the next release.

[jethro.fn]

Here is a pre-release of Ensymble v0.27, making it possible to adjust the heap size of a PyS60 application:

• Pre-squeezed version for Python v2.3: ensymble_python2.3-0.27-pre1.py
• Pre-squeezed version for Python v2.4: ensymble_python2.4-0.27-pre1.py
• Pre-squeezed version for Python v2.5: ensymble_python2.5-0.27-pre1.py

New features:

Both the py2sis and altere32 commands feature a new option --heapsize. It takes a comma-separated list of two integer values; the minimum and maximum heap size. Values can have a k or M suffix to denote kilobytes and megabytes, respectively. If only one value is given, both values are set to that value.

Minimum heap size determines if an application can start at all. If there is less memory available than the minimum heap size of an application, an error may be displayed (nothing was displayed when tested on a Nokia N95) and the application is not started. Default minimum heap size is 4 kB.

Maximum heap size limits the amount of memory available to an application. The application cannot allocate more memory than this limit. The default maximum heap size is 1 MB.

When setting the minimum heap size to a larger value than the maximum heap size, a phone may not even install the application (tested on a Nokia N95).

For example, when an application needs to be able to use 5 MB of memory:

ensymble.py py2sis --heapsize=4k,5M myapp.py