very strange problem with marshal

[sstoica]

hello,
I get a very strange problem with marshal when trying to serialize a big list and send it over a socket. It works ok on a pc-pc comunication, but it gives an EOF error when moving the client on the emulator:

the problem is that i can send the small and medium sized lists, but i get and EOFError: EOF read where object expected when i try to send the big list.

Can anyone see where's the problem ?

here's my (simple) test code:

server:

from socket import *
import marshal
import sys
sys.path.append("C:\customlibs")
import objparser

myHost = ''
myPort = 2000

s = socket(AF_INET, SOCK_STREAM)    # create a TCP socket
s.bind((myHost, myPort))            # bind it to the server port
s.listen(5)                         # allow 5 simultaneous
                                    # pending connections
stop = 0
stopconnection = 0

vertice_to_send = []
indice_to_send = []
string_to_send = [ (1,2,3) ]

#parse the obj
file = open ("C:\sphere.obj", "r")
parser = objparser.objParser()
parser.parse_file(file)
file.close()

vertice_to_send = marshal.dumps ( parser.get_vertices() )
indice_to_send = marshal.dumps ( parser.get_indices() )

print len(vertice_to_send)
print len(indice_to_send)
print len(marshal.dumps(string_to_send))


while not stop:
    # wait for next client to connect
    print "waiting for connection..."

    stop = 0
    stopconnection = 0
    connection, address = s.accept() # connection is a new socket
    print "connection accepted from", address

    while not stopconnection:
        print "after connection accepted"

        data = connection.recv(1024) # receive up to 1K bytes    
        if data:
            print data," recieved from ", address
        
            if data == "killme":
                stop = 1
                stopconnection = 1
                break

            elif data == "get_sphere_vertices":
                connection.send(vertice_to_send)
                print "after vertices sent"

            elif data == "get_sphere_indices":
                connection.send(indice_to_send)
                print "after indices sent"

            elif data == "test":
                connection.send( marshal.dumps(string_to_send) )
                print "after test sent"
            
        else:
            print "connection closed"
            stopconnection = 1

connection.close()              # close socket

client:

import marshal
from socket import *

serverHost = 'localhost'
serverPort = 2000


if __name__ == '__main__':

    file = open(u'C:\\emu_data_streamed.txt','a')

    s = socket(AF_INET, SOCK_STREAM)
    s.connect((serverHost, serverPort))

    """
    #test sending a small list (len = 25)
    s.send("test") 
    data = marshal.loads( s.recv(10240) ) #10K of data
    print data
    file.write( str(data) )
    """
    """ 
    #test sending a medium-sized list (len = 3845)
    s.send("get_sphere_indices") 
    data = marshal.loads( s.recv(10240) ) #10K of data
    print data
    file.write( str(data) )
    
    """
    #test sending a big list (len = 8645)
    s.send("get_sphere_vertices") 
    data = marshal.loads( s.recv(10240) ) #10K of data
    print data
    file.write( str(data) )

[sstoica]

Ok, so the length of the big list is 432. If I try

vertice_to_send = marshal.dumps ( parser.get_vertices()[0:409])

to send the first 409 elements of the list, it works. For more

vertice_to_send = marshal.dumps ( parser.get_vertices()[0:410])

I get the EOF error. I don't understand...

[sstoica]

ok, case sovled. if anyone is interested, the reading from the socket should be put in a while until you are SURE you've read all. s.recv(10240) doesn't guarantee that you actualy read all the data in this case (~8500).

while len(data) < msg_length:
        data += s.recv(1024) #1K of data

also, close the file at the end after writing to it !

[JOM]

reading from the socket should be put in a while until you are SURE you've read all. s.recv(10240) doesn't guarantee that you actualy read all the data

Usually socket data is buffered, that's true. When looking at e32socketmodule.cpp there's "#define MaxBufferSize 1024", which most likely defines internal buffer size as one kilo.

However did you notice you could have requested some buffer size by yourself, in theory at least:

    def recv(self, n, f=0, cb=None):
        self._recvsizehint=n

Cheers,

--jouni