Sending File to Server using HttpConnection

[sandeepkumar03]

Hi All,

I have J2ME Client, Apache Tomcat Server 5.5. I am able to send file from J2ME Client to server upto certain limit. But once that limit crosses, i get OutOfMemory error. I did search on forum. One link. There were some more. They were talking about sending data in chunks.

I tried to get all data in ByteArrayOutputStream. - But when i write on dataOutputStream.write(byteArrayOutputStream.toByteArray()). Got OutofMemory error.

I tried sending one by one byte

while ((input.read(nextByte, 0, 1)) != (-1)) {
   daos.write(nextByte[0]); //daos is outputstream on httpconnection
}

But here one httpConnection is opened and then writing on that gives Symbian OS Error -4 (Not enough memory).

I tried setting "Transfer Encoding" to "Chunked" - But this gave response from server side : java.io.IOException: Invalid chunk header.

One way could be opening the HttpConnection again and again for each chunk of data and appending that on server side. So open httpconnection - Send Chunk - close connection. But is it advisable to open the httpconnection again and again and sending chunk of data. Or we have some way to open the connection once and then send the chunk of data.

Just for test i tried to send the stream of data in respose of servlet. It was able to send around 8MB of file to client (Didn't try more than that). So receiving file from server is no issue. But sending file is giving issue.

How to chunk data and send it to server?

Please guide or give some pointers which could help.

Thanks,
Sandeep

[sandeepkumar03]

Hi All,

I tried to use flush also.

while ((input.read(nextByte, 0, 1)) != (-1)) {
   daos.write(nextByte[0]); //daos is outputstream on httpconnection
   daos.flush();
}

It does not throw OutOfMemory exception but on server only first byte is written. So after flush the data is not getting written.

Thanks,
Sandeep

[sandeepkumar03]

Hi All,

Tried with Sun Simulator (WTK 2.5.2). There i even tried to send a file of 40MB. It was sucessfully transferred by using os.flush() or even without using os.flush().

But on S60 3rd Edition FP2 Simulator. os.flush() is just writing first chunk of data. And without using os.flush() it throws OutOfMemory Error.

Anyone has tried this on S40 simulator?

I am not understanding how this could be solved. Please give some pointers

Thanks,
Sandeep

[sandeepkumar03]

Hi All,

Tried on S40 6th Edition Simulator as well. Here also os.flush() is just pushing first chunk of data. and without os.flush() it gives OutOfMemory Error.

Does it behaves similarly on Device also? I would need Static IP to test it. If any one had done file transfer through phone. Please let us know your observations.

Will flush() work on device? Or we have to do some thing else to send bigger file?

Please provide some pointers.

Thanks,
Sandeep

[jitu_goldie]

Hi All,

Tried on S40 6th Edition Simulator as well. Here also os.flush() is just pushing first chunk of data. and without os.flush() it gives OutOfMemory Error.

Does it behaves similarly on Device also? I would need Static IP to test it. If any one had done file transfer through phone. Please let us know your observations.

Will flush() work on device? Or we have to do some thing else to send bigger file?

Please provide some pointers.

Thanks,
Sandeep

Write the data in chunks in outputstream. Make a small chunk of data upto of 100 kb. Do Flush only one time when all the data is written.

[sandeepkumar03]

Hi Jitu,

Write the data in chunks in outputstream. Make a small chunk of data upto of 100 kb. Do Flush only one time when all the data is written.

We would write in chunks like this right

byte[] buffer = new byte[100 * 1024]; // 100 Kb buffer
while ((input.read(buffer , 0, buffer.length)) != (-1)) {
   daos.write(buffer); //daos is outputstream on httpconnection
}
daos.flush();

This was giving IOException Symbian OS Error -4 (Not enough memory) on S60 simulator. Haven't tested this on device. It might be a simulator problem?

Thanks,
Sandeep

[sandeepkumar03]

Hi,

As i kept on researching on this issue. I tried different permutations and combinations.

Server Issue : I thought it may be server issue. tried with JBoss instead of Tomcat. It seems to be making no difference.

Open Close Connection and Append Data on Server : This is also one option. As in single connection i am getting Symbain OS Error -4. Got some info related to this on Sending Data in Chunks thread.. In this Amit has mentioned about this. We can append the data on the server side by using FileOutputStream in append mode. Will this cause and issue in case chunk is lost and file might get corrupted?

Limiting the user for sending files may be till 2 MB : As larger file has problems. Limit user to some size. Is that a good option?

Device Testing Observations : Tested on E61, E71 both had different behviours. On E71 i was able to send around 4 MB but on S61 got Symbian OS Error -4. Operator also limits sometime. I had Vodafone live GPRS Connection it was not working using that. I had to get VMC enable( Mobile Office in case of Airtel) to make it work. Simple GPRS connection may not allow to send files for certain MIME Type.

Inputs on file transfer to server would help.

Thanks,
Sandeep

[grahamhughes]

On many implementations, all the data written to an HttpConnection is buffered. Flushing the OutputStream, calling getResponseCode(), openInputStream(), or several other methods (you can find a list in the JavaDocs for HttpConnection), causes the transaction to occur. At this point, all the buffered information is sent to the server, and the response from the server is read and buffered.

Because of this, you cannot send or receive more information in one transaction than you can fit in the heap.

If you want to send or receive more information, you will need to split it into multiple transactions (multiple HttpConnections).

Graham.

[sandeepkumar03]

Hi Graham,

I have checked on Sun WTK. flush() was working perfectly fine there. But not on S40, S60. I was just experimenting with the flush() to see if that avoids the OutOfMemory or Symbian OS Error -4. So that i would be able to send a bigger file. But now i can understand because of Heap limitation there would be limit in transactions.

If you want to send or receive more information, you will need to split it into multiple transactions (multiple HttpConnections).

So this seems to be the only option to do multiple connections. In case of bigger file, split it and send as seperate requests and then on server end join these using following.

FileOutputStream outStream = new FileOutputStream(backupFile, true);

And if we want to avoid this, we have to limit user to some boundry line may be 2MB or something like that.

Graham if you get some time, please see the different observations in the thread and give your opinion. Problem statement was to send bigger file onto server.

Best regards,
Sandeep

[grahamhughes]

OutputStream.flush() works fine on Series 40 and 60... it just doesn't do what you want. It does fulfil its contract, ensuring that buffered data is written to the destination. There is no requirement that the OutputStream still has to be useful afterwards (odd though that may seem).

Yes, you need to split the data into pieces, send each piece separately (separate HttpConnection), and re-assemble the pieces on the server. Or, as you say, limit the size to whatever fits in the memory.

Graham.

[sandeepkumar03]

OutputStream.flush() works fine on Series 40 and 60... it just doesn't do what you want. It does fulfil its contract, ensuring that buffered data is written to the destination. There is no requirement that the OutputStream still has to be useful afterwards (odd though that may seem).

Hi Graham,

I dont know but may be my understanding about flush() is incorrect. I just observed following things.

 byte[] data = new byte[4096]
 int count = inputStream.read(data, 0, data.length);
 while (count != -1) {
     outputStream.write(data, 0, count);
     outputStream.flush();
     count = inputStream.read(data, 0, data.length);
 }

On Sun WTK 2.5 this worked fine on for sending file through HTTPConnection or Writing locally on phone memory using FileConnection.

On S60, S40 this just sends the 4Kb of data while using HTTPConnection. I have to comment flush() line to make it work on S60, S40. I thought flush() would write the data to destination and heap space would be freed so that we dont have memory problems. I thought without using flush() there might be buffering which might be occupying heap space.

But this works with flush() when writing to local file using FileConnection on S60, S40.

So while writing to HTTPConnection output stream it seems to be a issue.

I don't know but J2ME gives lot of surprises on different devices/simulators . Needs experience to understand those. I have not tested on Sony Ericsson and others may be more surprises are awaiting

Best regards,
Sandeep

[grahamhughes]

The important thing to remember is that OutputStream is an abstract class, so you are actually dealing with objects of several different subclasses of OutputStream. Writing to a FileConnection object, you are using a different class of object than you are when writing to an HttpConnection, which is why they behave differently.

The confusion is that this:

httpout.write(data, 0, data.length);
httpout.flush();

works exactly as you expect. But this:

httpout.write(data, 0, data.length);
httpout.flush();
httpout.write(moreData, 0, moreData.length);
httpout.flush();

does not do what you expect. Only the first flush() actually seems to do anything. That's because, in many implementations (and this is not unique to Nokias), flushing an OutputStream from an HttpConnection causes the entire HTTP transaction to take place. All the bufferered output is sent to the server, and the response is read back from the server and buffered on the phone. Since the transaction is now complete, no more information can be sent, hence the second flush() doesn't seem to do anything.

So, flush() does work, but it leaves the HttpConnection object in a state where no more information can be sent (so only the first flush() actually does anything).

Expect surprises!

Graham.

[sandeepkumar03]

Thanks a lot Graham,

Now i understand and able to conclude it properly. And yeah surprises are required to bring excitement in life

And if surprises are not getting solved. We have Nokia Forum Experts

Best regards,
Sandeep