HTTP POST - Upload multiple files

[bgenady]

Hi, guys.

I have to upload mulitple files to some server.

I faced some strange problems there:

1. At first, I performed regular HTTP post with all required headers and in GetNextDataPart() I put the being uploaded file content:

TBool CXXEngine::GetNextDataPart(TPtrC8& aDataPart)
{
  if(iPostData) 
  {
	// Provide pointer to next chunk of data (return ETrue, if last chunk)
	// Usually only one chunk is needed, but sending big file could require
	// loading the file in small parts.
	aDataPart.Set(iPostData->Des());
  }
return ETrue;
}
void CXXEngine::ReleaseData()
{
  if(iPostData)
  {
	delete iPostData;
	iPostData = NULL;
}

}

It worked well in case of single upload (i.e. after each file upload some other HTTP GET was performed).
It also worked well on files with different sizes fom 500 bytes till Kb and Mb.

2.After that I had to implement upload sequence of multiple files. Here the problems started.

The process looks like that:
MyAppUi (to make things simple) called some UploadFoo(...) (like well-known IssueHTTPPostL(...) from HTTP example). This function performed all necessary transaction/headers/SubmitL()/etc. stuff. When I received ESucceeded/EFailed I notified AppUi about that and AppUi after 1sec timer called UploadFoo() again to upload next file (in timer thread).

On emulator worked fine, but on the device (N80) sometimes the application was closed. After some research I noticed that after several uploaded files ReleaseData() was not called. And after that on 2nd or 3rd file upload the application simple hang up.

After some additional research I tried to upload many files with less-than-1k length and "voila" - every file was uploaded.

My conclusion was that I had to divide my file content to chunks.
(I still don't understand why it works when i upload only one large file??)

I looked at HTTPEXAMPLECLIENT, at very useful posts of Piotr Kundu and reimplemented the GetNextDataPart() and ReleaseData() functions.

Now every time GetNextDataPart(TPtrC8& aDataPart) is called, I put into aDataPart next chunk of file content

TBool CXXEngine::GetNextDataPart(TPtrC8& aDataPart)
{
	TBool bRetVal = EFalse;
	if(iPostData)  // whole file buffer - HBufC8*
	{
		iPostDataChunk->Des().Zero(); // 1K HBufC8* that created in UploadFoo()
		TInt nTotalLen = iPostData->Des().Length();
		TInt nLastPos = inDataChunkCount*KMaxSubmitSize; // KMaxSubmitSize - 1024
		TInt nNewPos = (inDataChunkCount+1)*KMaxSubmitSize;
		if(nNewPos > nTotalLen)
		{
			if(nLastPos > nTotalLen)
			{
				bRetVal = ETrue;
				iPostDataChunk->Des().Copy(_L(""));
			}
			else
			{
				iPostDataChunk->Des().Copy( iPostData->Des().Mid(nLastPos) );
				inDataChunkCount++;
			}
		}
		else
		{
			iPostDataChunk->Des().Copy( iPostData->Des().Mid((inDataChunkCount*KMaxSubmitSize), KMaxSubmitSize) );
			inDataChunkCount++;
		}

		aDataPart.Set(iPostDataChunk->Des());
	}
	else
	{
		bRetVal = ETrue;
	}

	// return true if last chunk
	return bRetVal;

}

The ReleaseData() function changed also:

void CXXEngine::ReleaseData()
{
	// Clear out the submit buffer
	iPostDataChunk->Des().Zero();
	ibReleased = ETrue;
	// Notify HTTP of more data available immediately, since it's being read from file
	TRAPD(err, iTransaction.NotifyNewRequestBodyPartL());
	if (err != KErrNone)
	{
		_LIT(KNotifyNewRequestBodyPart, "NotifyNewRequestBodyPartL left");
		iObserver.StateChanged(eStateUnrecognisedEvent, KNotifyNewRequestBodyPart);
	}
}

I received -20

The questions are:
- If I divide my file content to chunks, should I implement multipart/form-data upload or I can simple set next chunk of data in GetNextDataPart()?

- I saw also that GetNextDataPart is called twice without ReleaseData() is called - is it ok?

I'm struggling with it couple of days...
I miss something very basic here.....
Any help will be more than appreciated.

Regards,
Genady

[nhammond]

Let me know how you get on with this one, Im struggling with it.

What sort of files are you uploading? images?

When the upload occurs on the server side is the data held in the POST array or the FILES array?

I have tried using content type of multipart/form-data and then construct the body myself, however none of the body seems to upload. I do specify a boundary as well.

[bgenady]

I found the problem. It's not directly related to file size.

As I said, I'm trying to upload multiple files to the server each after another.

When I submit HTTP POST transaction I put some information into custom headers. When server recieve incoming HTTP POST request it can decide to receive or not this file (it can be decided after headers observation).

That was happened in my case. Server received only headers and decided not to accept the file and returned "200 OK" without waiting for HTTP POST request body. It caused crash in some of Symbian HTTP stack related components after 2nd or 3rd subsequent upload.

What I would expect to get is some Error notification in MHFRunError(...) or in MHFRunL(...) callbacks - I receive nothing strange - EGotResponseHeaders, EGotResponseBodyData, EResponseComplete, ESucceeded.

It seems like a Symbian bug.

nhammond,
Unfortunately I don't deal with multipart/data-form. I put the file content in HTTP request body and upload it with content type "application/octet-stream" - in any case this info is irrelevant for the server, I put this header just because otherwise I receive some error about missing header

Regards,
Genady

[nhammond]

Say for example I wanted to upload a jpeg file. Can I just read the jpeg directly into a descriptorand post it? If so, when the server recieves the data how do I save the data back into a jpeg? Or do I have to convert it to binary before sending? This is probably out of the realms of this forum. Im using php my the way.

[utopia2006]

I also encounter the same problem, i send a file in serveral parts,but it return
aEvent.iStatus==-20 in the function of MHFRunL().If i send the file in one part,it can upload the file sucessfully.
Here is my codes:

TBool CHttpEngine::GetNextDataPart(TPtrC8& aDataPart)
{
TInt iLen = iPostData->Length() - iDataChunkCount * KMaxChunkSize;

if(iLen > KMaxChunkSize)
{
iReqBodySubmitBuffer->Des() = iPostData->Mid(iDataChunkCount * KMaxChunkSize, KMaxChunkSize) ;
iLast = EFalse;
}

else if(iLen>=0 )
{
iReqBodySubmitBuffer->Des() = iPostData->Mid(iDataChunkCount * KMaxChunkSize, iLen) ;
iLast = ETrue;
}

aDataPart.Set(iReqBodySubmitBuffer->Des());

++iDataChunkCount;

return iLast;
//return !(iLen - 1024 >0) ;

}

void CHttpEngine::ReleaseData()
{
TPtr8 buff = iReqBodySubmitBuffer->Des();
buff.Zero();

if(!iLast)
{

TRAPD(err, iTransaction.NotifyNewRequestBodyPartL());
}


}



TInt CHttpEngine::OverallDataSize()
{

if(iPostData)
{

return iPostData->Length();
}

else
{

return KErrNotFound ;
}

}