XML Response Parsing Problem with CParser, TDesC to TDesC8 Conversion problem

[samsbalu]

I am trying to receive a xml over HTTP and use CParser to parse it.

The HTTP API has MHTTPDataSupplier which returns a dataChunk which is a HBufC which is a 16 bit representation, But the CParser APIs need a TDesc8 and am having a problem with this inconsistency. If i try to just take the data chunk as a TptrC8 and create a HBufC8 from that, I am losing every alternate character as shown in the sample below.

Before conversion ::
<?xml version="1.0" encoding="utf-8"?><content><userid>578283</userid></content>

After Conversion ::
<xlvrin"."ecdn=uf8?<otn>uei>723/srd<cnet<

What is happenning is that the alternate characters are being lost in the 16 bit to 8 bit conversion!!!

can anybody help me on this,

Thanks in advance.

Here i am attaching my code pieces.


case THTTPEvent::EGotResponseBodyData:
{
// Get the body data supplier
MHTTPDataSupplier* body = aTransaction.Response().Body();


TPtrC8 dataChunk;

// GetNextDataPart() returns ETrue, if the received part is the last
// one.
TBool isLast = body->GetNextDataPart(dataChunk);

RDebug::Print( _L("***** ResponseBuffer length: %d"), dataChunk.Length() );



HBufC* buf = HBufC::NewLC(dataChunk.Length());
buf->Des().Copy(dataChunk);
RDebug::Print( _L("***** buf: %S"), buf );


// Append to iResponseBuffer
if (iResponseBuffer==NULL)
{
iResponseBuffer = buf->AllocL();
RDebug::Print( _L("***** ResponseBuffer: %S"), iResponseBuffer );
}
else
{
RDebug::Print( _L("***** ResponseBuffer content length: %d"), iResponseBuffer->Length()+buf->Length() );

iResponseBuffer = iResponseBuffer->ReAllocL(iResponseBuffer->Length()+buf->Length());
iResponseBuffer->Des().Append(*buf);
}
// xml parser StartParsingL method with TDesC

iXmlHandler->StartParsingL(*iResponseBuffer);

------------------
XmlHandler class
------------------
void CXmlHandler::StartParsingL( TDesC& xmlContent )
{


HBufC8 *pHeapUTF = CnvUtfConverter::ConvertFromUnicodeToUtf8L(xmlContent);
RDebug::Print( _L("***** content: %S"), pHeapUTF );

iParser->ParseBeginL();
iParser->ParseL( *pHeapUTF);
iParser->ParseEndL();


}


I tried with the following ::


HBufC* buf = HBufC::NewLC(xmlContent.Length());
buf->Des().Copy(xmlContent);


TPtrC8 representation((TUint8*)buf->Ptr(), buf->Size());

HBufC8* myBuf = HBufC8::NewLC( buf->Size());
myBuf->Des().Copy(representation);
RDebug::Print( _L("***** content8: %S"), myBuf );

Getting null characters at the end and missing even characters also.

Log information ::



26.975 ***** ResponseBuffer length: 80
26.975 ***** buf: <?xml version="1.0" encoding="utf-8"?><content><userid>578283</userid></content>
26.975 ***** ResponseBuffer: <?xml version="1.0" encoding="utf-8"?><content><userid>578283</userid></content>
26.985 ***** ResponseBuffer: <?xml version="1.0" encoding="utf-8"?><content><userid>578283</userid></content>
27.000 ***** content: <xlvrin"."ecdn=uf8?<otn>uei>723/srd<cnet<

[vaibhavjain]

I guess the problem lies here. You arent performing unicode conversion of incoming TDesC8 data.

HBufC* buf = HBufC::NewLC(dataChunk.Length());
buf->Des().Copy(dataChunk);
RDebug::Print( _L("***** buf: %S"), buf );

Changing that to
HBufC* buf = CnvUtfConverter::ConvertToUnicodeFromUtf8L(dataChunk)

should solve the issue.

Also one more point. Why are your trying to convert TDesC8 data to Unicode and then back to TDesC8?

Hope that helps,
VJ

[qt_phantom]

If you using only english chars in xml, Try this

void CXmlHandler::StartParsingL( TDesC& xmlContent )
{
HBufC8 *pHeap = HBufC8::NewL(xmlContent.Length());
TPtr8 pHeapPtr(pHeap->Des());
pHeapPtr.Copy(xmlContent);
RDebug::Print( _L("***** content: %S"), pHeapPtr );

iParser->ParseBeginL();
iParser->ParseL( pHeapPtr);
iParser->ParseEndL();
}

[samsbalu]

Hi Vaibhav,
The problem is that the data that I am receiving in the datachunk is expected to be a xml string as described in the sample data.

But when we alloc that buffer into a HBufC8 to send it to the CParser, the alternate characters are lost!!!

So <?xml .... in the data chunk, is displayed as <xl ... So my parser is failing.

but when we alloc from the datachunk to a HBufC then the <?xml is printed correctly as <?xml , but I am not able to send this to the CParser methods since they expect a TDesC8

thanks,

I guess the problem lies here. You arent performing unicode conversion of incoming TDesC8 data.

HBufC* buf = HBufC::NewLC(dataChunk.Length());
buf->Des().Copy(dataChunk);
RDebug::Print( _L("***** buf: %S"), buf );

Changing that to
HBufC* buf = CnvUtfConverter::ConvertToUnicodeFromUtf8L(dataChunk)

should solve the issue.

Also one more point. Why are your trying to convert TDesC8 data to Unicode and then back to TDesC8?

Hope that helps,
VJ

[vaibhavjain]

You are loosing alternate charecter due to inconsistant conversions that you are performing. Form converting TDesC8 to TDesC you are uing TDesC::Copy method as in

buf->Des().Copy(dataChunk);

but for conversion from TDesC8 to TDesC you are using

HBufC8 *pHeapUTF = CnvUtfConverter::ConvertFromUnicodeToUtf8L(xmlContent);

;

CnvUtfConverter::ConvertFromUnicodeToUtf8L will simply strip each alternate byte from the input (hi byte of the unicode charecter) and return the remaining bytes. Thats why you are loosing half of your data. Since xml is textual data (unicode) the best way to convert it TO/FROM TDesC8(UFT) is to use CnvUtfConverter class's static methods.

Alternate method would be (if you are only using UTF-8 data) is to use the TDes16/8::Copy() methods. But whatever methods you choose be sure to use same for both way conversions.

Your bug is simply due to mixing up of two conversion procedures. 2 Posts above ( 1 from me and another from qt_phantom) provide valid solutions to this bug.

Hope that helps,
VJ

[samsbalu]

Here i am using the same conversions from CnvUtfConverter

TBool isLast = body->GetNextDataPart(dataChunk);
RDebug::Print( _L("***** ResponseBuffer length: %d"), dataChunk.Length() );
HBufC* unicode = CnvUtfConverter::ConvertToUnicodeFromUtf8L(dataChunk);
HBufC8* utf = CnvUtfConverter::ConvertFromUnicodeToUtf8L(*unicode);

RDebug::Print( _L("***** unicode: %S"), unicode );
RDebug::Print( _L("***** utf: %S"), utf );

Log is ::

34.430 ***** ResponseBuffer length: 80
34.435 ***** unicode: <?xml version="1.0" encoding="utf-8"?><content><userid>578283</userid></content>
34.440 ***** utf: <xlvrin"."ecdn=uf8?<otn>uei>723/srd<cnet

[qt_phantom]

did you tried copy a 16-bit discripter directly to 8-bit discripor as in my previous reply? without using CnvUtfConverter etc.

[samsbalu]

Yes i tried with

HBufC8 *pHeap = HBufC8::NewL(xmlContent.Length());
TPtr8 pHeapPtr(pHeap->Des());
pHeapPtr.Copy(xmlContent);
RDebug::Print( _L("***** content: %S"), pHeapPtr );

getting User Panic 7

[skumar_rao]

RDebug::Print( _L("***** content: %S"), pHeapPtr ); has problem.

[qt_phantom]

sorry,
yes problem is
RDebug::Print( _L("***** content: %S"), pHeapPtr );

it should be

RDebug::Print( _L("***** content: %S"), pHeap );