Parsing atom feeds

**patmarc** · 2008-04-08, 10:00

Hello,

I have encountered problems with widgets when parsing atom feeds. I have been using the usual javascript method "getElementsByTagName()" to get the tags elements but some tags are partially or entirely not returned. The code below doesn't get the titles tags. At first I thought that the problems only occurred with tags using attributes but this new feed for testing has tags attributes like "summary" which get rendered properly. I am bit puzzled about this issue. Is it a problem with the widget engine or an error in the code ? I have previously made some other tests, the same code not working on widgets was working on firefox when parsing atom feeds.

Thanks in advance for your answers,

Patrice

Here is the code:

Javascript :

Code:

 
var ajaxRequest = null;

var url = 'http://feeds.sophos.com/en/atom1_0-sophos-security-news.xml';

window.onload = doRequest;

function doRequest()
{
	if (null == ajaxRequest)
	{
		ajaxRequest = new XMLHttpRequest();
	}
	ajaxRequest.onreadystatechange = ReqStateChange; 
	
	ajaxRequest.open( 'GET', url, true );
	ajaxRequest.send( null ); 
	
}
// state handler
function ReqStateChange() 
{
	// if request ready
	if ( ajaxRequest.readyState == 4 ) 
	{
		if ( ajaxRequest.status == 200 ) 
		{
			loadContent( ajaxRequest );
		} 
		else 
		{
			alert( "Error, could not load content" );
		}		
	}
}
function loadContent(argReq)
{
	var XMLData = null; 
	var title;
	var id;
	var summary;

	XMLData =  argReq.responseXML.documentElement;
	
	if (XMLData != null)
	{
		var titleNodes = XMLData.getElementsByTagName('title');
		var idNodes = XMLData.getElementsByTagName('id');
		var summaryNodes = XMLData.getElementsByTagName('summary');
		
		for (i = 0; i < titleNodes.length; i++)
		{
			title += titleNodes[i].childNodes[0].nodeValue + '<br\>';
		}
		for (i = 0; i < idNodes.length; i++)
		{
			id += idNodes[i].childNodes[0].nodeValue + '<br\>';
		}
		for (i = 0; i < summaryNodes.length; i++)
		{
			summary += summaryNodes[i].childNodes[0].nodeValue + '<br\>';
		}
			
	}
	document.getElementById("content").innerHTML = "<strong>Titles: </strong><br\>" + title + "<br\>" + "<strong>IDs: </strong><br\>" + id + "<strong>Summaries: </strong><br\>" + summary;
}

CSS:

Code:

#content {
	font-size: 12px;
	padding-top: 10px;
	display: display; 
}

XHTML:

Code:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>

<head>
	<link rel="stylesheet" type="text/css" href="index.css" />
	<script type="text/javascript" src="atom-widget.js"></script>
	
</head>

<body>

	
	<div id="content"></div>


</body>
</html>

Info.plist:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Nokia//DTD PLIST 1.0//EN"
"http://www.nokia.com/NOKIA_COM_1/DTDs/plist-1.0.dtd">
<plist version="1.0">
	<dict>
	<key>DisplayName</key>
	<string>atom widget</string>
	<key>Identifier</key>
	<string>com.widget.nokia</string>
	<key>MainHTML</key>
	<string>index.html</string>
	<key>AllowNetworkAccess</key>
	<true />
	</dict>
</plist>

**peppe@peppe.net** · 2008-04-08, 19:57

I've seen some issues related to namespaces being defaulted to XHTML even though you are parsing something other than XHTML (e.g. RSS or Atom). As a result some DOM methods are not returning what you expect if the attribute or tag name is something that exists in XHTML. Hopefully that gives you some ideas..

Peppe

**patmarc** · 2008-04-09, 10:27

Thanks Peppe for your answer. I might give a try to html instead of xhtml to see if it could help. On some previous tests I was trying to parse a tag named link with an 'href' attribute. Some of the content tags were returned but not all. Shouldn't xhtml recognize link tags and href attributes ? If xhtml was not able to recognize link tags and href attributes so why some of them are found ?

Thanks

**patmarc** · 2008-04-09, 13:54

A quick update. I have switched to HTML 4.01 but the problem still persists, not all the tags values are returned. The parsing was for href attributes from picture tags and term attributes from category tags.

**peppe@peppe.net** · 2008-04-09, 14:35

Originally Posted by patmarc

A quick update. I have switched to HTML 4.01 but the problem still persists, not all the tags values are returned. The parsing was for href attributes from picture tags and term attributes from category tags.

I didn't mean switch to HTML but rather that when using DOM methods in parsing the XML that the AJAX call has returned, some of the tags and attributes are considered by WebKit to be XHTML rather than RSS or Atom (or whatever) tags and attributes. E.g. if you have a title tag, id or lang attribute, etc. then they are considered to be in the XHTML namespace and not the default namespace for some reason.

As a test you can try to iterate through all the tags and dump out their tag names and namespace URIs. I believe you will find that your title tags have a namespace URI of "http://www.w3.org/1999/xhtml" while other tags have a null namespace (or possibly an Atom namespace).

Peppe

**xfour** · 2008-04-09, 18:42

If JSON is an option I would suggest using it, I have been successfully passing in data into the widget runtime on the n95 for some time now, with good results.

Sean

**peppe@peppe.net** · 2008-04-11, 07:44

Originally Posted by xfour

If JSON is an option I would suggest using it, I have been successfully passing in data into the widget runtime on the n95 for some time now, with good results.

I believe the original poster was talking about parsing ATOM, which rules out use of JSON since ATOM is an XML format. However JSON - being the native JavaScript object definition format - is an excellent choice for situations where you can define the data format yourself!

Peppe

**patmarc** · 2008-04-11, 12:10

Yeah that's right Atom is the mandatory data format I need to parse. I have made some other tests with a different code that the one previously posted. This new code works also fine on Firefox but not on widget. It's not able for instance to fetch all the link elements. Peppe as you suggested I have dumped those link tags name and name space URI. What I get is the right name space (http://www.w3.org/2005/Atom) but then after 12 elements (tags) parsed with the right tag name, I start to get a null value for the tag name, the name space URI is still Atom though. As I said previously this null return value doesn't occur when testing on Firefox.

**peppe@peppe.net** · 2008-04-11, 14:49

Originally Posted by patmarc

What I get is the right name space (http://www.w3.org/2005/Atom) but then after 12 elements (tags) parsed with the right tag name, I start to get a null value for the tag name, the name space URI is still Atom though. As I said previously this null return value doesn't occur when testing on Firefox.

That sounds very odd! I suggest trying with Safari as it has many of the same bugs and behaviors as WebKit does.

Peppe

**cristiano.diflora** · 2008-04-14, 12:52

Hi,

I am having almost the same issue (when parsing ATOM XML contents from a Javascript running within S60 browser).
I have tried to "debug" the situation with Safari, but unfortunately my code works perfectly in Safari

I really look forward to finding a workaround to this annoying issue.

Best regards,
-Cristiano

**peppe@peppe.net** · 2008-04-14, 13:16

Originally Posted by cristiano.diflora

I am having almost the same issue (when parsing ATOM XML contents from a Javascript running within S60 browser).
I have tried to "debug" the situation with Safari, but unfortunately my code works perfectly in Safari

I really look forward to finding a workaround to this annoying issue.

I would try to debug it by modifying the Atom data in various ways. For example change the order of the elements in the file to see if that changes anything. Copy-paste some element that is working now so that the entire file is just using working elements. Does it still fail at the 12th element? What if you add some extra stuff in the middle of the file that you don't even parse? Does it now fail sooner? Also try manually removing all namespaces from the file.

Those types of tests should help you narrow down whether it's the data, the sequence, order etc. that causes it to fail.

Perhaps if you can provide a link to a file that is failing for you it could help us in helping you debug this?

Peppe

**patmarc** · 2008-04-14, 14:21

Hello everyone,

I have removed the only namespace at the top of the Atom feed (xmlns="http://www.w3.org/2005/Atom"). The app download successfully the xml file but then seems to fail to process the resulting xml response. If there is few entries let's say 8, all of them will be processed properly. I was wondering if one of you knew about a way to make sure that the entire xml file has been downloaded ? I'll try by tomorrow to upload on my server one of the atom feed. If you look at the code pasted above you'll see already one of the atom feed. In this instance the titles were not fetched. (http://feeds.sophos.com/en/atom1_0-s...urity-news.xml).

Thank you for your help

**peppe@peppe.net** · 2008-04-14, 14:35

Looking at your code again I notice a few things. First of all you are creating break tags as <br\> instead of <br/>. Second, you are accessing text inside a tag using childNodes[0].nodeValue. If the content of the tag isn't normalized then you will not get the entire text content. Thus I'd recommend iterating through the child nodes and collecting text and cdata node values (not just text or cdata but *both*) and concatenating those. That might very well be where your problem
with partially cut off values might be.

I can't spot anything that would explain tag names being null though, but that was just debug code so perhaps you had some error there or something.

Peppe

**patmarc** · 2008-04-17, 10:03

I found out JFeed a lightweight JavaScript feed parser based on jQuery (http://www.hovinne.com/blog/index.ph...-parser-plugin) . On Safari and Firefox it parses properly all the elements but on widget it's not able to fetch the titles and links.
Here is the widget code:

jfeed-widget.js

Code:

jQuery(function() {

    jQuery.getFeed({
        url: 'nyt_atom.xml',
        success: function(feed) {
        
            jQuery('#result').append('<h2>'
            + '<a href="'
            + feed.link
            + '">'
            + feed.title
            + '</a>'
            + '</h2>');
            
            var html = '';
            
            for(var i = 0; i < feed.items.length && i < 5; i++) {
            
                var item = feed.items[i];
                
                html += '<h3>'
                + '<a href="'
                + item.link
                + '">'
                + item.title
                + '</a>'
                + '</h3>';
                
                html += '<div class="updated">'
                + item.updated
                + '</div>';
                
                html += '<div>'
                + item.description
                + '</div>';
            }
            
            jQuery('#result').append(html);
        }    
    });
});

jfeed-widget.html

Code:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>jFeed - jQuery feed parser plugin - example</title>
<style type="text/css">
    h3 { margin-bottom: 5px; }
    div.updated { color: #999; margin-bottom: 5px; font-size: 0.8em; }
</style>
<script type="text/javascript" src="jquery/jquery.js"></script>
<script type="text/javascript" src="build/dist/jquery.jfeed.pack.js"></script>
<script type="text/javascript" src="jfeed-widget.js"></script>


</head>

<body>
<h1>jFeed - jQuery feed parser plugin - example</h1>
<div id="result" />

</body>
</html>

Info.plist

Code:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Nokia//DTD PLIST 1.0//EN" "http://www.nokia.com/foo.dtd">
<plist version="1.0">
<dict>
	<key>DisplayName</key>
	<string>jfeed-widget</string>
		<key>Identifier</key>
	<string>com.nokia.jfeed-widget</string>
		<key>MainHTML</key>
	<string>jfeed-widget.html</string>
		<key>AllowNetworkAccess</key>
	<true/>
		<key>Version</key>
	<string>1.0</string>
</dict>
</plist>

You can have a test on Safari/Firefox browsers at the following address:

http://www.patmarc.com/jfeed (file example.html)

you'll find there as well the atom feed parsed nyt_atom.xml.

**peppe@peppe.net** · 2008-04-18, 07:23

Originally Posted by patmarc

You can have a test on Safari/Firefox browsers at the following address:

http://www.patmarc.com/jfeed (file example.html)

you'll find there as well the atom feed parsed nyt_atom.xml.

That Atom file is quite a bit larger than any file I've ever tried to parse. It could just be that you're running out of memory. As a test I suggest making the data shorter. Don't remove any elements - just take out some of the text (HTML) inside the CDATA elements for the content to make the file smaller.

I'm not suggesting that as a final solution but rather as a way to test what is causing the problems.

Peppe

Discussion Board

Thread: Parsing atom feeds

Thread Tools

Search Thread

Rate This Thread

Display

Similar Threads

In this Example how to get Attributes Value in xml Parsing??

Parsing wbxml file to xml file using kxml parser

org.xml.sax.SAXParseException in parsing XML only in real Series 60 3rd FP1 device

xml parsing & CMDXMLParser

XML Parsing

Posting Permissions