AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

[Klen]

Hello,

Just to make life easier to anyone who wants to do AR on N900 i have a few suggestions.

1.) Use openCV library for any tracking and image manipulation. It is an amazing library that is available in the repo.

M900:
$ apt-get install libcv4 libcvaux4 libhighgui4

Scratchbox:
$ apt-get install libcv4 libcv-dev opencv-doc libcvaux-dev libcvaux4 python-opencv libhighgui-dev libhighgui4

2.) For capturing video use openCV library calls or go through the hassle of GStreamer (appsink element) which can be integrated with QT using QGLWidget object. I attached source code of a project GstOpenGL that demonstrates how this can be done. Make sure you have qt 4.6 on your N900 or scratchbox. The GstOoenGL does not use openCV.

The code has not been cleaned and optimized but should save you time.

Cheers.
Klen

[Klen]

Hello,

GstOpenGL app shows you how to capture a video stream frame by frame using Gstreamer and display it on the QGLWidget component.

Make sure you have Qt 4.6 to get better performance. The app captures the video stream at 400×240 and in full screen runs at 20 FPS.

Hope you find this useful.

BTW: The code has not been cleaned and optimized, but should be useful.

GstOpenGL_V1.tar

Klen

[zr_yao]

Hi Klen you really save my life! Great job man!

[zr_yao]

Own update and correction, I finally found the issue and make the gles1 version working! and see that the original version is "live stream" on rotating box and also live background !

The issue was that in gles1 you cannot glTexture2D a 400x240 texture, instead use 512x512 and the sub texture it will do the trick.

For people who want to use gles1 version

const int TextureSizeW = 512;
const int TextureSizeH = 512;
int GLWidget::buffer_width=400;
int GLWidget::buffer_height=240;

void GLWidget::initializeGL ()
{
    glClearColor(0.1f, 0.1f, 0.2f, 1.0f);
    glEnable(GL_TEXTURE_2D);
    glGenTextures(1, &m_uiTexture);
   // m_uiTexture = bindTexture(QImage(":/qt.png"));
    glBindTexture(GL_TEXTURE_2D, m_uiTexture);
    glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB, TextureSizeW,TextureSizeH, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL );
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );

    m_fAngle = 0;
    m_fScale = 1;
    createGeometry();
    createBubbles(bubbleNum - bubbles.count());


}

void GLWidget::refresh_texture()
{
	if(CameraN900::scanning)
	{
		CameraN900::refresh_buffer(); //manually call appsink, not through the callback function

		if(CameraN900::buffer!=NULL)
		{
			glBindTexture( GL_TEXTURE_2D, m_uiTexture );
			unsigned char* data=(unsigned char *) GST_BUFFER_DATA (CameraN900::buffer);
			//glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB, buffer_width,buffer_height, 0, GL_RGB, GL_UNSIGNED_BYTE, data );
			glTexSubImage2D (GL_TEXTURE_2D, 0, 0, 0, buffer_width,buffer_height, GL_RGB, GL_UNSIGNED_BYTE, data);

		}
		gst_buffer_unref(CameraN900::buffer);
	}
}

My working paintGL looks as

void GLWidget::paintGL()
{
    createBubbles(bubbleNum - bubbles.count());

    QPainter painter;
    painter.begin(this);

    glClearColor(0.1f, 0.1f, 0.2f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    glEnable(GL_TEXTURE_2D);
    refresh_texture();
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );

      drawBackground(&painter);
//////////////////////////////////////////////////////////////////////////////
glMatrixMode(GL_PROJECTION);
    glPushMatrix();
    glLoadIdentity();
    glMatrixMode(GL_MODELVIEW);
    glPushMatrix();
    glMatrixMode(GL_TEXTURE);
    glPushMatrix();

    //Since OpenGL ES does not support glPush/PopAttrib(GL_ALL_ATTRIB_BITS) 
    //we have to take care of the states ourselves

    glEnable(GL_TEXTURE_2D);
    q_glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
    q_glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
    glEnable(GL_LIGHTING);
    glEnable(GL_LIGHT0);

    glShadeModel(GL_FLAT);
    glFrontFace(GL_CW);
    glCullFace(GL_FRONT);
    glEnable(GL_CULL_FACE);
    glEnable(GL_DEPTH_TEST);

    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();

    q_glRotate(f2vt(m_fAngle), f2vt(0.0), f2vt(1.0), f2vt(0.0));
    q_glRotate(f2vt(m_fAngle), f2vt(1.0), f2vt(0.0), f2vt(0.0));
    q_glRotate(f2vt(m_fAngle), f2vt(0.0), f2vt(0.0), f2vt(1.0));
    q_glScale(f2vt(m_fScale), f2vt(m_fScale),f2vt(m_fScale));
    q_glTranslate(f2vt(0),f2vt(-0.2),f2vt(0));

    q_vertexType matDiff[] = {f2vt(0.40), f2vt(1.0), f2vt(0.0), f2vt(1.0)};
    q_glMaterialv(GL_FRONT_AND_BACK, GL_DIFFUSE, matDiff);

    if (qtLogo)
        paintQtLogo();
    else
        paintTexturedCube();
    glMatrixMode(GL_MODELVIEW);
    glPopMatrix();
    glMatrixMode(GL_PROJECTION);
    glPopMatrix();
    glMatrixMode(GL_TEXTURE);
    glPopMatrix();

    glDisable(GL_LIGHTING);
    glDisable(GL_LIGHT0);

    glDisable(GL_DEPTH_TEST);
    glDisable(GL_CULL_FACE);
	if (m_showBubbles)
		foreach (Bubble *bubble, bubbles) {
			bubble->drawBubble(&painter);
	}
    QString framesPerSecond;
    framesPerSecond.setNum(frames /(time.elapsed() / 1000.0), 'f', 2);
   painter.setPen(Qt::white);
    painter.drawText(20, 40, framesPerSecond + " fps");
   painter.drawRect(0,0,799,479);
   painter.end();
    swapBuffers();
    QMutableListIterator<Bubble*> iter(bubbles);

    while (iter.hasNext()) {
        Bubble *bubble = iter.next();
        bubble->move(rect());
    }
    if (!(frames % 100)) {
        time.start();
        frames = 0;
    }
    m_fAngle += 1.0f;
    frames ++;
 }

and finally the drawBackground function for gles1 using texture as background

void GLWidget::drawBackground(QPainter *painter)
{
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();
glOrthof(0, 1, 0, 1, -1.0, 1.0);
glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();
glClear( GL_DEPTH_BUFFER_BIT);
glDisable(GL_BLEND);
glDisable(GL_ALPHA_TEST);
glDisable(GL_DEPTH_TEST);
glDisable(GL_CULL_FACE);



		q_vertexType afVertices[] = {
		    f2vt(0.0), f2vt(0.0), f2vt(1.0), f2vt(0.0),f2vt(0.0),f2vt(1.0),f2vt(1.0),f2vt(1.0)
		};
		glEnableClientState(GL_VERTEX_ARRAY);
		glVertexPointer(2,q_vertexTypeEnum,0,afVertices);

		GLfloat tw = GLfloat(buffer_width / TextureSizeW);
		GLfloat th = GLfloat(buffer_height / TextureSizeH);

		q_vertexType afTexCoord[] = {
// remember to scale your tex coord according to 400/512  240/512 value
		    f2vt(0.0f),f2vt(0.469f), f2vt(0.781f),f2vt(0.469f), f2vt(0.0f),f2vt(0.0f), f2vt(0.781f),f2vt(0.0f)
		
			//f2vt(0.0f),f2vt(th), f2vt(tw),f2vt(th), f2vt(0.0f),f2vt(0.0f), f2vt(tw),f2vt(0.0f)

		};
		glEnableClientState(GL_TEXTURE_COORD_ARRAY);
		glTexCoordPointer(2,q_vertexTypeEnum,0,afTexCoord);
   glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

glDisableClientState(GL_TEXTURE_COORD_ARRAY);
glDisableClientState(GL_VERTEX_ARRAY);


glEnable(GL_BLEND);
glEnable(GL_ALPHA_TEST);
glEnable(GL_DEPTH_TEST);
glEnable(GL_CULL_FACE);
glMatrixMode(GL_MODELVIEW);
glPopMatrix();
glMatrixMode(GL_PROJECTION);
glPopMatrix();

}

in my case it runs at around 15 fps, already happy about it! all credit goes to Klen!

Enjoy!
Previous Post under:=================


One question for curiosity and confirmation of my understanding:

I notice in your project you have a drawBackground() and is by default disabled, this is what I need in AR study, unfortunately when I enable it the drawing is correct but frame rate drop to 2 or 3 fps.

when I tap "show video stream" in app, it update the texture of the rotating cube, but then keep rendering with that frame, it is no "live stream" on my cube, is it the expect result?

I try to use
glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB, buffer_width,buffer_height, 0, GL_RGB, GL_UNSIGNED_BYTE, data );

as in refresh_texture and rend a quad without using QImage, it however does not work so far ( I use opengl es 1 version)


Have you correct this in similar way, meaning have you get these two effect: "live stream" on your rotating cube or use "rendering background as texture instead of paint QImage"
thanks for tip.

ZR_YAO

[Klen]

I notice in your project you have a drawBackground() and is by default disabled, this is what I need in AR study, unfortunately when I enable it the drawing is correct but frame rate drop to 2 or 3 fps.

This was just a development step so that I could make sure that the images are correctly decoded into QImage structures in order to eliminate the problem of texturing. This function is very slow because it uses QPainter to paint each frame to the screen instead of using the OpenGL textures. However, when drawing less complicated stuff than full images (i.e. text, squares, small images), like the white border and frame rate text in paintGL() function, this does not affect the perforamnce.

when I tap "show video stream" in app, it update the texture of the rotating cube, but then keep rendering with that frame, it is no "live stream" on my cube, is it the expect result?

Yes. This is the right response. It was just part of my development. This app is just a snipet app not a finished application.

[stone.chao]

Dear Klen:
You are so kind to share your excellent work. Thanks!

May I ask if your N900 already upgraded to PR1.2?
Since I copy DistanceQT in the tar file to my N900 and also do 'apt-get install libcv4 libcvaux4 libhighgui4', the installation is fine.
But when I execute DistanceQT in xrem, there is nothing happened.
Nor application neither error message shows.
I am wondering if I miss something on N900...

Thanks anyway, you already save many time for us.

[Klen]

Dear Klen:
You are so kind to share your excellent work. Thanks!

May I ask if your N900 already upgraded to PR1.2?
Since I copy DistanceQT in the tar file to my N900 and also do 'apt-get install libcv4 libcvaux4 libhighgui4', the installation is fine.
But when I execute DistanceQT in xrem, there is nothing happened.
Nor application neither error message shows.
I am wondering if I miss something on N900...

Thanks anyway, you already save many time for us.

To run the app you need QT 4.6. You get it with the new frameware PR1.2. You can flash your device to get the latest frameware if your device does not have one. To check if you have the right QT version do:

Nokia-N900:~# dpkg -l|grep libqt
ii  libqt4-core                                         4.6.2~git20100401-0maemo1+0m5              Qt 4 core module
ii  libqt4-dbus                                         4.6.2~git20100401-0maemo1+0m5              Qt 4 D-Bus module
ii  libqt4-gui                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 GUI module
ii  libqt4-maemo5                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 Maemo5 module
ii  libqt4-multimedia                                   4.6.2~git20100401-0maemo1+0m5              Qt 4 Multimedia Libraries
ii  libqt4-network                                      4.6.2~git20100401-0maemo1+0m5              Qt 4 network module
ii  libqt4-opengl                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 OpenGL module
ii  libqt4-phonon                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 Phonon Libraries
ii  libqt4-sql                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 SQL module
ii  libqt4-sql-sqlite                                   4.6.2~git20100401-0maemo1+0m5              Qt 4 SQLite plugin
ii  libqt4-svg                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 SVG module
ii  libqt4-webkit                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 WebKit module
ii  libqt4-xml                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 XML module
ii  libqt4-xmlpatterns                                  4.6.2~git20100401-0maemo1+0m5              Qt 4 XML Patterns module

Flashing N900:

1.) get latest version for here
http://tablets-dev.nokia.com/nokia_N900.php

2.) Turn off your device and disconnect USB

3.)
run the flasher:
$sudo flasher-3.5 -F <firmware-image> -f -R

NOTES: More info
http://wiki.maemo.org/Updating_the_tablet_firmware

[divanov]

To check if you have the right QT version do:

dpkg -l "libqt4-*"

[zr_yao]

Since I copy DistanceQT in the tar file to my N900 and also do 'apt-get install libcv4 libcvaux4 libhighgui4', the installation is fine.
But when I execute DistanceQT in xrem, there is nothing happened.
.

Just help to clarify a bit here. Unless Klen has updated the package, as I remember, the original DistanceQt package does not use OpenCV at all, it use Gstreamer for the camera frame capturing, Klen suggest to use OpenCV to "avoid Gstreamer hassle" which I fully agree with, to use opencv you can simple use cvcapture to get the camera frame, it in fact in turn use its own gstreamer library built in the highgui library, so Opencv help to wrap the gstreamer pipeline for you already.

That says if you like to incooperate your opencv into the DistanceQt, you can strip away all the gstreamer pipeline codes. I would guess the performance should be the same as Klen's version (I have not done any test) but use your own pipeline gives a lot more flexibility.

A simple way to verify Opencv way is to copy sample/c directory from Opencv to your ARMEL target and do a build_all, you can simply run i.e LKdemo, it works out of box.

[Dorfmeister]

Hi,

@zr_yao: Could you briefly elaborate on how you avoided a conflict with GLES2 regarding glTexImage2D() as described in http://talk.maemo.org/showthread.php?t=53328? How would the GLES2 version look like?

I'm currently trying to plot simple points into the image. To achieve this I inserted

for(int i=0; i<ret_num_corners; i++){
   painter.drawLine(result[i].x,result[i].y,result[i].x,result[i].y);
}

into the painter part of GLWidget::paintGL(). I checked the array and the right points are in, but it doesn't show anything. An alternative version using

glBegin (GL_POINTS);
  for (i = 0; i < ret_num_corners; i ++){
     glVertex2f (result[i].x, result[i].y);
  }
glEnd ();

doesn't work since Qt can't find GL/gl.h, which it previously recommended by itself. Any ideas how to achieve this rather simple task?

Again: How difficult is it to port zr_yao's work to GLES2? Any recommendations how to get started with GLES2?

Thanks in advance.


Best regards,
Dorfmeister

[divanov]

doesn't work since Qt can't find GL/gl.h, which it previously recommended by itself. Any ideas how to achieve this rather simple task?

GL/gl.h is OpenGL header, while Maemo is using OpenGL ES2 and emulates OpenGL ES.
Thus include should be probably GLES2/gl2.h

[gnuton]

Hi,
you can start to read http://wiki.maemo.org/OpenGL-ES then you can can take a look at these examples: http://www.imgtec.com/powervr/inside...nGLES2xSGX.asp
If you want a book you could take a look at "OpenGL ES 2.0 Programming Guide". The examples are downloadable from the author website but those doesn't compile in scratchbox.
You can find a good book review at http://developers.slashdot.org/artic.../10/06/1313224

[gnuton]

Right it's "GLES2/gl2.h"

[zr_yao]

Again: How difficult is it to port zr_yao's work to GLES2? Any recommendations how to get started with GLES2?

I'm a bit confused here, The package in post #1 and #2 use gles2, I add port of gles1 for convenience, why you say port to gles2? if you just mean to write the drawbackground() function in gles2, then it should be trival but I did not do it since again I use gles1.

In my case my N900 by default use gles2, so I can run i.e. hellogl_es2, not hellogl_es1, by reinstall my own build of qtgl I revert it to gles1 and did my work there, I lose performance in fact, so I suggest you to use DistanceQt (the original package in this thread)directly from gles2. I did porting simply avoid the shader programming that I'm too lazy to learn.

Hope that helps
BR/ zr_yao

[Dorfmeister]

Oh, dear. I really got confused - thanks for clarifying the matter. I bought the book you recommended and will dive into it soon. However, I'm still confused that my conventional OpenGL approach to plot points didn't work out. If anybody should know right away what mistake I made in my attempt and how to draw points from an array properly (probably drawLine methods), a short hint would be appreciated. Otherwise I'll try to figure it out myself as soon as I find time.


Best regards,
Dorfmeister