Using camera frame as OpenGL texture, problem!!

[abolfoooud]

Hi,

I am working on an app where i want to get a frame from the camera on the phone and bind it as a tecture on a polygon (square). after alot of tweeks i managed to have this done but it was too expenssive in terms of performance (draw part of the frame into a bitmap using CFbsBitGc:rawBitmap and then use that bitmap as my texture. This requires alot of switching between the GPU and the CPU). I am trying instead to get the bitmap raw data and create the data array manually. However i am having a problem here.

The bitmap i receive from the camera APIs has a display mode of EColor16MU and that does not work for the texture functions in OpenGL (they expect EColor16M only). I always get corrupted bitmap on the polygon when i take the frame from the camera straight away and use it as my tecture. I tried to change the displaymode using the SetDisplayMode() function that worked slightly better but not as it should (it gave me black and white color with horizontal colored lines every 5 or 6 lines). i believe there is something i am not getting right about the distribution of RGBA channels with EColor16MU.

I know that with EColor16M the layout of the channels is BGR instead of RGB. i tried treating the EColor16MU as UBGR rather than RGBU (U as unused byte) but still with no success.

Any body knows how can i work around the EColor16MU mode and use it as my texture map?

thanx in advance
AF

[miguelbl]

There is an example in simpleCubeTexture nokia example.

What you should do is to fill an array with RGB values byte per byte and not pixel per pixel.

So, you get the URGB, discard the U and copy R, G and B to the right place.

It works.

[abolfoooud]

Thanx for the reply.

this does not work because it is what i am exactly doing. The problem is with the format of the data array returned by DataAddress. since the exact layout of bytes is not known there will always be a distortion in the retreived channels.

So if i read Byte by byte i might have read an unused byte from the resturned data array. 16MU will always have that unused byte. 16M will not. So what i need is either to set the mode of the bitmap to be captured by the camera initially or to know how the RGBU of the color are arranged with 16MU. i tried several combinations but none worked. so i gues i am missing something here.

I will be looking at changing the format of the camera frame bitmap as i read somewhere in teh forum that it is possible.

cheers
AF

[miguelbl]

In My experience I don´t thing this can be done or that you are getting any benefit from it.

Just create an array with 3bytes per pixel lenght and copy the Image in the right format bit by bit.

There is an example in the cube with texture and also in the 3D example that tells you how. Also in the thread "converting CFbsBitmap to TBitmap" this conversion is done.

By the way, the textures need to be Power Of Two, while the camera images usually are not, so it means that you would need just a part of the image or to fill some parts of the array properly in order to get a POT texture from the camera frames. I have it working on one of my applications so it is possible to be done (although maybe there is a more straightforward and fast way).

[olhed]

The color channels are URGB, highest bits are unused.

If you want any kind of performance you really should try avoid any kind of extra copying and try feed the texture directly from CFbsBitmap to the GL texture.

Uploadnig texture every frame as you plan to do will have quite bad performance anyhow.

Don't GL support stride and padding between pixels when feeding textures to it?

Don't see why you would need an extra copy...

If the color formats don't match, make sure that the CFbsBitmap is generated in the right color mode from the beginning, DONT convert it every frame...

[abolfoooud]

olhed,

If the color formats don't match, make sure that the CFbsBitmap is generated in the right color mode from the beginning, DONT convert it every frame...

this is exactly what i want to achieve. as i am getting the frames (texture) from the camera the mode is always 16MU. I want to change this setting in teh camera but not sure how, and not sure if it is possible.

miguelbl,

"converting CFbsBitmap to TBitmap"

i think this thread is so useful. i skimmed through it very quickly and i guess it will be of a help.

By the way, the textures need to be Power Of Two, while the camera images usually are not, so it means that you would need just a part of the image or to fill some parts of the array properly in order to get a POT texture from the camera frames.

I had a work around this that is what i am trying to optimize to be frank with u.

Thanx for the help
AF

[olhed]

Looking more carfefully on the current APIs it seems you will have to make a conversion (not good )

I would recommend creating ARGB pixels from the URGB instead though, and load those into the glTexImage2D.

That should be a lot quicker then going via RGB pixels on ARM hardware

[abolfoooud]

I would recommend creating ARGB pixels from the URGB instead though, and load those into the glTexImage2D.

can you be more illustrative please?
thanx
AF

[miguelbl]

Looking more carfefully on the current APIs it seems you will have to make a conversion (not good )

I would recommend creating ARGB pixels from the URGB instead though, and load those into the glTexImage2D.

That should be a lot quicker then going via RGB pixels on ARM hardware

It is not possible to load ARGB pixels to OPENGL ES.

You need pure RGB pixels as said on openGL ES specification. There is no other way than copy them as said on the other thread. I know it is slow, but it is what we have!

[Flame]

The native texture format of graphics HW is typically not a linear format. It is either blocked or "twiddled" or somehow specifically organized to avoid memory access inefficiencies.

This format is HW specific and any texture that is fed from client API needs to be in any case converted to this HW specific format before it can be used as a texture in any rendering calls.

There is no fast path to use texture "as is". Even if the particular client format that the application supported would be present in the API, the conversion would probably be done in any case to HW specific format (in "optimal" case it is a full memcpy() of the texture. cannot be any better than that).

The only way in case of Symbian bitmaps is to convert the source bitmap to the OpenGL ES supported format (can sometimes be done directly in-place of the bitmap store), and then call the ES API to upload the texture.

This has the drawback that there are now two conversion steps (one in client, one in driver), but typically that in-place byte-swap is a lot faster than the actual conversion to HW specific format.

Typical usage pattern for texture mapping in games is to first load textures and then use them many times over thousands of frames. This kind of dynamic streaming of texture data can easily go outside of that optimal case. In those cases, ideally one should have a direct streaming of camera data to a 3D HW texture (to the HW format) facilitated by a custom API to do this (which is not available).

Jani Vaarala
Graphics Architect
Nokia

[olhed]

It is not possible to load ARGB pixels to OPENGL ES.

You need pure RGB pixels as said on openGL ES specification. There is no other way than copy them as said on the other thread. I know it is slow, but it is what we have!

Last time I looked, glTexImage2D accepted the GL_RGBA specifier for loading textures, so no, you don't have to use 24 bit GL_RGB.

Though, I forgot about that symbian bitmaps are really BGRA in memory. (because of byte order, if you read it into a 32 bit register, it will be ARGB, meaning A is highest bits)

Therefore, a channel reordering is needed, as Jani point out above.

BUT! no need to do this slower then needed, you should do it with 32 bit read/writes and not byte read/writes.
one read and one write per pixel, instead of one read and 3 writes. (or even worse, as in the example in the other thread, 3 reads and 3 writes per pixel)

[miguelbl]

Ok, you are right.

You can avoid accessing 3 times to the memory per pixel and probably you can do it with masks and one access, but anyway you should reorder every pixel, so I don´t know how much faster it will be knowing that you will always need to copy all the data anyway. I will try the BGRA to ARGM approach and see what happens.

I just went the other way because Nokia examples were like that. copying to RGB using GL_RGB instead of ARGB.

[abolfoooud]

thanx chaps for your hefty contributions

Flame,

In those cases, ideally one should have a direct streaming of camera data to a 3D HW texture (to the HW format) facilitated by a custom API to do this (which is not available).

we will be waiting patiently for this

olhed and miguelbl,

You can avoid accessing 3 times to the memory per pixel and probably you can do it with masks and one access

is a clever idea!! never thought of it. a little bit of optimization i believe is better than nothing

cheers
AF

[freesui1984]

Hi,

i also want to use the frame grabbed from in-built camera as a texture.i have get the frame from ViewFinderFrameReady function. And i transform the frame format(may be RGB565)to RGB888, and then in the container.cpp:

glTexImage2D(GL_TEXTURE_2D,0,3,256,128,0,GL_RGB,GL_UNSIGNED_BYTE,iBitmap->DataAddress());
    TSize size;
    size=this->Size();
    //set the view port
    glViewport( 0, 0, size.iHeight, size.iWidth );

    // Recalculate the view frustrum
    glMatrixMode( GL_PROJECTION );
    glLoadIdentity();
    GLfloat aspectRatio = (GLfloat)(size.iWidth) / (GLfloat)(isize.iHeight);
    glFrustumf( FRUSTUM_LEFT * aspectRatio, FRUSTUM_RIGHT * aspectRatio,
                FRUSTUM_BOTTOM, FRUSTUM_TOP,
                FRUSTUM_NEAR, FRUSTUM_FAR );
    glMatrixMode( GL_MODELVIEW );
    glLoadIdentity();

i want to draw a rectangle whose size is equal to the screen. And then i map it texture i defined.
But i find function glTexCoord2f and glVertex3f are not known. So how can i do that? could you give me some right codes?

Looking forward to fast replying. Thank you
Mia

[abolfoooud]

Right, you can do the following and i hope it helps.
Note that u pass the pointer returned by DataAddress() to function RenderTexturedBG below.

cheers
AF

//==========================================
//	BACKGROUND PLANE
static const GLbyte bgverts[8] = 
	{ 
        0, 0,
        1, 0,
        0, 1,
        1, 1
	};
 
static const GLubyte bgtex[8] =
	{
	0, 1,
	1, 1,
	0, 0,
	1, 0
	};


void DrawBGPoly()
    {
    glPushMatrix();
	    // Enable vertex arrays.
	    glEnableClientState( GL_VERTEX_ARRAY );
	    // Set array pointers.
	    glVertexPointer( 2, GL_BYTE, 0, bgverts );
	    // Enable texture arrays 
	    glEnableClientState( GL_TEXTURE_COORD_ARRAY );
	    // Set texture coords
	    glTexCoordPointer( 2, GL_BYTE, 0, bgtex );
	    glEnable(GL_TEXTURE_2D);
	    glBindTexture( GL_TEXTURE_2D, iTexturesID[0] );
	    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
	    glDisable(GL_TEXTURE_2D);
	    glDisableClientState(GL_TEXTURE_COORD_ARRAY);
	    glDisableClientState(GL_VERTEX_ARRAY);
    glPopMatrix();
    }


void PerspectiveMode()
    {
    // Load our projection matrix
    glMatrixMode( GL_PROJECTION );							
    glEnable( GL_DEPTH_TEST );
    glEnable( GL_CULL_FACE  );
	
    // Get rid of ortho mode which was lst pushed in the projection matrix mode
    glPopMatrix();											
    // Go back to model view matrix
    glMatrixMode( GL_MODELVIEW );
    }


void OrthoMode(TInt aLeft, TInt aRight, TInt aBottom, TInt aTop )
    {
    // Switch to our projection matrix so that we can change it to ortho mode, not perspective.
    glMatrixMode(GL_PROJECTION);						
    // Push on a new matrix so that we can just pop it off to go back to perspective mode
    glPushMatrix();			

    glDisable( GL_DEPTH_TEST );
    glDisable( GL_CULL_FACE  );
   
    // Reset the current matrix to our identify matrix
    glLoadIdentity();								
    // Pass in our 2D ortho screen coordinates (left, right, bottom, top, near, far).  
    glOrthof( aLeft, aRight, aBottom, aTop, -1, 1 );	
	
    // Switch to model view so that we can render the texture
    glMatrixMode(GL_MODELVIEW);								
    // Initialize the current model view matrix with the identity matrix
    glLoadIdentity();
    }


void RenderTexturedBG(TUint8* aCamFrameData)
    {
    // Textures are initialized in OpenGL ES API by this function.
    glGenTextures( 1, iTexturesID );	
    /* Bind the background texture to iTexturesID[0], 
       set the texture environment. */
    glBindTexture( GL_TEXTURE_2D, iTexturesID[0] );
    glTexImage2D(  GL_TEXTURE_2D, 0, GL_RGB, 256, 128, 
		   0, GL_RGB, GL_UNSIGNED_BYTE, (GLvoid*)aCamFrameData );
    glTexEnvx( GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE );
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
		    
    // Before drawing the background texture, we need to switch to ortho (2D) mode.  
    // Passing 1 instead of the implicit size of screen will make OpeGL use the full size of screen (1:1)
    OrthoMode(0, 1, 0, 1);	
    DrawBGPoly();
    PerspectiveMode();
		    
    glDeleteTextures(1, iTexturesID);
    }