Drawing On Top Of Video Stream (QT & gstreamer)

[Klen]

I am trying to build a simple augmented reality application that will overdraw graphics over a video stream. I achieved that by capturing video frames using gstreamer appsink (how to), converting them to QImage objects that can be drawn with QPainter onto QWidget.

As the appsink new_buffer callback is a static callback function, it can not trigger a repaint. In order to repaint frames, I created a QTimer that ticks 30 times a second and calls a repaint method. This puts a lot of stress on the processor and I also have problems with random crashes that come from the paintEvent function.

Questions:
1.) Is this the way one should implement drawing over a video stream in QT? Is it possible to trigger a repaint from a static function?
2.) Why is the application's performance so bad? It loads the processor almost to the maximum at the video resolution of 320×240? At this resolution the application achieves 30FPS, however, if I converting QImage to QPixmap (this fixed the crashing bug) the performance drops to 12 FPS. If I disable the repaint timer (drawing of frames on the screen), I get very little processor activity even at 640×480, which suggests the sink and conversion to QImage is not to blame.
3.) Why does my application crash when I use QImage and works OK when I use QPixmap;

//callback function of appsink (Gstreamer thread) 
void CameraN900::new_buffer (GstAppSink *_appsink, gpointer user_data)
{
	if(cold_start){
		//initialize appsink
		appsink=_appsink;
		gst_app_sink_set_drop(appsink,true);
		gst_app_sink_set_max_buffers(appsink,2);
		cold_start=false;
	}

	GstBuffer* buffer = gst_app_sink_pull_buffer(appsink);
	image_data = (unsigned char *) GST_BUFFER_DATA (buffer); //image_data is a static unsigned char *
	CameraField::createImage();
	gst_buffer_unref(buffer); //If I comment out this the application stops crashing, but fills up the memory
}

//static function  
void CameraField::createImage()
{
	*image_buffer=QImage(CameraN900::image_data, buffer_width, buffer_height,3*buffer_width, QImage::Format_RGB888); // loads data into an buffer_image
}

//GUI thread. Repinat triggered by a timer.
void CameraField::paintEvent(QPaintEvent * event)
{
	QPainter painter(this);
	painter.drawImage(0,0,*image_buffer);
}

Thanks,
Klen

[Klen]

Update on the thread:

Question 3 solved:
The application does not crash any more. Instead of calling gst_app_sink_pull_buffer in a new_buffer callback function, I call it inside the paintEvent. In this way all is done in the same thread and I avoid unreferencing the buffer before the image is drawn on the screen, however, the performance drops slightly (from 30 FPS to 25 FPS at 320×240).

1 and 2 remaining.

Any help will be meat by great joy and happiness that will fill the room of my office .

Klen

//callback function of appsink (Gstreamer thread) 
void CameraN900::new_buffer (GstAppSink *_appsink, gpointer user_data)
{
	//initialize appsink and disable emit new_buffer signal
}

//static function  
void CameraField::createImage()
{
	*image_buffer=QImage(CameraN900::image_data, buffer_width, buffer_height,3*buffer_width, QImage::Format_RGB888); // loads data into an buffer_image
}

//static function
void CameraN900::refresh_buffer(){ 
	if(appsink!=NULL){
		buffer = gst_app_sink_pull_buffer(appsink); //buffer us a static GstBuffer *
	}
}

//GUI thread. Repinat triggered by a timer.
void CameraField::paintEvent(QPaintEvent * event)
{
	CameraN900::refresh_buffer(); //manually call appsink, not through the callback function

	if(CameraN900::buffer!=NULL)
	{
		unsigned char* data=(unsigned char *) GST_BUFFER_DATA (CameraN900::buffer);
		QImage img(data, buffer_width, buffer_height,3*buffer_width, QImage::Format_RGB888); // 2 pixels width, 2 pixels height, 6 bytes per line, RGB888 format
		QPainter painter(this);
		painter.drawImage(0,0,img);
		gst_buffer_unref(CameraN900::buffer);
	}
}

[divanov]

What you probably want to use is to replace appsink with xvimagesink (and drop ffmpegcolorspace) http://www.gstreamer.net/data/doc/gs...imagesink.html and tell xvimagesink to render video frames on your X window. Simultaneously you can draw on the same window and your drawing will be visible on top of the video.

[Klen]

What you probably want to use is to replace appsink with xvimagesink (and drop ffmpegcolorspace) http://www.gstreamer.net/data/doc/gs...imagesink.html and tell xvimagesink to render video frames on your X window. Simultaneously you can draw on the same window and your drawing will be visible on top of the video.

Hello,

Thanks for your replay. I tried this already.

I managed to created an xvimagesink, linked it to my bas QWidget using gst_x_overlay_set_xwindow_id() and managed to thrown some PushButtons over it. The problems started, when I draw on top of base widget by adding another custom widget that draw a red square with the alfa value of 0.56. Instead of blending the square with the video stream, it blended it with the background color of the base widget (it was yellow so the result was orange). In case I created an empty custom widget I saw only the background color of the base widget instead of the video stream. If tried solving this problem by setting WA_OpaquePaintEvent flag to base widget, but ended up with black background.

App Screen Shot:

(Draw a red rectangle on a yellow base widget (bledning gives you orange). I also set a widget mask. It is blending with background instead of video stream)

I guess the problem is that video stream as the xovelray was not threated as a background of the widget and therefore wasn't being double buffered and redrawn bellow the transparent top widget. However, If I hide the widget from the screen the video stream redraw the previously covered area. The flicker of the screen was noticed.

Question:
How to make xoverlay act as a background of a widget that enables you to blend or at least overdraw graphics with transparent background.

Thank you very much for your help again. Pleas let me know if I can help you in any way except for stop asking for help on this forum .

Cheers,
Klen

[Klen]

I read on this post (wysota reply ) that the most optimized method for redrawing a video stream would be to use textures in OpenGL. I already tried it (new thread). It achieves better performance, but is still not fast enough.

Any other suggestions?

Tahnks,
Klen