How to set up QCamera resolution when working with a QVideoRendererControl (N950)

[konstantint]

Hello,

I am streaming raw frames from a QCamera on a N950 using the following code:

cam = new QCamera();
QMediaService *m = cam->service();
QVideoRendererControl* vrc = m->requestControl<QVideoRendererControl*>();
vrc->setSurface(myVideoSurface);

where myVideoSurface is a subclass of QAbstractVideoSurface which seems to be doing fine obtaining frames one by one in its present() method.

I have a couple of questions:
* Is the above method indeed the "proper" way to access raw video frames? I managed to end up with it via endless googling/trial and error, so I am not sure. Is there in fact some piece of documentation somewhere that explains this explicitly?

* How can I change the camera's resolution/aspect ratio? I have a hunch this might be related to QVideoEncoderControl but so far attempts to use it to change just the resolution resulted in strange results. I.e. if I do something like

QVideoEncoderControl *enc = m->requestControl<QVideoEncoderControl*>();
QVideoEncoderSettings sets = enc->videoSettings();
sets.setResolution(500, 500);
enc->setVideoSettings(sets);

my video surface indeed ends up receiving frames which are sized at 500x500, but if I decode and display them I get weird lines instead of proper frames. What is the right way of changing camera resolution and again, is there any good place documenting this?

Thanks,
Konstantin.

[gnuton]

Hi,
Are you sure you are using the right pixel format to decode the buffer?

[konstantint]

Well, the pixel format seems to be UYVY in all cases, I've borrowed the decoding procedure from here and things work fine unless I try to mess with QVideoEncoderControl. It is possible that I am doing this wrong too, indeed.

Hence I'd like to start with "theory". Is QVideoEncoderControl the right tool to go switching camera resolution in the first place? Maybe it is not "encoding" that I am willing to change but some low-level camera sensor characteristic related to how its CCD actually works, not how the resulting video is "encoded"?

Another funny thing is that if I set up QCamera into CaptureStillImage capture mode, the default resolution it provides to the videosurface is 640x480, but if it is CaptureVideo, the resolution is 1280x720. Where do these numbers come from and why it is the case, any ideas?

[meandnano]

Well, at the end of your present() method in the video surface you should have a QPixmap in RGB16 format. You can easily change it's aspect ration by calling QPixmap->scaled() before proceeding with this frame. didn't you try that?

[konstantint]

Well, yes, I can change ratios and stuff, but all this will happen to the already captured frame which was provided to me by the camera at, say, 640x480 resolution.
Presumably, the camera can do better (it is 8megapixels, isn't it?). So the question is - how do I tell the camera to send me high or low quality video at certain aspect ratios, more or less how the built-in camera app does it?

[gnuton]

Here is the implementation of Mobility Camera GStreamer backend which we use in harmattan.
When you modify QVideoEncoderSettings you actually changes parameters in the "video-encoder" GStreamer element (line 262 of [1]).

Resolutions settings come from GStreamer too. [2]
[1] https://qt.gitorious.org/qt-mobility...ideoencode.cpp
[2] http://gstreamer.freedesktop.org/dat...r-GstCaps.html

[konstantint]

<deleted post in favour of a more concise summary in the next one>

[konstantint]

OK, I think I have figured things out now. Thanks a lot guys. Here's the summary.

* Although I don't know a thing about gstreamer, I browsed around the code and got the idea that, in theory, using VideoEncoderControl to change camera resolution is indeed the right thing to do. I think that was what gnuton was trying to tell me.
* In my case I was getting a mingled picture because I was providing an unsupported resolution.
* To check for supported resolutions, I need to use videoEncoderControl.supportedResolutions(), yet the problem is that, if this method is called too early, it returns a confusing empty list.
* After playing around with all the possible QCamera signals and other places, I finally found a spot where supportedResolutions() did start to respond - the start() method of MyVideoSurface.
* This is somehow ironic that you need to set a correct resolution before you can start to receive video, but you will only get to know which are supported resolutions after you have started to receive video. Hence, for future reference of whoever comes to this thread, here are the resolutions for N950 back camera as reported by supportedResolutions().
QSize(252, 188), QSize(320, 240), QSize(352, 288), QSize(640, 480), QSize(848, 480), QSize(854, 480), QSize(1280, 720).
Most of those work fine, but not all. Attempting to use the first resolution leads to "CameraBin error: could not negotiate format".

Now here's the last, less important but perhaps an easy-to-answer question. Do I understand correctly, that if I really want to get a 8MP image I must use QCameraImageCaptureControl with its capture(filename) method and then read in the resulting file manually? Is there a shortcut, where I do not have to do the "save file - load file" procedure and the camera would write to some memory buffer directly?

Thanks!

[bokovoy]

OK, I think I have figured things out now. Thanks a lot guys. Here's the summary.

* Although I don't know a thing about gstreamer, I browsed around the code and got the idea that, in theory, using VideoEncoderControl to change camera resolution is indeed the right thing to do. I think that was what gnuton was trying to tell me.
* In my case I was getting a mingled picture because I was providing an unsupported resolution.
* To check for supported resolutions, I need to use videoEncoderControl.supportedResolutions(), yet the problem is that, if this method is called too early, it returns a confusing empty list.
* After playing around with all the possible QCamera signals and other places, I finally found a spot where supportedResolutions() did start to respond - the start() method of MyVideoSurface.
* This is somehow ironic that you need to set a correct resolution before you can start to receive video, but you will only get to know which are supported resolutions after you have started to receive video. Hence, for future reference of whoever comes to this thread, here are the resolutions for N950 back camera as reported by supportedResolutions().
QSize(252, 188), QSize(320, 240), QSize(352, 288), QSize(640, 480), QSize(848, 480), QSize(854, 480), QSize(1280, 720).
Most of those work fine, but not all. Attempting to use the first resolution leads to "CameraBin error: could not negotiate format".

Now here's the last, less important but perhaps an easy-to-answer question. Do I understand correctly, that if I really want to get a 8MP image I must use QCameraImageCaptureControl with its capture(filename) method and then read in the resulting file manually? Is there a shortcut, where I do not have to do the "save file - load file" procedure and the camera would write to some memory buffer directly?

Thanks!

Sensor supports different streaming modes. Some of those employ complex data transfer paths and high resolution ones are not allowing to achieve frame rates needed for practical video recording. Hardware resizer, for example, has certain limitations for maximum possible resolution to be processed in one pass. Thus, behind the scenes sensor is configured by video source element in GStreamer to perform with those limitations on resolutions that are making sense from optimal bandwidth and image quality use. The limitations on what modes are available are coming from multiple sources, thus it is not always possible to derive and define them statically. Thus, that ironic effect of required knowledge of a correct resolution prior to questioning the hardware.

As result, you can get 8MP single capture but for video streaming there is limit of 720p (1280x720) set to allow best performance. You theoretically also can get the sensor 8MP data without going through JFIF compression (on DSP in default camera pipelines) but then there are certain memory bandwidth limits when frame rates are concerned. For simplified use cases that QCamera API represents, what you get is what you can get without giving more ugly details.