Access camera viewfinder data
This article shows how access the raw image data on the camera viewfinder using Qt.
Article Metadata
Code Example
Tested with
Compatibility
Platform Security
Article
Contents |
Introduction
QML has a basic and easy to use camera component, as shown in the QML Camera example. Unfortunately in Qt 4.x this component does not provide any way to process the view finder frames while the camera is running.
This article explains you how to access raw viewfinder data from the camera on your Symbian-device using Qt's C++ Camera API. We then show how to integrate with the QZXing barcode scanner library in order to process QR codes straight from the viewfinder. The techniques demonstrated can be used for any number of other purposes, including manipulating captured frames prior to display.
The principles described also apply to MeeGo Harmattan, albeit with some changes required to support the device's native pixel formats, as explained in the article MeeGo Camera VideoSurface manipulation.
Accessing camera viewfinder data from Qt
To get access to viewfinder frames you need to create a QAbstractVideoSurface subclass that you set as the viewfinder for your QCamera instance. Then in your videosurface class you get access to the QVideoFrames coming from the camera.
Subclassing QAbstractVideoSurface
Create a class that inherits QAbstractVideoSurface and implement the abstract methods:
virtual QList<QVideoFrame::PixelFormat> supportedPixelFormats(
QAbstractVideoBuffer::HandleType handleType = QAbstractVideoBuffer::NoHandle) const = 0;
virtual bool present(const QVideoFrame &frame) = 0;
In the supportedPixelFormat()-method you return the pixel format (usually QVideoFrame::Format_ARGB32) that you support. The present()-method is called when you have set the surface to your QCamera as a viewfinder and you start getting frames. Here you map the image data from a QVideoFrame to your own QImage for processing.
See VideoSurface in CamTest for an example.
Creating a QML-element for the viewfinder
To get the viewfinder to also show on the screen we wrap it inside a small custom QML element. This is done by creating a QDeclarativeItem subclass and adding a QCamera instance and your newly created QAbstractVideoSurface subclass instance as it's members. Then you override the paint() method to draw the frames received from the surface to the QML element.
See ViewFinderWrapper in CamTest for an example.
Setting the surface to QCamera
Inside your QML viewfinder element you need to initialize a QCamera instance normally and then set your own surface implementation as it's viewfinder.
m_camera = new QCamera(this);
if (m_camera) {
connect(m_camera, SIGNAL(stateChanged(QCamera::State)), this, SLOT(onStateChanged(QCamera::State)));
connect(m_camera, SIGNAL(statusChanged(QCamera::Status)), this, SLOT(onStatusChanged(QCamera::Status)));
connect(m_camera, SIGNAL(error(QCamera::Error)), this, SLOT(onCameraError(QCamera::Error)));
m_camera->start();
m_cameraActive = true;
m_camera->setViewfinder( static_cast<QAbstractVideoSurface*>( &m_surface ) );
m_viewFinderActive = true;
emit runningChanged();
}
See ViewFinderWrapper.cpp in CamTest for an example.
Using the QML-viewfinder
In your application's main.cpp register your newly created QML element so that it can be used in your QML files:
#include "viewfinderwrapper.h"
qmlRegisterType<ViewFinderWrapper>("Codemancers", 1, 0, "ViewFinder");
See main.cpp in CamTest for an example.
Then you can use the viewfinder in your QML files:
ViewFinder
{
id: viewFinder
width: parent.width
height: width*3/4
Component.onCompleted: {
console.debug("ViewFinder QML-component loaded");
}
}
Adding a thread for processing
It is good practice not to block the UI-thread with heavy operations so the final step is to add a thread to do the actual processing of the frames. This is done by subclassing QThread and implementing the run() method:
#ifndef PROCESSINGTHREAD_H
#define PROCESSINGTHREAD_H
#include <QThread>
#include <QImage>
#include <QQueue>
class ProcessingThread : public QThread
{
Q_OBJECT
public:
explicit ProcessingThread(QObject *parent = 0);
virtual ~ProcessingThread();
signals:
void frameProcessed();
void queueFull();
public:
void stop();
void addFrameToProcessingQueue(QImage frame);
private:
virtual void run();
private:
QQueue<QImage> m_queue;
int m_queueMaxLength;
bool m_stopped;
};
#endif // PROCESSINGTHREAD_H
We added a method to add frames to the queue for processing so we have a small buffer if some frames take longer to process than others. We also set the maximum length for the queue and when it is reached we start skipping frames to catch up.
#include "processingthread.h"
#include <QDebug>
static const int QUEUE_MAX_LENGTH = 3;
static const int THREAD_SLEEP_MS = 25;
ProcessingThread::ProcessingThread(QObject *parent) :
QThread(parent), m_stopped(false), m_queueMaxLength(QUEUE_MAX_LENGTH)
{
}
ProcessingThread::~ProcessingThread()
{
stop();
}
void ProcessingThread::stop()
{
m_stopped = true;
}
void ProcessingThread::addFrameToProcessingQueue(QImage frame)
{
if (m_queue.length() < m_queueMaxLength) {
QImage threadCopy = frame.copy();
m_queue.enqueue(threadCopy);
} else {
emit queueFull();
}
}
void ProcessingThread::run()
{
// Process until stop() called
while (!m_stopped)
{
if (!m_queue.isEmpty())
{
QImage currentFrame = m_queue.dequeue();
// Here you can do whatever processing you need on the frame, like detect barcodes, etc.
emit frameProcessed();
}
else
{
// No frames in queue, sleep for a short while
msleep(THREAD_SLEEP_MS);
}
}
qDebug() << "Processing thread ending";
exit(0);
}
In the ViewFinderWrapper we create the thread when we create the camera:
void ViewFinderWrapper::startCamera()
{
Q_ASSERT(!m_camera);
Q_ASSERT(!m_processor);
m_processor = new ProcessingThread(this);
connect(m_processor, SIGNAL(frameProcessed()), this, SLOT(onFrameProcessed()));
connect(m_processor, SIGNAL(queueFull()), this, SLOT(onThreadCongested()));
m_processor->start();
m_camera = new QCamera(this);
if (m_camera) {
connect(m_camera, SIGNAL(stateChanged(QCamera::State)), this, SLOT(onStateChanged(QCamera::State)));
connect(m_camera, SIGNAL(statusChanged(QCamera::Status)), this, SLOT(onStatusChanged(QCamera::Status)));
connect(m_camera, SIGNAL(error(QCamera::Error)), this, SLOT(onCameraError(QCamera::Error)));
m_camera->start();
m_cameraActive = true;
}
}
When we received the frame from the videosurface, we just pass it along for processing to the thread:
void ViewFinderWrapper::frameReady()
{
m_receivedFrameCounter++;
emit frameCountChanged(m_receivedFrameCounter);
// Get the current frame from the video surface
QImage frame = m_surface.frame();
// Add received frame to processing thread for processing
if (m_processor) {
m_processor->addFrameToProcessingQueue(frame);
}
// And take a copy for ourselves for drawing it on the screen
m_currentFrame = QPixmap::fromImage(frame);
// Update the UI
update();
}
Integrating QZXing
QZXing is a Qt-port of the popular ZXing barcode reading library and can be found in Nokia Developer Projects: http://projects.developer.nokia.com/QZXing/wiki
Integrating it to our example is easy. Just follow the instructions in the project wiki (copy three folders to your Symbian Qt SDK). Then add a few lines to your .pro-file:
# Add QZXing
LIBS += -lqzxing
customrules.pkg_prerules = \
";QZXing" \
"@\"$$(EPOCROOT)Epoc32/InstallToDevice/QZXing_selfsigned.sis\",(0xE618743C)"\
" "
DEPLOYMENT += customrules
In your processing thread code we instantiate a QZXing-instance and in the run-loop give QImages for it to process:
ProcessingThread::ProcessingThread(QObject *parent) :
QThread(parent), m_stopped(false), m_queueMaxLength(QUEUE_MAX_LENGTH)
{
m_zxing = new QZXing(this);
// By default all formats are enabled but if you want to select only some:
m_zxing->setDecoder(QZXing::DecoderFormat_QR_CODE);
}
void ProcessingThread::run()
{
// Process until stop() called
while (!m_stopped)
{
if (!m_queue.isEmpty())
{
QImage currentFrame = m_queue.dequeue();
// Here you can do whatever processing you need on the frame, like detect barcodes, etc.
QString text = m_zxing->decodeImage(currentFrame);
if (!text.isEmpty() && text.length() > 0) {
emit stringDecoded(text);
}
emit frameProcessed();
}
else
{
// No frames in queue, sleep for a short while
msleep(THREAD_SLEEP_MS);
}
}
qDebug() << "Processing thread ending";
exit(0);
}
Then it is all just a matter of handling the stringDecoded() signal (the QZXing-version can be found in http://projects.developer.nokia.com/CamTest/browser/CamTest2).
Example
This article only shows you the basic principles involved. For a more concrete example see the full demo application Nokia Developer Projects: http://projects.developer.nokia.com/CamTest
- Downloadable SIS for Symbian Anna/Belle: http://projects.developer.nokia.com/CamTest/downloads/6
- Downloadable SIS for Symbian Anna/Belle with QZXing integrated: http://projects.developer.nokia.com/CamTest/downloads/7
- Source code as a ZIP-package: http://projects.developer.nokia.com/CamTest/downloads/5
- Source code as a ZIP-package with QZXing integrated: http://projects.developer.nokia.com/CamTest/downloads/8
Tomi - Great article!
Hi,
Your approach seems to be similar to that of the MirrorHouse example (http://projects.developer.nokia.com/mirrorhouse) but with this great documentation explaining the not-so-trivial bits. Nice work!!!
In your introduction you mention that there are some differences in MeeGo Harmattan. I'd propose you also add a note about Raster back-end restrictions on Harmattan so that it doesn't come as a surprise to anyone.
Cheers,
TomiTomi_ 14:15, 28 May 2012 (EEST)
Hamishwillee - Very nice indeed.
Very nice indeed.
Quite a lot of the other articles (e.g. MeeGo Camera VideoSurface manipulation) cover the material in this article quite superficially. I like what you've done, particularly the detail like explaining how you drop frames when processing takes too long. I see this as very complementary to articles like MeeGo Camera VideoSurface manipulation and those that explain image processing. It can also exist in parallel with those that cover multithreading in more detail.
My only two content thoughts were:
I have subedited the article for readability. Most of this was to use the markup (e.g. Icode) we use across most of the wiki. Please check you're happy with it still.
Regards
Hamishhamishwillee 08:39, 31 May 2012 (EEST)
Riussi - Good input
Hi guys,
Good input on the article and I'm okay with any formatting edits. I'll see if I manage to integrate your input in the article soon.riussi 09:08, 31 May 2012 (EEST)