Introduction
A recent stab at grabbing images from the Flea2 camera using APIs from the FlyCapture2 SDK by Point Gray Research (PGR). Additionally, the camera was to be used in “Format 7 mode”, so that we may grab partial regions of the complete image.
This application builds on the AsynchTriggerEx C++ example provided by PGR, so that the user can grab images from the camera either by way of software triggers, or from an external hardware triggers.
Using Format7 mode to grab partial images
The Format7ImageSettings data structure is modified in order to set the desired output image size, pixel formats etc. Standard FlyCapture API calls are used to perform all necessary camera actions: connect to camera, disconnect from camera, start image capturing, retrieve image buffers etc. When the using the default equipment setup, the input images obtained are grayscales of 24 x 480 pixels consisting of an array of dark gray spots against a lighter shade of gray background.
Converting FlyCapture images into OpenCV images
In the image grabbing loop, FlyCapture2-obtained images are converted into OpenCV-compatible images by copying the pixel data from one to the other:
memcpy( img->imageData,
image.GetData(),
data_size );
Thresholding the grayscale image into binary (black & white)
The OpenCV grayscale image is then thresholded to produce a black and white (binary) image, from which we can use for things like detecting the number and location of the spots. Depending on the equipment used and lighting arrangement, some experimentation is probably needed to come up with an optimal threshold value.
cvThreshold( img, // source_img,
img_bw, // destination image
140, // threhold val.
255, // max. val
CV_THRESH_BINARY ); // binary type );
Original FlyCapture grayscale image, set to a Format 7 partial image (24 x 480 pixels) and converted to OpenCV format:
Thresholded to a black and white (binary) image:
Full code listing:
// CameraTest.cpp : Defines entry point for console application.
#include "stdafx.h"
#include "FlyCapture2.h"
#include <cv.h>
#include <highgui.h>
// Disable C4996 warnings
#pragma warning(disable: 4996)
#define SOFTWARE_TRIGGER_CAMERA
using namespace FlyCapture2;
const int col_size = 24;
const int row_size = 480;
const int data_size = row_size * col_size;
// Forward declarations
bool CheckSoftwareTriggerPresence( Camera* pCam );
bool PollForTriggerReady( Camera* pCam );
bool FireSoftwareTrigger( Camera* pCam );
void PrintError( Error error );
void ReleaseImage( IplImage* pimg,
IplImage* pimg_bw,
CvMemStorage* pstorage );
void GrabImages( Camera* pcam,
IplImage* pimg,
IplImage* pimg_bw,
CvMemStorage* pstorage,
int cnt );
// Main Control Loop
int _tmain(int argc, _TCHAR* argv[])
{
Camera cam;
CameraInfo camInfo;
Error error;
BusManager busMgr;
PGRGuid guid;
Format7PacketInfo fmt7PacketInfo;
Format7ImageSettings fmt7ImageSettings;
CvMemStorage* storage = NULL;
IplImage* img = NULL;
IplImage* img_bw = NULL;
TriggerMode triggerMode;
// Create OpenCV structs for grayscale image
img = cvCreateImage( cvSize( col_size, row_size ),
IPL_DEPTH_8U,
1 );
img_bw = cvCloneImage( img );
storage = cvCreateMemStorage( 0 );
// Get Flea2 camera
error = busMgr.GetCameraFromIndex( 0, &guid );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
// Connect to the camera
error = cam.Connect( &guid );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
// Get camera information
error = cam.GetCameraInfo(&camInfo);
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
#ifndef SOFTWARE_TRIGGER_CAMERA
// Check for external trigger support
TriggerModeInfo triggerModeInfo;
error = cam.GetTriggerModeInfo( &triggerModeInfo ;
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
if ( triggerModeInfo.present != true )
{
printf( "Camera doesn't support external trigger!\n" );
return -1;
}
#endif
// Get current trigger settings
error = cam.GetTriggerMode( &triggerMode );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
// Set camera to trigger mode 0
triggerMode.onOff = true;
triggerMode.mode = 0;
triggerMode.parameter = 0;
#ifdef SOFTWARE_TRIGGER_CAMERA
// A source of 7 means software trigger
triggerMode.source = 7;
#else
// Triggering the camera externally using source 0.
triggerMode.source = 0;
#endif
// Set camera triggering mode
error = cam.SetTriggerMode( &triggerMode );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
// Power on the camera
const unsigned int k_cameraPower = 0x610;
const unsigned int k_powerVal = 0x80000000;
error = cam.WriteRegister( k_cameraPower, k_powerVal );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
// Poll to ensure camera is ready
bool retVal = PollForTriggerReady( &cam );
if( !retVal )
{
PrintError( error );
return -1;
}
// Set camera configuration: region of 24 x 480 pixels
// greyscale image mode
fmt7ImageSettings.width = col_size;
fmt7ImageSettings.height = row_size;
fmt7ImageSettings.mode = MODE_0;
fmt7ImageSettings.offsetX = 312;
fmt7ImageSettings.offsetY = 0;
fmt7ImageSettings.pixelFormat = PIXEL_FORMAT_MONO8;
// Validate Format 7 settings
bool valid;
error = cam.ValidateFormat7Settings( &fmt7ImageSettings,
&valid,
&fmt7PacketInfo );
unsigned int num_bytes =
fmt7PacketInfo.recommendedBytesPerPacket;
// Set Format 7 (partial image mode) settings
error = cam.SetFormat7Configuration( &fmt7ImageSettings,
num_bytes );
if ( error != PGRERROR_OK)
{
PrintError( error );
return -1;
}
// Start capturing images
error = cam.StartCapture();
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
#ifdef SOFTWARE_TRIGGER_CAMERA
if ( !CheckSoftwareTriggerPresence( &cam ) )
{
printf( "SOFT_ASYNC_TRIGGER not implemented on "
"this camera! Stopping application\n" );
return -1;
}
#else
printf( "Trigger the camera by sending a trigger pulse"
" to GPIO%d.\n",
triggerMode.source );
#endif
error = cam.StartCapture();
// Warm up - necessary to get decent images.
// See Flea2 Technical Ref.: camera will typically not
// send first 2 images acquired after power-up
// It may therefore take several (n) images to get
// satisfactory image, where n is undefined
for ( int i = 0; i < 30; i++ )
{
// Check that the trigger is ready
PollForTriggerReady( &cam );
// Fire software trigger
FireSoftwareTrigger( &cam );
Image im;
// Retrieve image before starting main loop
Error error = cam.RetrieveBuffer( &im );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
}
#ifdef SOFTWARE_TRIGGER_CAMERA
if ( !CheckSoftwareTriggerPresence( &cam ) )
{
printf( "SOFT_ASYNC_TRIGGER not implemented on this"
" camera! Stopping application\n");
return -1;
}
#else
printf( "Trigger camera by sending trigger pulse to"
" GPIO%d.\n",
triggerMode.source );
#endif
// Grab images acc. to number of hw/sw trigger events
for ( int i = 0; i < 25; i++ )
{
GrabImages( &cam, img, img_bw, storage, i );
}
// Turn trigger mode off.
triggerMode.onOff = false;
error = cam.SetTriggerMode( &triggerMode );
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
printf( "\nFinished grabbing images\n" );
// Stop capturing images error = cam.StopCapture();
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
// Disconnect the camera
error = cam.Disconnect();
if ( error != PGRERROR_OK )
{
PrintError( error );
return -1;
}
ReleaseImage( img, img_bw, storage);
return 0;
}
// Print error trace
void PrintError( Error error )
{
error.PrintErrorTrace();
}
// Check for the presence of software trigger
bool CheckSoftwareTriggerPresence( Camera* pCam )
{
const unsigned int k_triggerInq = 0x530;
Error error;
unsigned int regVal = 0;
error = pCam->ReadRegister( k_triggerInq, ®Val );
if ( error != PGRERROR_OK )
{
// TODO
}
if( ( regVal & 0x10000 ) != 0x10000 )
{
return false;
}
return true;
}
// Start polling for trigger ready
bool PollForTriggerReady( Camera* pCam )
{
const unsigned int k_softwareTrigger = 0x62C;
Error error;
unsigned int regVal = 0;
do
{
error = pCam->ReadRegister( k_softwareTrigger,
regVal );
if ( error != PGRERROR_OK )
{
// TODO
}
} while ( (regVal >> 31) != 0 );
return true;
}
// Launch the software trigger event
bool FireSoftwareTrigger( Camera* pCam )
{
const unsigned int k_softwareTrigger = 0x62C;
const unsigned int k_fireVal = 0x80000000;
Error error;
error = pCam->WriteRegister( k_softwareTrigger,
k_fireVal );
if ( error != PGRERROR_OK )
{
// TODO
}
return true;
}
// Tidy up memory allocated for images etc
void ReleaseImage( IplImage* pimg,
IplImage* pimg_bw,
CvMemStorage* pstorage )
{
cvReleaseImage( &pimg );
cvReleaseImage( &pimg_bw );
cvClearMemStorage( pstorage );
}
// Grab camera grayscale image and convert into
// an OpenCV image
void GrabImages( Camera* pcam,
IplImage* pimg,
IplImage* pimg_bw,
CvMemStorage* pstorage,
int cnt )
{
Image image;
#ifdef SOFTWARE_TRIGGER_CAMERA
// Check that the trigger is ready
PollForTriggerReady( pcam );
printf( "Press Enter to initiate software trigger.\n" );
getchar();
// Fire software trigger
bool retVal = FireSoftwareTrigger( pcam );
if ( !retVal )
{
// TODO.
}
#endif
// Retrieve image before starting main loop
Error error = pcam->RetrieveBuffer( &image );
if ( error != PGRERROR_OK )
{
// TODO.
}
// Copy FlyCapture2 image into OpenCV struct
memcpy( pimg->imageData,
image.GetData(),
data_size );
// Save the bitmap to file
cvSaveImage( "orig.bmp", pimg );
// Threshold to convert image into binary (B&W)
cvThreshold( pimg, // source image
pimg_bw, // destination image
145, // threhold val.
255, // max. val
CV_THRESH_BINARY ); // binary type );
// Save the bitmap to file
char buffer[ 20 ];
sprintf( buffer, "%d", cnt );
std::string mess = "B_W.bmp_";
mess.append( buffer );
mess.append( ".bmp" );
cvSaveImage( mess.c_str(), pimg_bw );
// Find connected components using OpenCV
// for black spots against a white background
CvSeq* seq;
int num_blobs = cvFindContours( pimg_bw,
pstorage,
&seq,
sizeof( CvContour ),
CV_RETR_LIST,
CV_CHAIN_APPROX_NONE,
cvPoint( 0, 0 ) ) - 1;
}
Other posts related to image detection
Tracking Coloured Objects in Video using OpenCV
Displaying AVI Video using OpenCV
Analyzing FlyCapture2 Images obtained from Flea2 Cameras
OpenCV Detection of Dark Objects Against Light Backgrounds
Getting Started with OpenCV in Visual Studio
Object Detection Using the OpenCV / cvBlobsLib Libraries
Comments
25 responses to “Integrating the FlyCapture SDK for use with OpenCV”
Does this work for you when you try more complicated opencv functions such as camshift() or getting image first moment? I am successfully getting the frames and converting them to opencv format. I can even use cvShowImage() to show what the camera sees, but as soon as I try more advanced functions like the ones mentioned above, the program crashes without any valuable error message. I am developing in c++.
Thank you!
Pkout
Hi Pkout, I have never tried the more advanced stuff like camshift, so I’m not sure what the cause of your crash is. Would it help if you would try a bare minimum prototype that uses this camshift function and see if it still crashes then? It might be a more fundamental issue. I remember having to overcome all kinds of gotchas when trying out this opencv stuff for the first time. If you could point people towards some code that might help.
Thanks for the response! I know the problem isn’t in my implementation of OpenCV because the same code works fine when I either load the video from a file (even if the file is an uncompressed RAW8 format video from FlyCapture) or when I connect a webcam and, therefore, avoid using the flycapture-to-opencv memcpy() conversion. It starts crashing only when I implement the memcpy() on the flycapture image object. Weird. I suspect its something silly, like incorrect matrix data alignment or something of that sort. But I have no idea how to fix it :(. Thanks!
opencv imaages often crash if width and height of the image is not in 4 bytes multiplies exactly
try to change the image size
Hello,
I download your code and tried to build this code. Unfortunately, I got an error ” fatal error C1083: Cannot open include file: ‘stdafx.h’: No such file or directory”.
Please tell me how to fix this problem, thanks.
PS: I use VS 2008 and OpenCV 2.1;
Thank you for your help.
Hi – I think I could have done with mentioning this in the post.
If in Microsoft Visual Studio, you start by creating an empty
project and paste in the code I have given then yes, the compiler in all probability will complain along the lines of ‘fatal error C1083: Cannot open include file: ‘stdafx.h’: No such file or directory’. This is because it being an empty project with nothing in or no settings will need to know where ‘stdafx.h’ lives. Sorry about that.
In your Project properties, go to C/C++ -> General -> Additional Include Directories, and tell it where this file lives. In my VS 2010, this was located in:
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\atlmfc\src\mfc
It is probably different for yours. Once this is set and you re-compile, you might then get the following error message :
fatal error C1189: #error : Building MFC application with /MD[d] (CRT dll version) requires MFC shared dll version. Please #define _AFXDLL or do not use /MD[d]
Go into Project Properties -> C/C++ -> Code generation -> Runtime Library
and set this to Multi-threaded Debug (/MTd) instead of Multi-threaded Debug DLL (/MDd)
Alternatively:
Instead of using an empty project, in your Visual Studio create a new console project, while making sure to leave the ‘Precompiled Header’ checkbox in the
checked (with tick) state, and you should not get any of these set up / compilation problems.
To do this In Visual Studio, create a new project, choosing ‘Win32 Console Applicatio’n among the list of available options. Once this problem is out of the way, it is then a matter of setting up your FlyCapture / OpenCV settings. This I originally found to be a little trickier! Let me know how you get on.
Andy
Dear Mr. Andy,
Thanks you for your instruction.
Now, I am trying to setting up Flycapture/OpenCV. I have already added “C:\Program Files\Point Grey Research\PGR FlyCapture\include”; “…\lib”; “…\src” and “…\bin” into VS2008 project.
When I built this code, I got some error:
error C2871: ‘PGRFlyCapture’ : a namespace with this name does not exist
error C2065: ‘Camera’ : undeclared identifier
error C2065: ‘pCam’ : undeclared identifier
fatal error C1903: unable to recover from previous error(s); stopping compilation
Beside that, I changed “#include “PGRFlyCapture.h” ” into the code instead #include “FlyCapture2.h”. Am I correct this step?
Please help me correct these error.
Thank you so much.
nguyen
Your next step would be to make sure the FlyCapture software has been correctly installed and set up in Visual Studio. More detailed information on this can be found on this post:
https://www.technical-recipes.com/2011/getting-started-with-the-flycapture-sdk/
It basically involves these steps:
1. Install the full FlyCapture software package.
2. Setting project dependencies
3. Importing the FlyCapture property sheet(s).
Andy
Dear Mr.Andy,
I still got the same error even though I have already set up FlyCapture with Visual Studio as following your instruction on pages that you suggested.
What do I need to do for next step?
Thank you.
I think I know what your problem is. The FlyCapture set-up example I gave is appropriate for a more recent version of the FlyCapture SDK.
“PGRFlyCapture” you are using pertains to an older version (1.8 or older perhaps?) whereas the one I was using (“FlyCapture2”) is for the FlyCapture v2.2 Release 14 – Windows.
Check out their software downloads page for their SDKs to see what I mean. I have updated the other blog posting in order to clarify this.
Hi are you definitely using
#include “FlyCapture2.h”
and
using namespace FlyCapture2;
in your code? It might be a good idea to strip out all the other code, and start with an empty main and any #includes / namespaces you need. I haven’t visited this area for a while, but I will re-examine it when I get the chance. If I see any new problems or get any more insights I will update this post accordingly.
In general, a ‘Compiler Error C2871’ will occur when you pass an identifier that is not a namespace to a using directive:
http://msdn.microsoft.com/en-us/library/1s0wz6d0%28v=vs.80%29.aspx
I got some error as follows:
Error Trace:
Source: .\IidcCameraInternal.cpp(584) Built: Feb 4 2011 16:54:35 – Error settin
g Format 7 information.
+-> From: .\Format7.cpp(432) Built: Feb 4 2011 16:54:47 – Error setting Format
7 information.
+-> From: .\Format7.cpp(954) Built: Feb 4 2011 16:54:47 – Invalid Format7 image
size (L:312 T:0 W:24 H:480 M:0 PF:2147483648)
+-> From: .\Format7.cpp(1246) Built: Feb 4 2011 16:54:47 – Invalid image size s
etting.
The code stops at:
error = cam.ValidateFormat7Settings( &fmt7ImageSettings,
&valid,
&fmt7PacketInfo );
I do have pt grey chameleon camera connected. Please help.
Thanks,
Hi James
I have experienced problems with validating Format 7 settings before, which I have described in another posting:
https://www.technical-recipes.com/2011/when-the-flycapture2-validateformat7settings-function-returns-pgrerror_iidc_failed-type-error-message/
The problem I had was in choosing infeasible height/width settings. I found that it worked for (say) 24 x 480 or 56 x 482 pixels but not 52 x 484 or 24 x 483 – because the height had to be a multiple of 8.
It took me ages to realize that this was the problem. I guess Point Grey have documented this somewhere, but I had overlooked it. I’m not sure if there are any other considerations for when using the chameleon range. It may also be worth Googling ‘Format 7 error FlyCapture’ or similar.
Very helpful, thank you for sharing.
I download your code and tried to build this code. Unfortunately, I got an error.
void ReleaseImage( IplImage* pimg, IplImage* pimg_bw, CvMemStorage* pstorage );
voidReleaseImage
Error: incomplete type in not allowed
void GrabImages( Camera* pcam, IplImage* pimg, IplImage* pimg_bw, CvMemStorage* pstorage, int cnt );
Error: “IplIMage” is undefined
Marcus
The last error was solved add a line:
#include “C:\Program Files\OpenCV2.3\build\include\opencv2\core\core_c.h”
But now I have more error.
1>—— Build started: Project: Iniciando Firewire, Configuration: Debug x64 ——
1>Build started 23/02/2013 09:56:33.
1>InitializeBuildStatus:
1> Touching “x64\Debug\Iniciando Firewire.unsuccessfulbuild”.
1>ClCompile:
1> All outputs are up-to-date.
1> Main.cpp
1>C:\Program Files\OpenCV2.3\build\include\opencv2/flann/logger.h(70): warning C4996: ‘fopen’: This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
1> c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\stdio.h(234) : see declaration of ‘fopen’
1>Main.cpp(288): error C2664: ‘FlyCapture2::Camera::ReadRegister’ : cannot convert parameter 2 from ‘unsigned int’ to ‘unsigned int *’
1> Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast
1>Main.cpp(313): error C2664: ‘FlyCapture2::Camera::ReadRegister’ : cannot convert parameter 2 from ‘unsigned int’ to ‘unsigned int *’
1> Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast
1>
1>Build FAILED.
Do you have any solution?
Best regards.
About last post the error happens here.
bool PollForTriggerReady( Camera* pCam )
{
const unsigned int k_softwareTrigger = 0x62C;
Error error;
unsigned int regVal = 0;
do
{
error = pCam->ReadRegister( k_softwareTrigger,
regVal );
if ( error != PGRERROR_OK )
{
// TODO
}
} while ( (regVal >> 31) != 0 );
return true;
}
For this variable regVal has a message:
Error argument of type “unsigned int” is incompatibile with parameter of type “unsigned int”.
Do you have any tip?
Best regards,
Marcus Rene
Hi Marcus
I think this may be an issue with certain code symbols not being displayed correctly.
I think an ampersand (‘&’) is missing in various sections of the code. Can you please try placing a ‘&’ behind regVal, as in:
pCam->ReadRegister( k_softwareTrigger,
®Val );
Cheers,
Andy
Hi Andy,
Thank you very much for your instructions.
I was a lit bit confused, because I had FlyCap for x64 and x86 at same computer.
Maybe I was charge for dll x86 and lib x64, or vice versa
Thank you very much, because the program works very well now..
Cheers
Marcus Rene
Hi Andy,
This has been extremely useful for a project I am working on involving a 3D scanner with openCV and a Point Grey camera.
The program takes a live stream from the camera, displays it and gives the user more options for calibration/scanning/ect. Before we got the point grey camera, we had the code set up to use the camera as a webcam (for testing phases) since openCV plays very well with them (using cvcapturefromCAM()). Point grey has directshow dlls which allows openCV to *attempt* to use the ptgrey camera, but my program crashes and it seems that many people have issues upon camera connection.
My question is:
Have you been able to use a live stream from the point grey camera to openCV? My code is entirely set up to use a livestream from the camera and it would take a huge amount of work the change everything to image by image capture.
One thought I have had is to use Flycapture2 to continuously capture to an AVI file and have openCV use capturefromAVI(), but simultaneously reading and writing to this file has proved somewhat difficult.
Also, I am using Win7 x64, VS2010 x86, FlyCapture2/camera driver x86, and openCV 2.4.4 and compile everything in 32bit. I have also tried every camera driver(x64 and x86) that point grey has to offer for my camera(FireFlyMV) with no luck.
Hi Max
Everything I’ve done so far with FlyCapture / OpenCV has been on an image-by-image basis. I’ve been lucky in that my images were also quite simple – rows of inkjet printed spots, albeit quite tiny in the order of microns.
My only advice is to keep experimenting to find out what works reliably and what doesn’t. I agree that some aspects of FlyCapture and OpenCV are more flakey than others.
Maybe it would be beneficial if (one day) I could experiment with using livestreams and share my experiences on this blog, but don’t hold your breath just yet. So much to do, so little time…
Hi,
I created a new Wings console project and copied the copied code along with including the FlyCapture SDK and OpenCV libraries inthe project. However, on running the executable,windows just crashes and restarts. I am not able to figure out the problem for it.
Thank you for any help.
Just forgot to add in my previous post that I am using a PGR Flea3 camera
I want to the acknowledge GPIO high after an image is captured. How can I set GPIO to high when image is captured and low and done capturing the image
I implemented your code. When I run it, it freezes at retrieveBuffer function. Then, I close the program manually. Then, I run again and close it manually again. After two freezes, it runs successfully. How can I solve that problem?