add convenience method & sample code for registration

[floe]

Thanks again @larshg and @xlz for your sample code. I've rewritten it to not use OpenCV, as we probably want to remove that dependency at some point (see also the PR by @christiankerl). Also, I have not yet implemented the depth buffer from @xlz's example, will do that soon.

[xlz]

is rounding the right way to go here - no subpixel precision?

RGB images are much oversampled than depth images. More sampling precision on RGB would have little improvement.

 r_off = d_off*3;

3 is just hardcoded bytes_per_pixel value. VA-API and Tegra hardware jpeg decoders return in BGRA format. They would need 4 for bytes_per_pixel.

[floe]

Good point about bytes_per_pixel, I'll fix that.

[floe]

Any test reports from Windows or MacOS? I'm reluctant to merge this otherwise.

[gaborpapp]

Thanks for your work on this. I tested on MacOS 10.9.5. Registration seems to work, although I get some noise at the bottom of the image.

Also I get a crash on exit, which might not be related, but I don't get it with the latest 585d4c4 upstream/master.
Crash report:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libusb-1.0.0.dylib 0x00000001091a0886 usbi_log_v + 70 (core.c:2290)
1 libusb-1.0.0.dylib 0x000000010919eb5f usbi_log + 127 (core.c:2378)
2 libusb-1.0.0.dylib 0x00000001091a9e76 darwin_cancel_transfer + 438 (darwin_usb.c:1746)
3 libusb-1.0.0.dylib 0x00000001091a3163 libusb_cancel_transfer + 99 (io.c:1500)
4 libfreenect2.dylib 0x0000000108f8128a libfreenect2::usb::TransferPool::cancel() + 138 (deque:299)
5 libfreenect2.dylib 0x0000000108f902cb libfreenect2::Freenect2DeviceImpl::stop() + 411 (libfreenect2.cpp:567)
6 Protonect 0x0000000108f6eb99 main + 3033 (Protonect.cpp:106)
7 libdyld.dylib 0x00007fff876a15fd start + 1

[xlz]

The noise:

    uint8_t registered[depth->height*depth->width*3];
    registration.apply(rgb,depth,registered);
    cv::imshow("registered", cv::Mat(depth->height, depth->width, CV_8UC3, registered));

Need a memset here, and probably it's not a good idea to define a huge array on stack. registered = cv::Mat::zeros() should work.

The crash is a race condition, discussed in #212.

[gaborpapp]

Thanks! memset and the xlz@c6f6e57 commit mentioned in #212 indeed fixed both issues.

[floe]

Thanks for testing - instead of memset, I've modified the inner loop to also set skipped pixels to zero, so we don't need to memset everything to zeroes which will get overwritten anyway.

[gaborpapp]

Thanks. This also solves the issue.

[xlz]

FTBFS with MSVC 2013 here:

uint8_t registered[depth->height*depth->width*rgb->bytes_per_pixel];

Error C2057: expected constant expression

[floe]

Ooops, OK (a bit surprising, but as you said, it's probably a better idea to move the buffer to the heap anyway). I'll fix that ASAP.

[xlz]

Also, this line causes stack overflow with MSVC:

libfreenect2::Registration registration(dev->getIrCameraParams(), dev->getColorCameraParams());

After replacing this and registered with heap allocation, it works correctly on Windows.

[floe]

Also, this line causes stack overflow with MSVC:

Now that is somewhat surprising, why should this line be an issue? IMHO it should be valid C++, can you post a stacktrace?

[xlz]

The stack overflow happens upon entering main().

Probably because libfreenect2::Registration is quite large on main()'s stack: 3392KB.

[floe]

Ah, OK. I would have assumed the default stack is a lot bigger, but it's fixed now.

[larshg]

I had to add #include <stdint.h> to make it work for VS2013 Update 4 else uint8_t is unknown.

And it seems that default stack is 1 MB :)

[floe]

Also fixed, now using unsigned char* which is probably the most portable solution.

[larshg]

Works great 👍

[JuantAldea]

Given that the depth frame size is already known to be 512x424 if I recall correctly, is there any reason to not to allocate outside of the loop?

[floe]

Depending on the color frame decoder, rgb->bits_per_pixel is either 3 or 4, so I'd like to keep it consistent.

[floe]

OK, since everything works as expected across all major platforms, I'll merge this tomorrow; additional features will be added via separate PRs. Thanks for contributing, everyone!

[JuantAldea]

This are the results I am getting from the registration:

i) noticeable radial distortion in the closet. Shouldn't the depth data be undistorted first?

ii) pixel duplication, you can see my hand and the pen appear two times. Is this normal? Shouldn't registration be a bijection?

Camera calibration parameters used are those returned by the lib.

I'm quite newbie on this subject so I might be missing something.

[floe]

@JuantAldea

1. the depth data is already being undistorted, but the built-in parameters are probably just not as good as those you would get from an individual chessboard calibration. In any case, if you compare it to the raw depth data, I think the distortion is already somewhat better.
2. this is a fundamental geometric limitation, from the POV of the color camera, the pixels on the right are directly behind the pen, too. This will be solved in a second PR by using a depth buffer.

[JuantAldea]

@floe Thanks.

I'd say it looks like the depth isn't undistorted but the color is "distorted" so that it matches the depth.

I've superimposed the three images (registration, depth, and IR) as layers on this [1] Gimp file. You can play a little with the transparency and the layer order.

[1] https://www.dropbox.com/s/t2nnf737jvt4b3y/IR_DEPTH_REGISTERED.xcf?dl=0

[floe]

@JuantAldea good point, got it backwards myself. Nice example, thanks!

[JuantAldea]

So is this working as intended? Shouldn't the depth be undistorted?

[floe]

As far as we know, the parameters stored in the device are specific for this operation only, i.e. mapping color onto depth. Unless @sh0 knows about some more internals (such as the tables at the end of DepthCameraParamsResponse), this is all we can currently support without external calibration.

[JuantAldea]

Got it, thanks.

[floe]

Actually, maybe I was posting too soon: in theory, the parameters for standalone depth undistortion should be there, too. I'll re-read the old discussion in #41 tomorrow.

[floe]

I was obviously posting this too late at night yesterday evening, of course it's possible to undistort the depth image separately - that's exactly what I added undistort_map for in the first place :-)

[robotsorcerer]

I'm trying to do a face detection with the Protonect.cpp code. I need to apply a haarclassifier on the registered image. I tried with opencv haarcascade class

//-- Read the registration stream
registration.open( -1 );
if( registration.isOpened() )
{
for(;;)
{ registration >> frame;
//-- 3. Apply the classifier to the frame
if( !frame.empty() )
{ detectAndDisplay( frame ); }
else
{ printf(" --(!) No captured frame -- Break!"); break; }
int c = waitKey(10);
if( (char)c == 'c' ) { break; }
}
}
}

dev->stop();
dev->close();

but I kept getting a member error error: request for member ‘open’ in ‘registration’, which is of pointer type ‘libfreenect2::Registration*’ (maybe you meant to use ‘->’ ?) registration.open( -1 );
What is the correct way to access the registered/depth/rgb/undistorted frames?

[floe]

@lakehanne see here: https://github.com/OpenKinect/libfreenect2/blob/master/examples/protonect/Protonect.cpp#L149-L152

[robotsorcerer]

Thanks! I seem to have some preliminary results on a face detection algorithm using haarcascades

my code here.
Will appreciate contributions~!