Thanks to all of you who are interested in our algorithm. This is a new filtering method to extract DTM from LiDAR point cloud with only a few easy-to-set parameters. We also have submitted it to a scientific journal. Currently, the revised version of the paper is under review. We will publish our paper here as soon as it is accepted, and more details can be found on this paper. Please cite the paper if you use it for your publication.
In particular, we would like to thanks daniel, who builds a wonderful platform for developers and users. We also appreciate for your great help and quick response.
Actually, this plugin is under development, anyone have any suggestions, please contact us by email : wpqjbzwm@126.com or leave message here.
Add ground filtering plugin (CSF) to the official repository of cloudcompare
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
I cleaned the code, removed unnecessary memory copies as well as unused variables. This reduces the processing time and memory footprint. I also added a lot of safety checks to avoid a crash due to a lack of memory.
I updated the online beta version:
http://www.cloudcompare.org/release/Clo ... bin_x64.7z
I think the biggest optimization that could be done easily would be to remove this heavy mechanism based on a std::map and a key formed with the X and Z coordinates truncated as strings. I'm just not yet confident enough with the code to change this. Can you explain me a bit more what's the exact idea behind this?
I updated the online beta version:
http://www.cloudcompare.org/release/Clo ... bin_x64.7z
I think the biggest optimization that could be done easily would be to remove this heavy mechanism based on a std::map and a key formed with the X and Z coordinates truncated as strings. I'm just not yet confident enough with the code to change this. Can you explain me a bit more what's the exact idea behind this?
Daniel, CloudCompare admin
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
Hello,
Daniel, thanks for your kind help. Are you learning Chinese? Currenly, all our code comments are in Chinese, perphaps we may translate it to English in the future.
we use std::map because we want to find the nearest LiDAR point for each cloth particle, this really can be optimized. First of all, we project all the LiDAR points and cloth particles to a horizontal plane (which is xz plane, y is the vertical direction). Then for each projected cloth particle, we find the nearest LiDAR point in this plane. Here, we use CGAL to query the N nearest points, and it returns an array with N points. But only the coordinates (x and z) of these points are given, what I need is the ID or index. with ID, I can query the y coordinate of these nearest points. When ID is not available , I just made a std::map which builds a map between xz and y.
So, I think these can be improved. And I also found that the kNN algorithm implemented in CGAL is not very fast. This may be replaced by other well establised kNN library.
In addtion, I have question about cloudcompare. When it loads a very large point cloud file, it may take a relatively long time. But sometimes, I just want to have a quick view of this dataset. Is it possible to read the data every a few points, which makes it faster to read and rotate. This is just like a subsampling of original point cloud. I don't know is it easy to implement ?
Daniel, thanks for your kind help. Are you learning Chinese? Currenly, all our code comments are in Chinese, perphaps we may translate it to English in the future.
we use std::map because we want to find the nearest LiDAR point for each cloth particle, this really can be optimized. First of all, we project all the LiDAR points and cloth particles to a horizontal plane (which is xz plane, y is the vertical direction). Then for each projected cloth particle, we find the nearest LiDAR point in this plane. Here, we use CGAL to query the N nearest points, and it returns an array with N points. But only the coordinates (x and z) of these points are given, what I need is the ID or index. with ID, I can query the y coordinate of these nearest points. When ID is not available , I just made a std::map which builds a map between xz and y.
So, I think these can be improved. And I also found that the kNN algorithm implemented in CGAL is not very fast. This may be replaced by other well establised kNN library.
In addtion, I have question about cloudcompare. When it loads a very large point cloud file, it may take a relatively long time. But sometimes, I just want to have a quick view of this dataset. Is it possible to read the data every a few points, which makes it faster to read and rotate. This is just like a subsampling of original point cloud. I don't know is it easy to implement ?
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
Another queation, I find there are several pcl_*.dll files in the this beta version of cloudcompare,
Can pcl functions be used in cloudcompare, such as keypoints and feature descriptors?
Do you have a plan to put more functionalities into PCL wrapper plugin?
Thank you very much!
Can pcl functions be used in cloudcompare, such as keypoints and feature descriptors?
Do you have a plan to put more functionalities into PCL wrapper plugin?
Thank you very much!
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
For the kNN algorithm: CloudCompare has a unique octree structure (CCLib::DgmOctree). And it's very effective ;). I'll try to change that and see if it helps. And more generally for your std::map you should avoid using a string as key (it's very slow to build the two strings from the double coordinates, and it takes a lot of memory). Maybe a simple quantization of the two coordinates would have been sufficient (i.e. knowing the bounding-box you can convert a double coordinate to a 32 or 64 bits integer using the integer's full range).
And while I'm speaking about implementation: one big bottleneck in your code was that you were copying big vectors each time you passed them as parameters to other functions (or as return value). Always prefer to pass them by reference (with '&').
Regarding loading partial clouds from files: it really depends on the file format. Most of the format are read with third party libraries (E57, LAS, PLY, etc.) and therefore it won't be possible to change anything. And for the ASCII files, as you don't know before hand the number of points, it's not easy to skip points. But I guess it should still be possible in a kind of hacky way... We should add this to the TODO list.
And regarding PCL: the PCL dlls are brought by the qPCL plugin. It has been developed by Luca Penasa and I don't know what are his plans for the future. But anyone is free to extend the set of supported functions.
And you can add PCL as a dependency to your own plugin (but be sure that it's really necessary as PCL is huge ;). You can look at how qPCL does it.
And while I'm speaking about implementation: one big bottleneck in your code was that you were copying big vectors each time you passed them as parameters to other functions (or as return value). Always prefer to pass them by reference (with '&').
Regarding loading partial clouds from files: it really depends on the file format. Most of the format are read with third party libraries (E57, LAS, PLY, etc.) and therefore it won't be possible to change anything. And for the ASCII files, as you don't know before hand the number of points, it's not easy to skip points. But I guess it should still be possible in a kind of hacky way... We should add this to the TODO list.
And regarding PCL: the PCL dlls are brought by the qPCL plugin. It has been developed by Luca Penasa and I don't know what are his plans for the future. But anyone is free to extend the set of supported functions.
And you can add PCL as a dependency to your own plugin (but be sure that it's really necessary as PCL is huge ;). You can look at how qPCL does it.
Daniel, CloudCompare admin
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
Thank you very much to point out these problems to improve the efficiency of CSF plugin, we are also seeking to optimize it.
Do you have compared the performance between CCLib::DgmOctree and other well-known kNN library, such as ANN (http://www.cs.umd.edu/~mount/ANN/) and flann (http://www.cs.ubc.ca/research/flann/) ?
In addtion, do you ever have used any other LiDAR filtering method, how do you think of our CSF in aspects of userbility, efficiency and accuracy? This is really important to us. we just want to know some comments about our algorithm.
Thank you again!
Do you have compared the performance between CCLib::DgmOctree and other well-known kNN library, such as ANN (http://www.cs.umd.edu/~mount/ANN/) and flann (http://www.cs.ubc.ca/research/flann/) ?
In addtion, do you ever have used any other LiDAR filtering method, how do you think of our CSF in aspects of userbility, efficiency and accuracy? This is really important to us. we just want to know some comments about our algorithm.
Thank you again!
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
I just updated the plugin (same link as before):
http://www.cloudcompare.org/release/Clo ... bin_x64.7z
I used CC's octree instead of CGAL. On the rasterize and distance calculation steps, it goes almost 10 times faster (also because CC's octree is doing things in parallel). CC's octree could be even faster if it was able to work in 2D, but sadly it's 3D only for now.
(You can still revert to the old code by checking the 'QCSF_CGAL_SUPPORT' option in CMake).
I also added an option to export the cloth mesh at the end of the process (I was curious ;):
And regarding your last questions, I compared CC's with 2 other algorithms during my PhD and my Octree was faster. However it mainly depends on what you want to do. CC's octree is very good when applying a process on the whole cloud (which is exactly what we do when computing distances, etc.). But it's not that good for random queries done on a few points only.
I know flann and ANN are very popular. But I feel they are too generic to be very efficient (especially on big point clouds). They are still very good tools when developing code outside of CC.
http://www.cloudcompare.org/release/Clo ... bin_x64.7z
I used CC's octree instead of CGAL. On the rasterize and distance calculation steps, it goes almost 10 times faster (also because CC's octree is doing things in parallel). CC's octree could be even faster if it was able to work in 2D, but sadly it's 3D only for now.
(You can still revert to the old code by checking the 'QCSF_CGAL_SUPPORT' option in CMake).
I also added an option to export the cloth mesh at the end of the process (I was curious ;):
- CSF_cloth.jpg (111.29 KiB) Viewed 221190 times
I know flann and ANN are very popular. But I feel they are too generic to be very efficient (especially on big point clouds). They are still very good tools when developing code outside of CC.
Daniel, CloudCompare admin
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
I don't have so much experience in bare-earth extraction plugins (I only used the Rasterize tool and CANUPO to do this so far).
I think your approach is clever and it's very easy to understand how it works (which is important in order to set the parameters correctly and therefore it increases the usability). And for the few clouds I tested it worked pretty well. Especially if I check the 'Slope processing' checkbox (why isn't it checked by default by the way?).
I think other user could tell you more!
I think your approach is clever and it's very easy to understand how it works (which is important in order to set the parameters correctly and therefore it increases the usability). And for the few clouds I tested it worked pretty well. Especially if I check the 'Slope processing' checkbox (why isn't it checked by default by the way?).
I think other user could tell you more!
Daniel, CloudCompare admin
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
Downloaded the last version and I find it pretty slow compared to the development version I found previously.
If it is possible, I'd like to see a new option to the plug in: right now, after you used it, the plug in creates 2 differents files so I have to manually add a new SF as "class", add a value for each point cloud (1 for ground, 2 for non-ground for example) and then merge both clouds together. I think it could be very helpful to add an option to check to directly create merged cloud!
Otherwise, I'm using this plug in a terrestrial scans and it works pretty nicely, except of course for the "flying" parts of it ;)
Great job
If it is possible, I'd like to see a new option to the plug in: right now, after you used it, the plug in creates 2 differents files so I have to manually add a new SF as "class", add a value for each point cloud (1 for ground, 2 for non-ground for example) and then merge both clouds together. I think it could be very helpful to add an option to check to directly create merged cloud!
Otherwise, I'm using this plug in a terrestrial scans and it works pretty nicely, except of course for the "flying" parts of it ;)
Great job
Re: Add ground filtering plugin (CSF) to the official repository of cloudcompare
Pretty slow?! Can you post the timings that appear in the Console? (the various phases now appear with the corresponding timing).
Daniel, CloudCompare admin