Quality Report accuracy results vs. Actual DSM output accuracy resutls

Hello,

I recently performed a PPK mission with a lab-calibrated X4S. I had several GCPs and check points as checks, which were established in the field with RTK-GNSS (please see attached Quality Report). Overall I was very happy the results after reviewing the Quality Report after Step 1 & reoptimization.
ISWS_08-25-19_report.pdf (2.0 MB)
Then I exported the DSM/Ortho after completing Steps 2 and 3 and assessed the output files in Virtual Surveyor and Civil 3D. I used Virtual Surveyor to decimate the full resolution DSM from 2.5cm grid to 0.5m and exported a LandXML for import into Civil 3D along with the Ortho. I then remarked all the GCPs and Check points in Civil 3D based on the imported Surface and Ortho.

Horizontally, the accuracies were pretty good and matched the Pix4D Quality Report. However, vertically, there was more error than I was expecting…especially directly on the points used in Pix4D as GCPs. Please see tabulated results below. When comparing the GCPs between the Quality Report and this table, you will see the vertical discrepancies.

Am I misunderstanding what the Quality Report “Geolocation Details” section on GCP and Check Point accuracy mean after Step 1? Is there a chance the Point Cloud Densification step could potentially not hold the elevations at the GCPs? If so, I must ask what does the Geolocation Details section represent then and how does this get translated through to the point cloud and DSM generation steps? Are these accuracies only specific to the Automatic Tie Point cloud and may not push to the point cloud and DSM? I was always taught the ATP was the infrastructure for the Step 2 and 3 data products, so I’m not sure why the DSM output elevations aren’t consistent with the Quality Report.

Thanks,

Mike

Hi,

I have checked the quality report generated by Pix4Dmapper and the results looks very good, i.e. the RMS error ranges from 0.55 to 1.55 cm (the expected accuracy when using PPK/RTK). The geolocation results provided by our quality report were correctly interpreted and the georeferencing carried out using GCPs is also propagating into the final results (point cloud, DSM, DTM, orthomosaic etc.). If you want to cross-check, please mark some sample GCPs directly in the rayCloud and verify if the values are correct.
I don’t really know what happened in Virtual Surveyor or Civil 3D, but I would assume something changed when the resolution of the DSM got resampled to 0.5 meters. I do not have the knowledge on how to ensure that the GCPS/checkpoints are correctly marked in Civil 3D. For this, I would advise contacting the technical support of these 3rd party softwares or checking our Community threads, there might be a similar topic where some savvy users found a solution to this problem.
If you are not sure about the quality of your DSM/ortho, I would recommend comparing them with other datasets for the same study area (aerial photographs, ground truth data, satellite images etc.). Alternatively you could generate the DSM/ortho in other 3rd party software and compare them with the Pix4D outputs.

Cheers,
Teodora

Thanks for your response , but lets be clear. The Quality Report shows wonderful results. However, lets just focus on GCPs and how they’re used for the Automatic Tie Point Cloud and resulting DSM output then.

Please see image 1 below of a selected GCP within the Pix4D interface. Note the elevation of -85.931ft “Initial” and -85.945ft “Computed”. As you can see they’re very close…just as the Quality Report indicates:

However, when I pick around the adjacent ATP points, they’re not the same as the GCP elevation…which is odd because the 2’ x 2’ aerial target I used is on flat ground. Please see the following images of selected ATPs (after re-optimization step):


Note how the ATP points have elevations around -85.6ft to -85.7ft

This elevation range is what my DSM is showing…not what the GCP elevation should be. So my question is, why doesn’t the ATP (after re-optimization) or the resulting DSM output not factor in the GCP elevation at this location? I was expecting the DSM at a minimum to at least add the elevation of the GCP in the center of the target correct?

As a double check, I checked my DSM creation settings and I did have Noise Filtering and Surface Smoothing checked. However, if Pix4D actually factored in the GCP elevation at this location into the equation, I shouldn’t see a 0.2ft elevation difference between the Pix4D Quality Report and the calculated ATP and resulting DSM correct???

This gives me the impression that the Pix4D Quality Report is misleading. Please explain why there is a difference for the adjacent ATP points that are so close to this GCP. As you know, the output files are what is used for engineering design…not the quality report. This is very concerning indeed. Please clarify why there is a such a large discrepancy.

I should also say I now see this issue in multiple projects at GCP locations. Very troubling.

Thank you.

Hi Mike,

In theory, the uncertainties close to the GCPs should be very small and increase for images further away. This can be improved by distributing the GCPs homogeneously in the project. We recommend using approximately 5-10 GCPs (the more, the better). Please check out this article for the best practice: Getting GCPs on the field or through other sources.

If you notice this difference in the DSM or DTM, this might be related to the interpolation algorithms used for generating these outputs. As you probably know, spatial interpolation is the process of using points with known values to estimate values at other unknown points. Looking at the screenshots you attached, I see that there are not many ATP found near that GCP, which can results in significant uncertainties within the interpolated surface. Have you tried both interpolation algorithms, Triangulation and Inverse Distance Weighted for the DSM, in order to see if there is any difference?

One more thing: coordinate systems (CS). For the horizontal CS, I see you are using the NAD83 (2011)/California zone 6 for the images, GCPs, and outputs. However, I see no vertical CS being defined in your project, which might be a problem if the images and GCPs are 3D (with X, Y, Z coordinates). In the database, Pix4Dmapper lets you choose between a bunch of global geoid models or set a constant height shift in relation to the ellipsoid. This is acceptable in terms of accuracy for consumer drones (meter accuracy), but not very appropriate for RTK/PPK drones. If you want to make sure that the vertical accuracy is high, I would recommend converting the (original) vertical CS of the dataset with a transformation tool (please consult your local mapping institute) and importing it in Pix4Dmapper by using the same CS for image, GCP and outputs, so that Pix4Dmapper does not perform any additional conversions. Please read more about geoids here: How to define Pix4D outputs with respect to a Geoid model.

Cheers,
Teodora

Hello Teodora and thank you for your response and sending basic links to educate me on how to use the software. Please see my responses to your comments below:

In theory, the uncertainties close to the GCPs should be very small and increase for images further away. This can be improved by distributing the GCPs homogeneously in the project. We recommend using approximately 5-10 GCPs (the more, the better).

I am well aware of this and for this project I had well distributed GCPs and Check Points throughout the project as you can see from the Quality Report.

If you notice this difference in the DSM or DTM, this might be related to the interpolation algorithms used for generating these outputs. As you probably know, spatial interpolation is the process of using points with known values to estimate values at other unknown points. Looking at the screenshots you attached, I see that there are not many ATP found near that GCP, which can results in significant uncertainties within the interpolated surface. Have you tried both interpolation algorithms, Triangulation and Inverse Distance Weighted for the DSM, in order to see if there is any difference?

As the Quality Report indicated, I used Inverse Distance Weighted. The screen shots I provided were showing 4-5 ATPs within the 2’ x 2’ Aerial Target. The GCP is in the center of this target. The two ATPs I highlighted in the screen shots were less than 8-10 inches away from the GCP. Uncertainties in this case should be very low.

The problem I have is that the Pix4D Quality Report is misleading. It gives a user the impression that accuracies are achieved at the GCP and Check Point Locations, but when you take the time to actually check the outputs from Pix4D (Point Cloud, DSM), you don’t get the GCP elevation at the center of target. This is the heart of my concern and should be addressed.

One more thing: coordinate systems (CS). For the horizontal CS, I see you are using the NAD83 (2011)/California zone 6 for the images, GCPs, and outputs. However, I see no vertical CS being defined in your project, which might be a problem if the images and GCPs are 3D (with X, Y, Z coordinates). In the database, Pix4Dmapper lets you choose between a bunch of global geoid models or set a constant height shift in relation to the ellipsoid. This is acceptable in terms of accuracy for consumer drones (meter accuracy), but not very appropriate for RTK/PPK drones. If you want to make sure that the vertical accuracy is high, I would recommend converting the (original) vertical CS of the dataset with a transformation tool (please consult your local mapping institute) and importing it in Pix4Dmapper by using the same CS for image, GCP and outputs, so that Pix4Dmapper does not perform any additional conversions. Please read more about geoids here: How to define Pix4D outputs with respect to a Geoid model.

Correct I used NAD83 (2011)/California zone 6 for the images, GCPs, and outputs. I used a Trimble RTK survey-grade equipment to set the GCPs and Check Points and also used a PPK drone. Since Pix4D doesn’t support “local” Geoid Models (like Geoid 12B), I have to use the Arbitrary vertical system. So this is not the issue. Thank you for pointing me toward the link for Geoid models, but that is not helpful in this case.

As I stated before, I should be able to import the full resolution point cloud or DSM into a 3rd party program and query the elevations in the center of my GCP aerial targets and get the same elevation as reported by the Pix4D Quality Report. However, I don’t get the same elevation. I think I have shown in my example that the ATPs around the example GCP are 0.2’ off from the GCP elevation <1’ away. Therefore, I just don’t see how interpolation would cause 6-7 cm of error.

My question is how should we check the Pix4D outputs at GCP and Check Point Locations then??
Do you have an article that explains that procedure?

I think I have demonstrated that the elevation of my example GCP doesn’t get pushed out to the Point Cloud or DSM. Looks like the Point Cloud densification step and resulting DSM step use the ATPs, but do not add in the GCP elevations , which is very problematic, as the Point Cloud and DSM outputs are ultimately used.

Michael, I have a few comments and observations:

  1. The DJI X4S is a pretty good mapping camera, but everything about the mission must be perfect in order to get the kind of accuracy you’re looking for. I have essentially the same camera in my Phantom 4 Pro. What % overlap did you fly the mission? It should be at least 75/75. What capture software did you use? I have to use Drone Deploy to capture the images, as Pix4DCapture overrides the manual camera settings in DJI Go. Manually setting the focus to infinity and shutter speed to at least 1/800s is mandatory to prevent blurry images. The slightest out of focus images will add error.

  2. With your GSD of 0.086’, vertical accuracy between 0.172-0.258’ (2-3xGSD) is acceptable for elevations derived from photogrammetry. It’s possible to get 1-2x GSD vertical accuracy if all the surveying and image capture procedures are very precise. Your vertical RMSE of 0.113’ is actually very good (1.3x GSD). If you need higher accuracy, you have to fly lower or get a different camera. For example, using a 42MP Sony RX1RII camera, I get more accurate results at 400 feet than I do with my 20MP P4Pro camera at 200 feet.

  3. To generate the “cleanest” and most accurate point cloud possible, set the minimum number of matches to 6 instead of 4. This will result in fewer overall points in the point cloud, but the resulting DSM will be more accurate.

  4. How was the RTK GCP/Checkpoint data collected? Using VRS or a local base with radio link to the rover? I’ve seen VRS shots vary by nearly 0.1’. When at all possible, I use a total station. For your project I would use a base-rover and not VRS. Your survey methodology has to result in points that are at least as accurate as your GSD.

  5. I check my accuracy in ArcGIS and calculate RMSE values using the full resolution DSM. In ArcGIS I import my checkpoints then run a geoprocessing tool to extract the DSM values at each checkpoint. Then calculate my errors. I wouldn’t try to calculate errors from a resampled DSM.

I hope my comments are helpful.

1 Like

Thank you for taking the time to help Andrew. You made some excellent suggestions and gave me food for thought to consider for the next mission. Please see my responses below to your questions.

The DJI X4S is a pretty good mapping camera, but everything about the mission must be perfect in order to get the kind of accuracy you’re looking for. I have essentially the same camera in my Phantom 4 Pro. What % overlap did you fly the mission? It should be at least 75/75. What capture software did you use?

I agree with your comment about set up being key. I was using a lab calibrated X4S from Klau Geomatics at 80% front and 65% side overlaps with UgCS mission planning software. I used Klau PPK software to resolve my baselines to my local Trimble R8-M3 Base Station position. Prior to PPK processing though I did baseline processing to 3 nearby CORS stations and 2 concurrent static Trimble R6 M4 GNSS set ups on NGS monuments to determine my base station position. I confirmed the Trimble Business Center final computed position against Trimble RTX, OPUS, and SMARTNET computed positions as well.

I have to use Drone Deploy to capture the images, as Pix4DCapture overrides the manual camera settings in DJI Go. Manually setting the focus to infinity and shutter speed to at least 1/800s is mandatory to prevent blurry images. The slightest out of focus images will add error.

For the same reason, I use UgCS to allow for proper manual exposure settings.

With your GSD of 0.086’, vertical accuracy between 0.172-0.258’ (2-3xGSD) is acceptable for elevations derived from photogrammetry. It’s possible to get 1-2x GSD vertical accuracy if all the surveying and image capture procedures are very precise. Your vertical RMSE of 0.113’ is actually very good (1.3x GSD). If you need higher accuracy, you have to fly lower or get a different camera. For example, using a 42MP Sony RX1RII camera, I get more accurate results at 400 feet than I do with my 20MP P4Pro camera at 200 feet.

Yes, I totally agree. I recently bought a RX1RII camera and custom gimble for my Dji M200. Amazing camera with crisp images and low distortion at 400’ (1.55cm GSD). However, recently the internal focus motor has caused image size to be cut in half and soft images (despite having all the manual exposure settings dialed in). This could be caused by too much vibration from the gimble rig and the M200. I have sent the camera off for repair so I hope they can fix it b/c I prefer that camera too. My X4S is my backup work horse. I usually fly at 1.5cm GSD (~180-200ft AGL) with that one, so I may have been pushing the limits with this mission at 2.7cm GSD.

Furthermore, I weighted down the aerial targets with rocks. Please see image below.

This is the problem point shown in images further up the thread. I suspect that the rocks caused the ATP to determine high points in relation to the center GCP point. However, when the DSM is subsequently created with Noise Filtering and surface smoothing checked ON, it blends the four corners, which would account for the ~+0.2’ vertical error I’m seeing. At the GCP it is fine, but the DSM output seems to have a smooth-factoring on the top of these small rocks, which disregard the GCP at the center of target. It is just a guess though. I just thought the GCP point was the anchor and regardless, the outputs should at a minimum should have that elevation close to the RTK-GNSS observations as the Quality Report indicated.

To generate the “cleanest” and most accurate point cloud possible, set the minimum number of matches to 6 instead of 4. This will result in fewer overall points in the point cloud, but the resulting DSM will be more accurate.

Ok I was always taught 4 to ensure somewhat efficient processing times. I will try with 6 as a test.

How was the RTK GCP/Checkpoint data collected? Using VRS or a local base with radio link to the rover?

I used a Trimble RTK (base/rover) system + 2 continuously logging GNSS setups on NGS monuments.

I’ve seen VRS shots vary by nearly 0.1’. When at all possible, I use a total station. For your project I would use a base-rover and not VRS. Your survey methodology has to result in points that are at least as accurate as your GSD.

I agree.

I check my accuracy in ArcGIS and calculate RMSE values using the full resolution DSM. In ArcGIS I import my checkpoints then run a geoprocessing tool to extract the DSM values at each checkpoint. Then calculate my errors. I wouldn’t try to calculate errors from a resampled DSM.

Only difference is I use Virtual Surveyor so I can efficiently view the full resolution DSM and Ortho in 3D and do other analyses. I use ArcMap too to reduce the ortho size for Civil 3D consumption only.

Regardless, my checks on the decimated surface from Virtual Surveyor in Civil 3D could be some of the problem here, which is why I went back to the original raw output files generated from Pix4D and did the same checks. I see the same error.

I guess this is where I think the GCP placement relative to the surroundings is key. Meaning don’t place them near objects that could interfere with ATP and point cloud/dsm generation. Also I’m not going to use rocks to weight down the targets anymore. I will drive anchors in the ground and make target as smooth as possible on level ground.

I hope my comments are helpful.

They were thank you very much!

1 Like

I have a PhD colleague that suggests using rigid panels for targets as opposed to flexible mesh or vinyl targets, for this very reason. He made custom targets out of aluminum sheets with vinyl iron cross decals. He said the area around the GCP needs to be perfectly flat. Since he has 4 degrees to my 1, I trust his judgement lol!

It certainly looks like your methodology is very good. Hopefully you can figure this out.

Ok great advice. Thanks again for your tips.

I would love to create something similar to how you guys created those custom targets. I recently saw little 12" x 12" blue and black targets by my house with a PK Nail through the center…looked perfect for a RX1RII…probably too small for an X4S mission though. The material was rigid and light…like cardboard, but more rigid. Did your friend have to have the aluminum cut custom too?

What kind of drone are you using a RX1RII with? Wingtra?

Thanks again Andrew.

Mike

Since you’re using PPK, 12”x12” is plenty big enough. Especially if you process step 1 first then pick the GCPs using ray cloud. You should be within a few inches before you pick the first point.

I’m flying a 110 ac industrial construction site monthly with my P4Pro and BAAM Tech PPK. The site already had 6 permanent survey monuments established before my first flight. One in each corner and two near the center. Perfect
for my needs. Initially I was placing my targets on the monuments before each flight. Before my last flight I used bright orange gorilla tape to make a square in the center of each monument about 4” in size. At 300 ft I can easily see them with my P4Pro. Now
all I have to do is setup my base and fly.

I don’t own the Wingtra…yet. I demoed it at my construction site project. It’s soooo much faster than a multi rotor. The 42mp Sony camera is terrific. With PPK and 0 GCPs, I was able to achieve vertical accuracy of 2-3xGSD. With
3 GCPs I got close to 1xGSD. I’m extremely impressed with the Wingtra system. I just need to get more work to justify it.

With my P4Pro at 200 ft it takes 80-90 minutes and 5 batteries to fly this site. With the Wingtra I can fly at 400 ft and takes 20 minutes and 1 set of batteries. As construction is progressing they have brought in several cranes
over 200 ft tall. Now I have to fly at 300 ft, which results in lower accuracy. With the Wingtra it wouldn’t be an issue.

Ok thanks again Andrew! You sound like you have a very good handle on this stuff too, so thank you again for your time. Yes the Wingtra looks great and their prices are coming down at last check.

When you fly with the Sony have you ever had a focus motor issue due to vibrations? I use typically set it to Extra Fine; Manual Exposure/Manual Focus to infinity; 1/2000; f/stop: 6-7 for sunny days; White Balance = Daylight; ISO Auto. With these settings, I get good results, but the processing time can be long in Pix4D.

Good luck on your project and thanks again!

Hi Mickael,

I have almost the same observations.
My study area was in France, I used RGF_1993_Lambert_93.
I had 8 targets that I ckecked with a GNSS Spectra SP80 in RGF_1993_Lambert_93.
I used a Phantom 4 pro UAV with PIX4Dcapture to take the images.
In Pix4D mapper, I have chosen the target n°6 as a checkpoint (this target n°6 has been selected in 18 images). The 7 others targets were the GCP. It was a test with PIX4Dmapper.

When I imported my DSM in Arcgis tool (À propos d’ArcGIS | Logiciel et services de cartographie et d’analyse) with a spatial resolution of 3cm/pixel, I checked the errors between the values of Z with the 8 targets (GNSS Spectra SP80) and the values of Z in the DSM generated with PIX4Dmapper…

nom (1)
Error Z (Z in DSM made with PIX4Dmapper - Z GNSS Spectra SP80 in cm) (2)
Error Z (PIX4D report in cm) (3)
difference (cm) (4)

(1) (2) (3) (4)
1 1.420750046 0.3 1.120750046
2bis 1.015453529 -0.4 1.415453529
3 2.408021355 -0.1 2.508021355
4 2.356291389 0.7 1.656291389
5 3.769297218 -0.7 4.469297218
6 4.583511734 1.72 2.863511734
8 4.455088615 0 4. 455088615
9 2.303889465 -0.7 3.003889465

mean 2.789037919
min 1.015453529
max 4.583511734
RMSE (Z) 3.057209821

In the PIX4Dmapper report, the RMSE (Z) is 0.005m = 0.5 cm, I found 3.05 cm

How do you explain these differences ?
How to use the results of the report ?

Thanks you
Best regards
Natacha

Hi Natacha,

It would be extremely helpful if you shared the quality report with us, so that we can take a better look.

Best,
Teodora

Hello Natacha,

Based on my review of your project reports, the Pix4D Quality Report looks correct. The key here is to remember that the Quality Report section on GCP/Check Point error after Step 1 re-optimization essentially gives an assessment of how close Pix4D can re-project to the original GNSS GCP XYZ coordinates. The Automatic Tie Point (ATP) cloud is then calculated/reconstructed and the subsequent data products are then created off of this ATP. So based on your project, I am guessing there are low-lying crops and grass (low to high grasses, etc). Depending on where you placed the GCP targets and how much space you left around the GCP targets, any adjacent natural surrounding vegetation may introduce some unwanted error in the ATP. Furthermore, in the Quality Report, you can see very large error ellipses as well, which is typical for this kind of terrain/vegetation.

This uncertainty in the aerotriangulation for elevation then translates on to the Point Cloud densification step and then on to the DSM creation step. Also, it looks like you specified “medium” smoothing, which further degrades how closely the area where the GCP is found matches the terrain at the “true” GCP target.

Anyway, horizontally you will be fine. But bringing in the down-sampled DSM into ESRI ArcMap will show values “higher” then the true GCP elevation. This is just the nature of the process. You can try to create a new surface using a 3rd party app that creates a surface from the point cloud itself. You may see the elevations a little lower and thus a little closer to the GCP elevations.

I understand your frustration though. If I had a say in the matter…I would ask Pix4D to consider incorporating a radius of influence around each project GCPs (say 1meter radius) where the ATP is forced to = the GCP elevation within the radius. Then it can calculate / reconstruct the ATP around the outer radius’ of the GCPs like normal. This would lesson the impact of existing terrain features that may throw off the ATP reconstruction around the GCPs.

I hope this helps.

Hello Michael,

“Based on my review of your project reports”

Thank you to have taken time for it.

“the Pix4D Quality Report looks correct. The key here is to remember that the Quality Report section on GCP/Check Point error after Step 1 re-optimization essentially gives an assessment of how close Pix4D can re-project to the original GNSS GCP XYZ coordinates. The Automatic Tie Point (ATP) cloud is then calculated/reconstructed and the subsequent data products are then created off of this ATP. So based on your project, I am guessing there are low-lying crops and grass (low to high grasses, etc). Depending on where you placed the GCP targets and how much space you left around the GCP targets, any adjacent natural surrounding vegetation may introduce some unwanted error in the ATP.”

Yes, it was exactly this surface.

“Furthermore, in the Quality Report, you can see very large error ellipses as well, which is typical for this kind of terrain/vegetation.”

I didn’t know it came from that.

“This uncertainty in the aerotriangulation for elevation then translates on to the Point Cloud densification step and then on to the DSM creation step. Also, it looks like you specified “medium” smoothing, which further degrades how closely the area where the GCP is found matches the terrain at the “true” GCP target.”

Ok, I finally found the best parameters to minimize my error

“You can try to create a new surface using a 3rd party app that creates a surface from the point cloud itself. You may see the elevations a little lower and thus a little closer to the GCP elevations.”

Do you mean a software like CloudCompare? Which app do you recommend to create a new surface? Is it easy to take it in hand, is there good documentation?

Thank you
Natacha

Hello Michael,

I had some questions about the M200 platform you are running the RX1RII on. Did you make the custom gimble or is it something that can be purchased? Is is fixed or multi axis? Is the camera/gimble controlled via UGCS? Were you able to determine the cause of the internal focus motor failure? Has it happened since the repair?

I know that’s alot but would appreciate any input as I am trying to figure out how to run the RX1RII or even the Leica Q2 on something other than the Wingtra.

I also wanted to thank you and Andrew for your expertise and input on achieving high accuracy products.

Thank you,

Joshua