Large Dataset Crashes Pix4D

Hello,

I am working on a multipart project that has 3 subprojects with roughly 2,600 images each. I initially tried to run the project in one chunk that had about 8,000 images but Pix4D would crash when generating the quality report after step 1. So I broke the project into 3 subprojects that have all been processed with good quality reports.

Now that I have created three subprojects with MTPs for each project the software seems to crash when generating the quality report at the end of step 1. I decided to skip the quality report and run step 2 but now Pix4D throws an error in the processing pane (attached below).

For context, this project was flown on the same day, same altitude and with the same gimbal angle. The sensor used was the Zenmuse P1.

Any suggestions on how to merge these projects?

Hello,
It is difficult to say what could be causing the crash. The P1 is a very high resolution sensor and with 8000 images you are working with a very large dataset. This will be pushing your computer’s resources to the limit. Having sufficient RAM will be very important in this case. However we will need to look at the quality report and log files to understand what is causing this. You can upload them here or alternatively you can open a support ticket at:
support.pix4d.com

Hi Mike,

Our machine is running on 128 GB of ram. I am unable to generate a quality report to share. How would I access the log files? Once I access the log files I will be unable to share them on the forum, but I will happily share them through a support ticket.

Hi sungliengk,
Definitely open a support ticket and one of our support engineers will walk you through the process of extracting the log file.

Pix4DMapper can handle 8,000 images in one project. I’ve processed up to 11,000 42mp images at once. Be sure to uncheck generate orthomosaic preview in quality report option. That will cause a crash for large projects.

Hi @sungliengk

Have you tried running your project with PIX4Dmatic? This should work with the hardware you have.

Tip: for P1 images, I would set the setting “image scale” to “1/2” instead of “1/1” in the “calibrate” step.

We have not tried Pix4Dmatic. Would we potentially run into a RAM issue processing the whole project at once? I was reviewing the log file from the current project and we seem to max out on ram during the densification in step 2.

If you use the default settings for the “Densify” step, I would think it should be alright with the 128Go RAM you have. I would be more worried about the hard drive space, I would make sure there are a few hundred GB storage space available and that you’re running all that on an SSD drive. It’s worth a try in any case.

That makes sense. We have several hundred GB of hard drive space available. I revisited the log and it looks like Pix4D Mapper crashed at the ‘Computing scale pyramid…’ step. Is this a project we could process on the Pix4Dmatic trial software?

Yes, I would give it a go. The “trial” is a limited time license (15 days) of the full PIX4Dmatic license, it’s meant for doing tests and checking if the product works as expected before a potential purchase, so it’s all about trying projects out. Let me know if you try it! I’m curious to know if it works for you.

I have our project processing on Pix4Dmatic currently. Since I am not using any GCPs will my report still give me a RMS error? We are relying on the RTK accuracy of the M300.

Currently, we do not have RMS error on the camera positions only in the software or the quality report. We would like to expand on such functionality during next year. Note that it is possible to use “Checkpoints” only, which don’t influence the result but help you check the accuracy. This is different from a Ground Control Point (GCP) which would be taken into account in the processing. One way to check the reconstruction quality in case you don’t have either of these, would be to add MTPs across the project and check the reprojection the have, if it’s below 1px it should be fine.

Hi Pierangelo,

Thank you for all of your support while we explore Pix4Dmatic. We have been able to successfully process our project with some great results. I do have one new question though. Do you have any documentation on what is within the ‘processing’ folder of the project? To better understand our file management it would be great to know what the elements are within the project folder.

Thanks again for your assistance, Pix4Dmatic has been a great tool so far.

Hi @sungliengk, thanks for your feedback, I’m glad it was useful and that you got some great results! We have an article that documents our “outputs”: https://support.pix4d.com/hc/en-us/articles/360044950312-Outputs. The outputs are saved by default in the “exports” folder of the project, but you can adjust this in the export settings. Is this what you were looking for?

Hi @Pierangelo_Rothenbuhler, I was more looking for clarification on what exactly is inside the processing folder in the project. I’ll attach an image of the folders I am curious about.

One additional question. Is there a save as option in Pix4Dmatic? Our workflow requires us to move projects between machines and I am concerned the file paths will be disrupted if I copy and past projects. Any suggestions on how to address this?

Screenshot 2021-10-29 084744

Hi @sungliengk

These folders contain files that are generated by PIX4Dmatic after each processing step, these are the basis for the interaction with outputs inside of PIX4Dmatic as well as for any export one creates:

  1. “reconstructions” corresponds to the “Calibrate” step in the processing options.
  2. “densifications” corresponds to the “Densify” step.
  3. “dsms” corresponds to the “DSM” generation step.
  4. “orthomosaics” corresponds to the “Orthomosaic” generation step.

For example, the “densifications” folder typically has the “.bpc” file of the dense point cloud you see in PIX4Dmatic. The “.bpc” file is a proprietary format from Pix4D that ensures point clouds can be loaded quickly and efficiently in PIX4Dmatic, so that it is easier to interact with point clouds (even huige ones) in PIX4Dmatic. The same idea happens for the “.p4tiff” in the dsms and orthomosaics folders.

As for the “save as”, we do not have this at the moment. We plan to add a way to make a backup of your project in the future, which should address this question. Right now, if you move the entire project folders, it should work alright as there is a mechanism to “link” your images with your project. You can try it out on a small project:

  • When the project is closed, move the project folder to another place.
  • Open the project in PIX4Dmatic (e.g File > Open)
  • You will see icons in the image viewer that show the images are not linked.
  • Right click on the images > “Find images”
  • Select the images folder
  • You’re good to go :slight_smile:

Let me know if this is not working for you. Hope the information above is what you were looking for.

Hi @Pierangelo_Rothenbuhler,

This is all great information that helps clarify my questions. I really appreciate your help in navigating Pix4Dmatic.

1 Like