I’m troubleshooting some unexpected behavior while processing a ~2000 photo project (covering ~40ha with 90/90 overlap) and I’m hoping someone here may have some insight.
Step 1. Initial processing: 1 hour, 45 minutes
Step 2. Point cloud densification + 3D textured mesh generation: ~8 hours
Step 3. Orthomosaic + DSM generation: ~12 hours
First, does the relative allotment of time across the 3 steps make sense? That is, would you expect Step 3 to take 50% longer than Step 2?
Second, I’m noticing that my CPU is just *cranking* for the full 12 hours for Step 3. My understanding from the Pix4D Knowledge Base was that there ought to be low CPU usage for this step, and that the hard disk defined the processing speed. I’m hardly noticing any activity at all on the hard disk (assessing this using Windows 10 Task Manager), while the CPU is at 100% (and 90 degrees Celsius; eep!).
I’m running this project on an Alienware Aurora R7 with the following specs:
CPU: Intel 6-core 8700K
GPU: NVIDIA GeForce 1080Ti
RAM: 64GB @ 2666MHz
SSD: 512 GB Samsung NVMe (Pix4D, the project, and the original image files are all stored on this disk)
I updated the BIOS on my machine, and it does seem to run less hot but it still runs at 100% CPU usage on Step 3 (specifically the “Generate orthomosaic [group 1]” substep) with 0% disk usage. I’m also noticing the memory and GPU usage are each around 40% during this step.
Pix4D, the original photos, and the project being run are all on the SSD which is also the boot disk. I understood that the disk would be under much heavier load during Step 3 (with other hardware under “low usage”), so I’m trying to figure out if there’s something in my workflow that I ought to check in to.
I find your problem very interesting because it seems like the computer hardware is balanced and at the first glance your setup seems fine.
It is hard to say what processing time should be for a given project, but definitely Step 3 seems to be taking surprisingly long. Processing time is not only related to hardware and number of pictures but also the image resolution and processing options.
One potential thing to check is to see if your GPU driver is up to date. You can check for the newest version for your driver on NVIDIA website. Another one is to check in your log file how much RAM do you have available for a processing. This is the total RAM available to Pix4D during processing that is not being used by other applications. You should ideally not be using any other resource-intensive applications while processing if possible.
Let me know how it works and if you won’t see any results consider opening a support ticket and include the quality report and log file.
It does appear that all of (or at least most of) my RAM is being allocated for processing. In the “Notifications and Resources” section of “Processing Options”, the maximum number of cores and of RAM is selected for use (i.e., the slider bars are at their max). On the log itself, it does appear that all 64GB are available for processing. A few different lines reference “Total RAM: 64GB Available RAM: XXGB” where XX is between 52 and 56 depending on the %of RAM currently being used, which I suppose makes sense. If there are 64GB total, and there is 20% RAM usage, there’d be ~52GB left…
I had updated my GPU driver using the Support | Dell US page, but upon checking the NVIDIA page I found a more recent driver. I updated that and am currently running a project. I’ll see if the behavior changes at all and will report back regardless! Thanks for the tip.
One thing I did notice is that running pix4dmapper.exe from the command line sped up processing on my last project. Step 1 took a similar amount of time as before (on the long side of similar sized projects I’ve been running-- about 2.5 hours). Step 2 was faster (6 hours versus 8 hours) and Step 3 was faster as well (9 hours versus 12+ hours). Of course, that could have been because the objects in the imagery were less complex (all forested areas, but the project that went a bit faster was also a thinner forest).
Edit:
I’ll add that the imagery is complex vegetation over complex terrain, so I would expect processing times to be relatively long compared to similarly sized/densely photographed projects. The camera is the DJI Zenmuse X3, a 12MP camera.
Edit 2:
I’m also making sure to not run any other tasks during processing.
I’m waiting for your reply. We have noticed that the outdated driver has a huge impact on processing and sometimes even it crashes the software so I’m very curious how the software behaves after updating it.
I’ll keep you posted on how the fully updated GPU driver affects performance.
Right now the project I started yesterday is still running. According to the quality report, Step 1 took 2.5 hours, Step 2 took 6.25 hours, and Step 3 has been running so far for about 13 hours. The substep it is on (“Generate orthomosaic [group 1]”) is 14/16 complete.
If you think it would be really helpful, I can rerun a project I ran a couple of days ago with just the updated driver being the difference between the runs. Otherwise, I will keep cranking through the mountain of projects I need processed and can submit quality reports/logs from different projects in an open ticket if that is equally helpful (or only a little less helpful).
Thanks for your attention to this!
Edit
An update: The project finished, with Step 3 taking 15.5 hours. Anything else I ought to try? Perhaps this is the expected amount of time for a project like mine? I’m happy to open up a ticket if you think that would help uncover anything!
Edit 2
I’m running another project that appears to be taking a similar amount of time. ~1.5 hours for Step 1, and Step 2 has been running for ~8 hours so far (and ought to finish soon).
For posterity: I was still having trouble keeping my CPU temperatures down on my Alienware Aurora R7 machine, so I did 2 things–
I used the Alienware Command Center Thermal Controls to make the fan speeds under manual control, and set a fan curve that makes the top fan run at ~85% when the CPU gets to 70 degrees Celsius.
Coming back to this: I seem to have much more success with running the steps via the command line separately.
My steps are:
Create a new project, load all of the pictures into it, then select the Processing Options that I want (which I’ve already saved as a template). I’m implementing all the advice from this page to help with processing images with lots of dense vegetation (https://support.pix4d.com/hc/en-us/articles/202560159-How-to-improve-the-outputs-of-dense-vegetation-areas-). For the Point Cloud Densification, I’m setting the “image scale” to 1/2 and using the “multiscale” option.
Times for Step 3 are dramatically faster (5 to 8 hours instead of 8 to 15 hours), and I don’t *think* (but can’t be certain) that it’s related to less complex projects.
This was all with Pix4DMapper version 4.2.27, not the recently released 4.3
I’m glad that you were able to receive better results using the command line!
Just for confirmation, am I right saying that you processed the same project twice; normally and via the command line with the same processing options but also in the meantime you updated the GPU drivers?
The reducing time is impressive and I’m happy to learn more through your case.
Unfortunately, I didn’t do the test of re-processing the same project using the GUI and then the command line interface. Rather, I’m working through processing 36 different projects with similar characteristics and seemed to notice improvement in timing by switching to the command line interface compared to some of the previous projects that I processed with the GUI.
So I can’t say for certain that the CLI is speeding things up, but the average time across several projects is much less.
These cookies are necessary for the website to function and cannot be switched off in our systems.
They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences,
logging in, or filling in forms. These cookies do not store any personally identifiable information.
These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site.
They help us to know which pages are the most and least popular and see how visitors move around the site.
All information these cookies collect is aggregated and therefore anonymous.
If you do not allow these cookies we will not know when you have visited our site, and will not be able to monitor its performance.
These cookies may be set through our site by our advertising partner (Google).
They may be used by Google to build a profile of your interests and show you relevant adverts on other sites.
They do not directly store personal information but are based on uniquely identifying your browser and internet device.
If you do not allow these cookies, you will experience less targeted advertising.