Support Website Contact Support Blog

[Desktop] [Pix4Dmapper 4.2.27] slow step 3 processing with full CPU usage?

Hi all,

I’m troubleshooting some unexpected behavior while processing a ~2000 photo project (covering ~40ha with 90/90 overlap) and I’m hoping someone here may have some insight. 

Step 1. Initial processing: 1 hour, 45 minutes

Step 2. Point cloud densification + 3D textured mesh generation: ~8 hours

Step 3. Orthomosaic + DSM generation: ~12 hours

 

First, does the relative allotment of time across the 3 steps make sense? That is, would you expect Step 3 to take 50% longer than Step 2?

Second, I’m noticing that my CPU is just *cranking* for the full 12 hours for Step 3. My understanding from the Pix4D Knowledge Base was that there ought to be low CPU usage for this step, and that the hard disk defined the processing speed. I’m hardly noticing any activity at all on the hard disk (assessing this using Windows 10 Task Manager), while the CPU is at 100% (and 90 degrees Celsius; eep!).

I’m running this project on an Alienware Aurora R7 with the following specs:

CPU: Intel 6-core 8700K

GPU: NVIDIA GeForce 1080Ti

RAM: 64GB @ 2666MHz

SSD: 512 GB Samsung NVMe (Pix4D, the project, and the original image files are all stored on this disk)

 

Any thoughts would be greatly appreciated!

Does anyone have any insights into this behavior?

I updated the BIOS on my machine, and it does seem to run less hot but it still runs at 100% CPU usage on Step 3 (specifically the “Generate orthomosaic [group 1]” substep) with 0% disk usage. I’m also noticing the memory and GPU usage are each around 40% during this step.

Pix4D, the original photos, and the project being run are all on the SSD which is also the boot disk. I understood that the disk would be under much heavier load during Step 3 (with other hardware under “low usage”), so I’m trying to figure out if there’s something in my workflow that I ought to check in to.

 

Hi Michael!

I find your problem very interesting because it seems like the computer hardware is balanced and at the first glance your setup seems fine.

It is hard to say what processing time should be for a given project, but definitely Step 3 seems to be taking surprisingly long. Processing time is not only related to hardware and number of pictures but also the image resolution and processing options.

One potential thing to check is to see if your GPU driver is up to date. You can check for the newest version for your driver on NVIDIA website. Another one is to check in your log file how much RAM do you have available for a processing. This is the total RAM available to Pix4D during processing that is not being used by other applications. You should ideally not be using any other resource-intensive applications while processing if possible.  

Let me know how it works and if you won’t see any results consider opening a support ticket and include the quality report and log file.

1 Like

Thanks for the feedback!

It does appear that all of (or at least most of) my RAM is being allocated for processing. In the “Notifications and Resources” section of “Processing Options”, the maximum number of cores and of RAM is selected for use (i.e., the slider bars are at their max). On the log itself, it does appear that all 64GB are available for processing. A few different lines reference “Total RAM: 64GB Available RAM: XXGB” where XX is between 52 and 56 depending on the %of RAM currently being used, which I suppose makes sense. If there are 64GB total, and there is 20% RAM usage, there’d be ~52GB left…

I had updated my GPU driver using the dell.com/support page, but upon checking the NVIDIA page I found a more recent driver. I updated that and am currently running a project. I’ll see if the behavior changes at all and will report back regardless! Thanks for the tip.

One thing I did notice is that running pix4dmapper.exe from the command line sped up processing on my last project. Step 1 took a similar amount of time as before (on the long side of similar sized projects I’ve been running-- about 2.5 hours). Step 2 was faster (6 hours versus 8 hours) and Step 3 was faster as well (9 hours versus 12+ hours). Of course, that could have been because the objects in the imagery were less complex (all forested areas, but the project that went a bit faster was also a thinner forest).

Edit:

I’ll add that the imagery is complex vegetation over complex terrain, so I would expect processing times to be relatively long compared to similarly sized/densely photographed projects. The camera is the DJI Zenmuse X3, a 12MP camera.

Edit 2:

I’m also making sure to not run any other tasks during processing.

Hi Michael,

I’m waiting for your reply. We have noticed that the outdated driver has a huge impact on processing and sometimes even it crashes the software so I’m very curious how the software behaves after updating it. 

Great! I first updated my GPU driver to try to fix Pix4D crashing during Step 2 (following advice from here: https://community.pix4d.com/t/2800-pix4d-desktop-crashes), but I only updated it to the most recent version on dell.com/support, rather than the actual most recent version.

I’ll keep you posted on how the fully updated GPU driver affects performance.

Right now the project I started yesterday is still running. According to the quality report, Step 1 took 2.5 hours, Step 2 took 6.25 hours, and Step 3 has been running so far for about 13 hours. The substep it is on (“Generate orthomosaic [group 1]”) is 14/16 complete.

If you think it would be really helpful, I can rerun a project I ran a couple of days ago with just the updated driver being the difference between the runs. Otherwise, I will keep cranking through the mountain of projects I need processed and can submit quality reports/logs from different projects in an open ticket if that is equally helpful (or only a little less helpful).

Thanks for your attention to this!

 

Edit

An update: The project finished, with Step 3 taking 15.5 hours. Anything else I ought to try? Perhaps this is the expected amount of time for a project like mine? I’m happy to open up a ticket if you think that would help uncover anything!

 

Edit 2

I’m running another project that appears to be taking a similar amount of time. ~1.5 hours for Step 1, and Step 2 has been running for ~8 hours so far (and ought to finish soon).

 

For posterity: I was still having trouble keeping my CPU temperatures down on my Alienware Aurora R7 machine, so I did 2 things–

  1. I used the Alienware Command Center Thermal Controls to make the fan speeds under manual control, and set a fan curve that makes the top fan run at ~85% when the CPU gets to 70 degrees Celsius.

  2. I turned off the “Always Ready Mode” following advice from here (https://na.alienwarearena.com/ucf/show/1465512/boards/technical-support-1/ForumPost/urgent-alienware-aurora-r5-overheating-as-fans-go-idle) which was making my fans go idle if I stopped moving my mouse for >10 minutes even though Pix4D was still working my CPU at 100%.

Coming back to this: I seem to have much more success with running the steps via the command line separately.

My steps are:

Create a new project, load all of the pictures into it, then select the Processing Options that I want (which I’ve already saved as a template). I’m implementing all the advice from this page to help with processing images with lots of dense vegetation (https://support.pix4d.com/hc/en-us/articles/202560159-How-to-improve-the-outputs-of-dense-vegetation-areas-). For the Point Cloud Densification, I’m setting the “image scale” to 1/2 and using the “multiscale” option.

Next, I’ll close out of Pix4DMapper, run cmd to bring up a shell and follow advice from this page (https://support.pix4d.com/hc/en-us/articles/202560329-How-to-use-the-command-line) to run just the initialization step, then just the point cloud densification step, then just the orthomosaic step.

pix4dmapper -c -i D:\mining\quarry.p4d

*wait for Step 1 to finish*

pix4dmapper -c -d D:\mining\quarry.p4d

*wait for Step 2 to finish*

pix4dmapper -c -o D:\mining\quarry.p4d

Times for Step 3 are dramatically faster (5 to 8 hours instead of 8 to 15 hours), and I don’t *think* (but can’t be certain) that it’s related to less complex projects.

This was all with Pix4DMapper version 4.2.27, not the recently released 4.3

Hope this helps someone. 

Hi Michael!

I’m glad that you were able to receive better results using the command line!

Just for confirmation, am I right saying that you processed the same project twice; normally and via the command line with the same processing options but also in the meantime you updated the GPU drivers?

The reducing time is impressive and I’m happy to learn more through your case.

Cheers! 

 

 

 

Hi Beata,

Unfortunately, I didn’t do the test of re-processing the same project using the GUI and then the command line interface. Rather, I’m working through processing 36 different projects with similar characteristics and seemed to notice improvement in timing by switching to the command line interface compared to some of the previous projects that I processed with the GUI.

So I can’t say for certain that the CLI is speeding things up, but the average time across several projects is much less.

Mike

Hi Micheal,

Thank you for your reply. I’ll make an attempt to discover the reason why processing via command line is much faster.

I’ll keep you apprised of the situation in case I’ll find out what is the cause.

Cheers!