Matic is processing 4500 image job slow

xzamora · September 26, 2022, 3:52pm

Trying to process a 4500 image job that is about 120 acres or so.
My PC specs
Processor: Intel(R) Core™ i9-10900X CPU @ 3.70GHz 3.70 GHz
RAM: 128 GB
Graphics: NVIDIA GeForce RTX 3080 Ti (Total memory 77678 mb)

No problem uploading pictures or GCPS. No problem with the initial calibrate so I can tie points.
Once I try to process densify, mesh, ortho, and dsm with all exports included it takes a handful of hours to reach 60% or so. Never saw the full amount of time because my PC crashes before it finishes.
Trying to process again as I type this and it is barely at 5% in about a 60 min or so time span.

My processing settings after I tied my GCPS
Calibrate: Large scale and corridor; Scalable Standard; Image Scale 1/1; Keypoints Auto at 10000; Internals Confidence Low
Densify: Hardware Accelerated; Image Scale 1/2; Multiscale On; Density Optimal at 3; Noise filter Off; Sky filter Off
Mesh: Aerial; Texture size 8192x8192; deghosting Weak; Sky Mask Off
DSM: Surface smoothing at 2, Interpolation On
Ortho: Deghosting and oblique Off
With all exports active, Quality report, Dense point cloud, mesh, DSM as compression, Ortho as GeoTIFF

I have a 35mm Zenmuse P1
I was also originally saving the project folder to an external device via USB but changed location to my PC to see if it was a USB lag. Still having the same problem regardless of saving it to my PC
Anything I can do to reduce my processing time? Most of our projects are about of this size

Thank you
Richard

Rosana_Lin · September 27, 2022, 3:59am

Hi @xzamora

May I ask if you had enough disk space?

The minimum requirement of disk space for a project with (2,000-5,000 images at 20 MP) is 100 GB - 200 GB free space.

Best regards,
Rosana (she/her)

xzamora · September 30, 2022, 7:03pm

Yes I am using a 2TB external to save this data. I have 690GB free space on my (C:) drive and initially tried it through my (C:) drive but it crashed. So I am currently trying it on an external

Andrew_Milanes · October 1, 2022, 1:07pm

External usb drive is definitely the bottleneck. Just add another 2tb nvme drive to your workstation. Only cost about $250.

Andrew_Milanes · October 2, 2022, 1:36pm

Also, 5% completion in 1 hour isn’t bad. At that rate it will only take 20 hours to process.

xzamora · October 3, 2022, 11:38pm

How long would you say it should take to just calibrate 4500 images? I feel 3-4 hours is a lot longer than what I see it takes other peoples’ projects to process.

Rosana_Lin · October 4, 2022, 3:44am

Hi @xzamora

May I ask if one of the 2TB external or the (C:) drive happen to be a SSD?

SSD would speed up the processing much better than HDD.

Also, is the deghosting option enabled? If this is enabled, it increases the processing time.

Best regards,
Rosana (she/her)

Andrew_Milanes · October 4, 2022, 1:20pm

For reference, here is the quality report for a project I just finished yesterday.
StTammany_Lakeshore_20220930-quality_report.pdf (22.6 KB)

xzamora · October 4, 2022, 10:06pm

Yes my external is a sandisk solid state drive 2TB

My deghosting is disabled. My computer crashes on the densify step. I have the noise and sky filter disabled. 1/2 image scale hardware accelerated.

Thanks for the help

Jonathan_Dennis · October 5, 2022, 8:51am

Hi xzamora,

I see that most of the time has been spent in densification. Can you please share the log file? Are you also able to comment on, or share the image content? My first suspition is that the image content is dictating the processing time.

Are you running windows 11? I see there is an issue with Windows 11 updates slowing down GeForce GPUs.

Using an external drive is not optimal, but understand why employed. Agree with Andrew with recommendation to install a larger/faster C drive.

Regards,
-Jon

xzamora · October 5, 2022, 3:12pm

Hey Jonathan,
My log files are too large to share on here. I’m processing 4500 images. My camera is a Zenmuse P1 45mp
I am currently running windows 10pro
Would you be able to recommend a good nvme drive? Currently have a 1TB nvme KIOXIA but only have 600GB free storage left
Thanks for all the help

xzamora · October 5, 2022, 3:14pm

Thanks for the support Andrew. Currently asking our IT company to order us another nvme drive.
Currently have a 1TB nvme KIOXIA with only 600GB free storage

Andrew_Milanes · October 6, 2022, 12:01am

I would request a 2tb Samsung 980 Pro or 4tb Seagate Firecuda 530 nvme. The Samsung is about $250 and the Seagate is around $700.

scotty_wilkings · May 26, 2023, 1:42am

Those speeds aren’t bad, Your setup is only as good as your weakest link I.e. your external drive. My set up is as follows

i9 13900k

128gb ddr5 @6300

7900xtx 24gb

2tb sony 990 PCie gen 4 operating disk

2x 4tb crucial p3 PCie Gen 3 storage disk

msi z790 carbon

loads of cooling etc etc

I’m looking at upgrading my storage drives, if you monitor your hardware use at each stage of the processing you can find your weak points. Obviously my CPU clocks out for most of the process, the next component that clocks out next is my storage drive as data is pulled and written on the disk. Hence upgrading to some PCIe Gen 4 disks. Very rarely to i reach near capacity of my memory or GPU, albeit anything greater than 128gb memory on a i9 is useless.

I think your times aren’t to bad, I am currently processing a 4730 image project with data acquired through a P4RTK. times as follows…

calibration

large scale corridor

scalable standard

image scale 1/1

keypoints auto

confidence low

Enabled ITPS

time 1hr 41m

Rematch and Reoptimize

18m 41s

Densify point cloud

ROI: disabled

Image Scale 1/2

Multiscale enabled

density: optimal

Minimum matches: 3

Noise: enabled

Sky: disabled

Processing time 4hr 32m

The rest is currently ongoing and will report back with times and results. It seems that the following steps “generate wireframe”“generate DSM” “export point cloud” etc are moving quite quickly. My plan is to process the same job on mapper and compare the results. I am a bit sceptical about the ability and results from matic so far.

Andrew_Milanes · May 26, 2023, 9:51pm

Scotty, Matic also uses nvidia GPUs for processing. You would see a substantial improvement in various steps, such as ortho generation, if you switched from AMD to nvidia GPU.

Rosana_Lin · May 29, 2023, 2:10am

Hi Scott,

Indeed like Andrew said, an AMD GPU does not have the Cuda core for PIX4Dmatic to leverage the most out of it. Only NVIDIA GPUs have Cuda cores. Some users had observed a slower processing speed when using an AMD GPU.

However, those using an AMD GPU can still set up the graphic settings to ensure the GPU’s resources are being given to PIX4Dmatic as a priority. You can follow how to set this up in the article below, which provides step-by-step instructions.

How to set PIX4Dmatic to use a discrete graphics card (Windows 10 and Windows 11)

Other good practices to ensure the hardware resources are being prioritized to a photogrammetry processing software (i.e., PIX4Dmatic) include :

Close 3-rd party software such as another GIS software or gaming software (software that also consumes the majority of hardware resources). You can check this in Task Manager with the Ctrl+Alt +Delete keyboard shortcut. The Ctrl+Alt +Delete keyboard shortcut brings up the Task Manager window or a drop-down menu that allows a user to see the status of all currently running programs.
Close all Google Chrome windows as they tend to consume RAM/memory very much.

If others also have tips on increasing processing time, feel free to share them. I love to hear about them!

Best regards,
Rosana (she/her)