I am trying to run densification on a project with 3250 images. I’m running it on a machine with 24cpus and 110GB RAM. The settings I need to use are:
Algorithm: Hardware accelerated (Nvidia T4 installed)
Image scale: 1/1
Multiscale: Off (I’d rather have accurate points at the 1/1 scale than more points)
Min matches: 6 ( I care more for accurate positions than more points of lower accuracy)
Noise filter: On
Sky filter: On
history.p4mpl
// Wed May 10 19:33:02 2023
CommandGraph {
GenerateSkyMasks {
}
GenerateGaPmvsDensePointCloud {
settings.imageDownscaleFactor : 0
settings.pointDensity : PointDensity.High
settings.minNbrOfMatches : 6
settings.multiScale : false
settings.denoise : true
settings.skyless : true
resources.useGpu : GpuUsage.Automatic
}
}
// Thu May 11 07:28:01 2023
SaveProject {
}
// Thu May 11 07:47:55 2023
CommandGraph {
GenerateSkyMasks {
}
GenerateGaPmvsDensePointCloud {
settings.imageDownscaleFactor : 0
settings.pointDensity : PointDensity.High
settings.minNbrOfMatches : 6
settings.multiScale : false
settings.denoise : true
settings.skyless : true
resources.useGpu : GpuUsage.Automatic
}
}
// Thu May 11 15:42:50 2023
SaveProject {
}
After about 4 hours I am getting the following error:
An unknown error occured
bad allocation
What sucks the most about this error is that none of the 4 hours of work gets saved, so the Densification needs to be restarted from scratch.
I read in a different topic that the issue might be that there isn’t enough RAM, but according to the logs below, the RAM utilization was never at higher than 80% and when the error happened it was at about 50%. Does anyone have any idea of what the issue/solution could be?
Obviously, I could just try different settings, perhaps lower ones, but unfortunately, I’m not getting good enough results with lower settings, so that’s not a good solution for my situation.
Thinking that it may be simply configuration drifting, or a random error, I tried it twice, with almost identical results. Both failed a little bit after 4 hours.
Log1
[2023-05-10 19:33:02][00%CPU][11%RAM][Processing] Running
[2023-05-10 19:33:02][00%CPU][11%RAM][Generate sky masks] Running
[2023-05-10 19:33:02][00%CPU][11%RAM][Generate sky masks](Info) All sky masks are available, the sky masks will not be generated again.
[2023-05-10 19:33:02][00%CPU][11%RAM][Generate sky masks] Finished
[2023-05-10 19:33:02][00%CPU][10%RAM][Generate sky masks](Success) Generation of sky masks finished
Processing time 0s
[2023-05-10 19:33:02][00%CPU][11%RAM][Densify sparse point cloud] Running Region of interest: Disabled
Image scale: 1/1
Multiscale: Disabled
Density: High
Minimum matches: 6
Noise filter: Enabled
Sky filter: Enabled
[2023-05-10 19:33:03][00%CPU][11%RAM][Densify] Running [free storage: 352.32 GB]
[2023-05-10 19:33:06][08%CPU][13%RAM][Densify](Info) Arguments validated.
[2023-05-10 19:33:06][08%CPU][13%RAM][Densify](Info) running pix4d densify 3.8.2
[2023-05-10 19:33:06][08%CPU][13%RAM][Densify](Info) densifying from 3250 cameras and 23292230 seeds
[2023-05-10 19:33:06][08%CPU][13%RAM][Densify](Info) Resources: {"gpuMemory":10970431488,"hostMemory":78726930432,"useGPU":true}
[2023-05-10 19:33:06][08%CPU][13%RAM][Densify](Info) Settings: {"depthLimitPercentile":0.949999988079071,"imageScale":0,"limitDepth":true,"minImageSize":512,"minNoMatch":6,"multiScale":false,"partitionInputScene":true,"patchesFiltering":true,"pointDensity":1,"regularizedMultiscale":true,"uniformityThreshold":0.03125,"windowSize":7}
[2023-05-10 19:33:07][08%CPU][13%RAM][Densify](Debug) Parameters: {
"cluster": {
"maxClusterSize": 450,
"minSharedPointCount": 10
},
"exp": {
"cellSize": 1,
"filterPatches": true,
"maxProcessingPatchCount": 65536,
"maxSelectionOverlap": 0,
"minExpansionRate": -1.0,
"minFilledCells": 6,
"minProcessingPatchCount": 1024,
"qualityMetrics": "entropyLevel",
"selectionPercentile": 0.75,
"switchToCPUThreshold": 1000
},
"filter": {
"maxDepth": 471.6746520996094,
"visibilityFiltering": true
},
"image": {
"downscalingKernel": [
1,
4,
6,
4,
1
],
"softRankFilterWindow": 7
},
"init": {
"camSelectionMethod": "first",
"cameraSelectionBatchSize": 9,
"cosMaxAngle": 0.17364817766693041,
"cosMinAngle": 0.9998476951563913,
"entropyThreshold": 0.03125,
"entropyThresholdUpperBound": 0.125,
"maxCandidateRefCam": 5,
"maxProjectionsCount": 9,
"minProjectionsCount": 3,
"regularize": false,
"znccThreshold": 0.699999988079071
},
"opt": {
"epsilon": 1e-05,
"finiteDiffEpsilon": 0.0003162277571391314,
"gradTolerance": 1e-06,
"initialTrustRegion": 100.0,
"maxIterations": 10,
"numDiffMethod": "forward",
"paramTolerance": 0.0001
},
"score": {
"anchor": "center",
"dMethod": "mad",
"gridSize": 7,
"imageLevel": 0,
"levelsCount": 1,
"marginPixels": 0
},
"seedInRoiRatioThreshold": 0.1
}
[2023-05-10 19:33:07][08%CPU][13%RAM][Densify](Debug) host memory forecast: 19044.982570
[2023-05-10 19:33:07][08%CPU][13%RAM][Densify](Info) Checking input seeds reprojections...
[2023-05-10 19:33:12][04%CPU][13%RAM][Densify](Debug) 0.000000% of projections from camera (declared as visible) to seed point are invalid.
[2023-05-10 19:33:12][04%CPU][13%RAM][Densify](Debug) Task: [densification]/[seeds reprojection check] finished.
[2023-05-10 19:33:12][04%CPU][13%RAM][Densify](Info) Checking input seeds reprojections... done
[2023-05-10 19:33:12][04%CPU][13%RAM][Densify](Info) Region-of-interest input filtering...
[2023-05-10 19:33:17][04%CPU][13%RAM][Densify](Debug) Task: [densification]/[region of interest filtering] finished.
[2023-05-10 19:33:17][04%CPU][13%RAM][Densify](Info) Region-of-interest input filtering... done
[2023-05-10 19:33:17][04%CPU][13%RAM][Densify](Info) Create overlapping views camera graph...
[2023-05-10 19:33:29][04%CPU][13%RAM][Densify](Debug) Task: [densification]/[overlapping views camera graph creation] finished.
[2023-05-10 19:33:29][04%CPU][13%RAM][Densify](Info) Create overlapping views camera graph... done
[2023-05-10 19:33:29][04%CPU][13%RAM][Densify](Info) Create global scene...
[2023-05-10 19:33:31][02%CPU][14%RAM][Densify](Debug) Task: [densification]/[global scene creation] finished.
[2023-05-10 19:33:31][02%CPU][14%RAM][Densify](Info) Create global scene... done
[2023-05-10 19:33:31][02%CPU][14%RAM][Densify](Info) creating densification clusters...
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Debug) Task: [densification]/[cluster creation] finished.
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) creating densification clusters... done
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) Created 8 clusters out of 3250 cameras with cameras counts [418, 418, 416, 415, 399, 396, 394, 394, ] (0% overlap) and seeds count [1330308, 1131638, 1294120, 1164313, 1280444, 1372203, 1343523, 1163137, ]
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) Scene [1/8] extracting initial seed-scene...
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) Scene [1/8] extracting initial seed-scene... done
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) Scene [1/8] Extracted initial seed-scene with 418 cameras and 1330308 points.
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) computing look-up tables...
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) computing look-up tables... done
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[computing LUTs] finished.
[2023-05-10 19:33:39][04%CPU][14%RAM][Densify](Info) loading 418 images...
[2023-05-10 19:35:51][04%CPU][20%RAM][Densify](Info) loading 418 images... done
[2023-05-10 19:35:51][04%CPU][20%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[loading images] finished.
[2023-05-10 19:35:51][04%CPU][20%RAM][Densify](Info) initializing scene expansion...
[2023-05-10 19:40:06][84%CPU][26%RAM][Densify](Info) initializing scene expansion... done
[2023-05-10 19:40:06][84%CPU][26%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[seed initialization] finished.
[2023-05-10 19:40:06][84%CPU][26%RAM][Densify](Info) expanding from 1054641 seeds...
[2023-05-10 23:08:22][80%CPU][43%RAM][Densify](Debug) generated 178757451 points, 27.9128% cells filled
filterPatches: 1.1365e+06ms
finalizePatches: 3.0468e+06ms
initializePatches: 3.06575e+06ms
makeNeighborCandidatePatches: 412238ms
optimizePatches: 2.18566e+06ms
outputPatches: 13670.8ms
selectCandidatePatches: 1.68627e+06ms
maximum number of seeds in queue: 23972818
[2023-05-10 23:16:50][04%CPU][53%RAM][Densify](Info) expanding from 1054641 seeds... done
[2023-05-10 23:16:52][04%CPU][53%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[expansion] finished.
[2023-05-10 23:16:52][04%CPU][53%RAM][Densify](Info) Scene [1/8] expanded to 178757451 points.
[2023-05-10 23:16:52][04%CPU][53%RAM][Densify](Info) filtering 178757451 points...
[2023-05-10 23:37:12][84%CPU][65%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[post computation]/[filter patches] finished.
[2023-05-10 23:37:12][84%CPU][65%RAM][Densify](Info) filtering 178757451 points... done
[2023-05-10 23:37:12][84%CPU][65%RAM][Densify](Info) colorizing 163194414 points...
[2023-05-10 23:41:31][46%CPU][74%RAM][Densify](Info) colorizing 163194414 points... done
[2023-05-10 23:41:31][46%CPU][74%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[post computation]/[colorization] finished.
[2023-05-10 23:41:31][46%CPU][74%RAM][Densify](Info) Scene [1/8] colorized 163194414 points.
[2023-05-10 23:41:31][46%CPU][74%RAM][Densify](Info) returning 163194414 generated points...
[2023-05-10 23:43:07][05%CPU][52%RAM][Densify](Info) returning 163194414 generated points... failed
[2023-05-10 23:43:09][17%CPU][35%RAM][Densify](Error) Exception thrown in callback or sink. Re-throwing
[2023-05-10 23:43:09][17%CPU][35%RAM][Densify] Error
[2023-05-10 23:43:11][17%CPU][35%RAM][Densify sparse point cloud](Error) An unknown error occurred
bad allocation
[2023-05-10 23:43:11][17%CPU][35%RAM][Create point cloud] Cancelled
[2023-05-10 23:43:11][17%CPU][35%RAM][Densify sparse point cloud] Error Region of interest: Disabled
Image scale: 1/1
Multiscale: Disabled
Density: High
Minimum matches: 6
Noise filter: Enabled
Sky filter: Enabled
[2023-05-10 23:43:11][76%CPU][35%RAM][Densify sparse point cloud](Error) Densification failed
Processing time 4h 10m 8s
[2023-05-10 23:43:15][76%CPU][11%RAM][Processing] Finished
[2023-05-10 23:43:15][76%CPU][11%RAM][Processing] Error
[2023-05-10 23:43:15][76%CPU][11%RAM][Processing](Error) Processing failed
Log2
[2023-05-11 07:47:55][00%CPU][13%RAM][Processing] Running
[2023-05-11 07:47:55][00%CPU][13%RAM][Generate sky masks] Running
[2023-05-11 07:47:55][00%CPU][13%RAM][Generate sky masks](Info) All sky masks are available, the sky masks will not be generated again.
[2023-05-11 07:47:55][00%CPU][13%RAM][Generate sky masks] Finished
[2023-05-11 07:47:55][00%CPU][12%RAM][Generate sky masks](Success) Generation of sky masks finished
Processing time 0s
[2023-05-11 07:47:55][00%CPU][13%RAM][Densify sparse point cloud] Running Region of interest: Disabled
Image scale: 1/1
Multiscale: Disabled
Density: High
Minimum matches: 6
Noise filter: Enabled
Sky filter: Enabled
[2023-05-11 07:47:55][00%CPU][13%RAM][Densify] Running [free storage: 352.31 GB]
[2023-05-11 07:47:59][07%CPU][14%RAM][Densify](Info) Arguments validated.
[2023-05-11 07:47:59][07%CPU][14%RAM][Densify](Info) running pix4d densify 3.8.2
[2023-05-11 07:47:59][07%CPU][14%RAM][Densify](Info) densifying from 3250 cameras and 23292230 seeds
[2023-05-11 07:47:59][07%CPU][14%RAM][Densify](Info) Resources: {"gpuMemory":11036983296,"hostMemory":77422273536,"useGPU":true}
[2023-05-11 07:47:59][07%CPU][14%RAM][Densify](Info) Settings: {"depthLimitPercentile":0.949999988079071,"imageScale":0,"limitDepth":true,"minImageSize":512,"minNoMatch":6,"multiScale":false,"partitionInputScene":true,"patchesFiltering":true,"pointDensity":1,"regularizedMultiscale":true,"uniformityThreshold":0.03125,"windowSize":7}
[2023-05-11 07:48:00][07%CPU][14%RAM][Densify](Debug) Parameters: {
"cluster": {
"maxClusterSize": 443,
"minSharedPointCount": 10
},
"exp": {
"cellSize": 1,
"filterPatches": true,
"maxProcessingPatchCount": 65536,
"maxSelectionOverlap": 0,
"minExpansionRate": -1.0,
"minFilledCells": 6,
"minProcessingPatchCount": 1024,
"qualityMetrics": "entropyLevel",
"selectionPercentile": 0.75,
"switchToCPUThreshold": 1000
},
"filter": {
"maxDepth": 471.6746520996094,
"visibilityFiltering": true
},
"image": {
"downscalingKernel": [
1,
4,
6,
4,
1
],
"softRankFilterWindow": 7
},
"init": {
"camSelectionMethod": "first",
"cameraSelectionBatchSize": 9,
"cosMaxAngle": 0.17364817766693041,
"cosMinAngle": 0.9998476951563913,
"entropyThreshold": 0.03125,
"entropyThresholdUpperBound": 0.125,
"maxCandidateRefCam": 5,
"maxProjectionsCount": 9,
"minProjectionsCount": 3,
"regularize": false,
"znccThreshold": 0.699999988079071
},
"opt": {
"epsilon": 1e-05,
"finiteDiffEpsilon": 0.0003162277571391314,
"gradTolerance": 1e-06,
"initialTrustRegion": 100.0,
"maxIterations": 10,
"numDiffMethod": "forward",
"paramTolerance": 0.0001
},
"score": {
"anchor": "center",
"dMethod": "mad",
"gridSize": 7,
"imageLevel": 0,
"levelsCount": 1,
"marginPixels": 0
},
"seedInRoiRatioThreshold": 0.1
}
[2023-05-11 07:48:00][07%CPU][14%RAM][Densify](Debug) host memory forecast: 18752.513519
[2023-05-11 07:48:00][07%CPU][14%RAM][Densify](Info) Checking input seeds reprojections...
[2023-05-11 07:48:05][04%CPU][14%RAM][Densify](Debug) 0.000000% of projections from camera (declared as visible) to seed point are invalid.
[2023-05-11 07:48:05][04%CPU][14%RAM][Densify](Debug) Task: [densification]/[seeds reprojection check] finished.
[2023-05-11 07:48:05][04%CPU][14%RAM][Densify](Info) Checking input seeds reprojections... done
[2023-05-11 07:48:05][04%CPU][14%RAM][Densify](Info) Region-of-interest input filtering...
[2023-05-11 07:48:10][04%CPU][14%RAM][Densify](Debug) Task: [densification]/[region of interest filtering] finished.
[2023-05-11 07:48:10][04%CPU][14%RAM][Densify](Info) Region-of-interest input filtering... done
[2023-05-11 07:48:10][04%CPU][14%RAM][Densify](Info) Create overlapping views camera graph...
[2023-05-11 07:48:23][04%CPU][14%RAM][Densify](Debug) Task: [densification]/[overlapping views camera graph creation] finished.
[2023-05-11 07:48:23][04%CPU][14%RAM][Densify](Info) Create overlapping views camera graph... done
[2023-05-11 07:48:23][04%CPU][14%RAM][Densify](Info) Create global scene...
[2023-05-11 07:48:26][02%CPU][15%RAM][Densify](Debug) Task: [densification]/[global scene creation] finished.
[2023-05-11 07:48:26][02%CPU][15%RAM][Densify](Info) Create global scene... done
[2023-05-11 07:48:26][02%CPU][15%RAM][Densify](Info) creating densification clusters...
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Debug) Task: [densification]/[cluster creation] finished.
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) creating densification clusters... done
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) Created 8 clusters out of 3250 cameras with cameras counts [418, 418, 416, 415, 399, 396, 394, 394, ] (0% overlap) and seeds count [1330308, 1131638, 1294120, 1164313, 1280444, 1372203, 1343523, 1163137, ]
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) Scene [1/8] extracting initial seed-scene...
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) Scene [1/8] extracting initial seed-scene... done
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) Scene [1/8] Extracted initial seed-scene with 418 cameras and 1330308 points.
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) computing look-up tables...
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) computing look-up tables... done
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[computing LUTs] finished.
[2023-05-11 07:48:34][04%CPU][15%RAM][Densify](Info) loading 418 images...
[2023-05-11 07:50:56][04%CPU][21%RAM][Densify](Info) loading 418 images... done
[2023-05-11 07:50:56][04%CPU][21%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[loading images] finished.
[2023-05-11 07:50:56][04%CPU][21%RAM][Densify](Info) initializing scene expansion...
[2023-05-11 07:55:17][84%CPU][32%RAM][Densify](Info) initializing scene expansion... done
[2023-05-11 07:55:17][84%CPU][32%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[seed initialization] finished.
[2023-05-11 07:55:17][84%CPU][32%RAM][Densify](Info) expanding from 1054641 seeds...
[2023-05-11 11:32:51][77%CPU][48%RAM][Densify](Debug) generated 178757451 points, 27.9128% cells filled
filterPatches: 1.24237e+06ms
finalizePatches: 3.02307e+06ms
initializePatches: 3.07174e+06ms
makeNeighborCandidatePatches: 428333ms
optimizePatches: 2.18249e+06ms
outputPatches: 13339.6ms
selectCandidatePatches: 2.0886e+06ms
maximum number of seeds in queue: 23972818
[2023-05-11 11:42:05][04%CPU][64%RAM][Densify](Info) expanding from 1054641 seeds... done
[2023-05-11 11:42:06][04%CPU][64%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[expansion]/[expansion] finished.
[2023-05-11 11:42:06][04%CPU][64%RAM][Densify](Info) Scene [1/8] expanded to 178757451 points.
[2023-05-11 11:42:06][04%CPU][64%RAM][Densify](Info) filtering 178757451 points...
[2023-05-11 12:03:56][85%CPU][81%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[post computation]/[filter patches] finished.
[2023-05-11 12:03:56][85%CPU][81%RAM][Densify](Info) filtering 178757451 points... done
[2023-05-11 12:03:56][85%CPU][81%RAM][Densify](Info) colorizing 163194414 points...
[2023-05-11 12:08:20][45%CPU][84%RAM][Densify](Info) colorizing 163194414 points... done
[2023-05-11 12:08:20][45%CPU][84%RAM][Densify](Debug) Task: [densification]/[all clusters]/[cluster_0]/[post computation]/[colorization] finished.
[2023-05-11 12:08:20][45%CPU][84%RAM][Densify](Info) Scene [1/8] colorized 163194414 points.
[2023-05-11 12:08:20][45%CPU][84%RAM][Densify](Info) returning 163194414 generated points...
[2023-05-11 12:09:57][06%CPU][52%RAM][Densify](Info) returning 163194414 generated points... failed
[2023-05-11 12:09:57][06%CPU][52%RAM][Densify](Error) Exception thrown in callback or sink. Re-throwing
[2023-05-11 12:09:57][06%CPU][52%RAM][Densify] Error
[2023-05-11 12:09:58][06%CPU][52%RAM][Densify sparse point cloud](Error) An unknown error occurred
bad allocation
[2023-05-11 12:09:58][06%CPU][52%RAM][Create point cloud] Cancelled
[2023-05-11 12:09:58][06%CPU][52%RAM][Densify sparse point cloud] Error Region of interest: Disabled
Image scale: 1/1
Multiscale: Disabled
Density: High
Minimum matches: 6
Noise filter: Enabled
Sky filter: Enabled
[2023-05-11 12:09:58][74%CPU][50%RAM][Densify sparse point cloud](Error) Densification failed
Processing time 4h 22m 2s
[2023-05-11 12:10:03][74%CPU][11%RAM][Processing] Finished
[2023-05-11 12:10:03][74%CPU][11%RAM][Processing] Error
[2023-05-11 12:10:03][74%CPU][11%RAM][Processing](Error) Processing failed