[Desktop / Cloud Hybrid] [Pix4Dmapper Pro 4.3.31] imagery with different spectral signatures; one block after subproject merge, two blocks after cloud processing

PIX4Dmapper PIX4Dmapper Questions/Troubleshooting
1

Hi all,

I’m trying to process a project that has both RGB and multispectral imagery. The cameras were flown simultaneously, so there is complete overlap in their coverage. The RGB camera is a DJI Zenmuse X3 and the multispectral camera is a Micasense RedEdge3, both of which are in the Pix4D database. There’s lots of front and side overlap (90% / 90% measured 100m below the X3 lens, and I flew at 120m so slightly higher than 90% ground overlap for the X3), and so I’m interested in dialing in the processing parameters. I’ve successfully processed many RGB projects, and am now looking to integrate the multispectral data that I have by setting up the right Pix4D workflow.

I’m having trouble with the proper order of events in the workflow to ensure the finished product has a single block. The short version is that I can get a single block from each camera rig separately (one block for the RGB camera and one block for the multispectral camera) by customizing the keypoints image scale during Step 1, and I can get a single block when merging the projects by using manual tie points, but I get two blocks returned in the finished product after uploading the project to the cloud for processing of Steps 2 and 3. It seems that Step 1 is redone in the cloud using one keypoints image scale for the whole project, and the different values of this parameter that I found for the different camera rigs to ensure a single block output are ignored.

What step am I missing to ensure that the correct keypoints image scale parameter for each camera rig is used when proceeding to Steps 2 and 3 of a merged project?

Thanks for your help!

 

I’ve been following the excellent tutorial on processing imagery with different spectral signatures. I’ve also been following the other excellent tutorial on merging projects. I’m using a hybrid Desktop/Cloud workflow which I’ll describe in case it’s helpful to other folks:

  1. Give all images a unique name (the X3 and RedEdge keep images with the same name distinct by putting them in different folders, so I turn the folder structure into part of the file name. E.g., 0000SET/000/IMG_0001_1.tif becomes 0000SET_000_IMG_0001_1.tif)

  2. Subset the images to just the ones taken over the survey area (remove images taken while the drone is taking off or travelling to the survey area but not yet over the survey area. I do this by extracting the latitude/longitude and altitude from the EXIF metadata in each photo and removing photos that are outside the bounds of the survey area or photos that are taken while the aircraft is not at survey altitude-- I use a bit of a buffer and include all photos where the aircraft is >100m above ground level. I do this step programmatically in R).

  3. Start a new Pix4D project for just the RGB images. Only perform Step 1, and find parameters such that a single image block is returned.

  4. Start a second new Pix4D project for just the multispectral images. *Make sure to add the calibration panel images to the project*. Only perform Step 1, and find parameters such that a single image block is returned.

Note: If you don’t add the calibration panel images to the project, you are still able to incorporate the radiometric correction by pointing the Step 3 processing option for calibration to wherever those photos live on your computer. BUT, if those photos aren’t *added as part of the project*, then they won’t be uploaded to Pix4D Cloud should you decide to use cloud processing. All the rest of your photos will upload, and then the project will break as soon as it starts on Pix4D Cloud. Since you can’t add photos to a project after it has been created, this breakage means you have to start all over creating a new mulitspectral project, merging it, reoptimizing it, and uploading it.

  1. Start a third new Pix4D project and check the box indicating that this new project will be a merge of other projects. Select the RGB and multispectral projects as the ones to merge.

Note: there’s a dialog box that pops up saying that some objects have identical names in the projects and should Pix4D treat them as the same object. There aren’t any identically named objects (like tie points or ground control points) in my projects, so I’m not sure what this is referring to. I’ve just been clicking “yes, treat them as identical”

Note 2: There’s also 3 dialog boxes that come up (one for each processing Step) saying that the processing parameters for each Step are different between the two projects. I know this is true (and part of the benefit of the merging workflow), so I’ve just been clicking “Okay”.

  1. Once the merge is complete, I follow the tutorial to mark several (3 to 4) manual tie points using the rayCloud. I ensure that I mark photos for each manual tie point that are in the RGB group and that are in the multispectral group. I’ve been marking between 20 and 40 photos per manual tie point.

Note: For me, unlike what is described in the tutorial, the green cross never matches up to the same point across all the images. Instead, the green cross matches nicely to *either* the RGB set of photos OR the multispectral set of photos. The green cross is in the vicinity, but doesn’t overlap with, the points I’ve marked as manual tie points on the *other* set of photos. This seems to be fine, and perhaps expected, given that the whole point is to tie together image blocks that are offset from each other.

  1. After marking manual tie points, I export them to a file in case I need them again and click Process > Reoptimize

  2. A quality report isn’t automatically generated after reoptimizing, so I click Process > Generate Quality Report so that I have a report that reflects the effort to tie the image blocks together.

  3. I look at the quality report to ensure a single block results from the reoptimization step.

  4. If one block is generated, I open up the Processing Options, keep the Step 1 parameters the same, and change the Steps 2 and 3 parameters to my liking. I want to generate a point cloud and a reflectance map and use the calibration photos for the index, so I fill in those parameters. I uncheck the Step 1 box, and check the boxes for Steps 2 and 3.

Note: I always leave the Step 1 parameters unchanged. I’ve found that if I change these parameters, even if the box is unchecked indicating that I don’t want to perform the Step 1 processing, then the project thinks I haven’t done Step 1 at all, the “Initial Processing” text at the bottom of the Pix4D application window turns red instead of green, and the option to jump right to Steps 2 and 3 become grayed out.

  1. I click Project > Upload Project Files and check the box indicating that I want to start cloud processing as soon as the photos are uploaded.

  2. When I download the results from the cloud processing, I look at the quality report and find that there are 2 blocks and way fewer “matches per calibrated image” compared to the quality report that I generated after merging the projects, adding manual tie points, and reoptimizing. It appears that Step 1 was redone with a single value for the keypoints image scale (ignoring that I had a different value for this parameter in each of the projects that I merged).

If I manage to get a single image block after the Step 1 procedure and merging subprojects with different spectral signatures, how can I ensure that I get a single block after sending my customized Step 2 and 3 parameters to Pix4D Cloud?

2

Any thoughts on this? Is it just not possible to incorporate different Step 1 parameters for different image sets in a Pix4D Cloud workflow?

Since writing, I had success with a different project where I still processed Step 1 on the separate subprojects and used them to create the Manual Tie Points, but then uploaded the whole thing to the cloud using the higher value of the keypoints image scale for Step 1 between the two projects. That is, I used a keypoints image scale of 0.5 for the RGB subproject and a keypoints image scale of 1.0 for the multispectral subproject, then merged them, set the Step 1 keypoints image scale parameter to 1, then uploaded to the cloud.

Still trying to figure out if that might work for the project that used a 0.5 keypoints image scale for RGB and a 2 keypoints image scale for the multispectral. I tried uploaded the merged project using 2 as the keypoints image scale for the whole project, but got worse results.

My next step is to add some more manual tie points, then set the Step 1 keypoints image scale to 1 and try uploading again.

3

Hi Michael,

Sounds really interesting. Can you write a support ticket with the quality reports of the individual projects and the merged project? That would help us investigate the issue more efficiently. 

The key point image scale is set to 2 as default in ag multispectral as the resolution of multispectral images are low. However, if there are very few features, you can reduce the scale and try. It’s like bringing a book close to your eyes (the blurry objects become clear) though the viewing area is reduced. For RGB it is kept as 1. However, if you use 0.5 for RGB and 2 for multispectral, the variation is very large. Pix4D uses the keypoint image scale to do extract keypoints or features from the images. For example, When changing the key points images scale from 1/2 to 1, more keypoints will be extracted as the software will have a larger search area. When you lower the scale, the search area will be less but will have more details.

More MTPs will be definitely a great idea. Do send us your quality reports. 

4

Hi Momtanu,

Thanks for the information. That makes sense that having keypoint image scales too different from each other might cause trouble.

Even though the RGB keypoints image scale was at 0.5 and the multispectral was at 2.0, the project merged successfully into 1 block. Here’s the quality report showing the successful Step 1 merge, even with really different keypoint image scales: https://www.dropbox.com/s/q7hamr2u8zw9rbb/eldo_3k_1_pre-upload-to-pix4d-cloud_report.pdf?dl=0

The problem was that when I uploaded this project to complete steps 2 and 3, 2 blocks were returned and it seemed like the “merge” never happened: https://www.dropbox.com/s/gb5jdmjb4mkmkbw/eldo_3k_1_0.5-keypoints-scale_report.pdf?dl=0

That is, it seems like Pix4D cloud just treated the uploaded images all the same and tried to do a Step 1 with all the images at once using the same keypoint image scale.

I think the general question I have is: are all of the benefit of merging projects after Step 1 lost if you try to do the rest of the processing using Pix4D Cloud? Can Pix4D Cloud handle merged projects?

5

Hi Michael,

We generally do not recommend merging multispectral and RGB datasets. The knowledgebase article talks about merging datasets from RGB and modified camera. This is because a modified camera is somewhat similar to RGB (only the bands can be different). They are composite images of several bands. However, multispectral images are processed as a rig.

In general, Pix4D can handle merging projects. You need to uncheck step 1 when you create the merged project and upload it to cloud. That makes sure the software does not process step 1 again. 

You are right about the generation of 1 block from image scale 0.5 and 2. It was my bad. This is because the key point image scales are just used to generate the automatic tie points/sparse point cloud. That shouldn’t affect the merging. 

6

Momtanu,

Thank you so much for these insights!

In the knowledgebase article, I was interpreting the first ‘Warning’ box text saying… "If the resolutions of the two cameras are significantly different: 

  • Calibration of the images in Step 1. Initial Processing will be improved if the processing options are adapted to each set of images. For this the following merging procedure is recommended: Merging projects."

… to mean that it would be possible to merge a project having RGB imagery from one camera with a project having multispectral imagery from a 2nd camera. In any case, I have managed to get 4 projects successfully processed this way! The results are beautiful: A 5-band reflectance map perfectly overlaid on the RGB orthomosaic and (critically) the DSM. It’s exactly what I hoped would be possible with your software. 

I wonder if I’m able to have some of this success because the X3 camera and the RedEdge camera don’t have pixel resolutions that are *too* different? For instance, Pix4D is reporting a GSD of ~5cm for the X3 rig and about 8cm for the RedEdge rig.

Oh, that is great to know that Pix4D Cloud can handle the merged projects. The first time I tried uploading a merged project, I did uncheck the box for Step 1, but Step 1 was still re-done in the Cloud anyway. I have a guess as to what I might have done wrong though: perhaps after I unchecked Step 1 I didn’t save the project before uploading the project files?  I will try this and see whether I can avoid having Pix4D cloud redo Step 1.

Thanks again for all of your help!

7

Hi Michael, Yes your project is amazing. It is great to see you got such good results. I have got in touch with the cloud team to see what they say about step 1 re-processing for your project. I will get back to you once I have more information.

1 Like
8

Momtanu,

Hah! Thanks. The projects that I haven’t shared yet are even better! I’m always happy to share the raw imagery and processed data if it’s helpful to you all.

I learned from Alice over on this thread (https://community.pix4d.com/t/7540/comments/360003812111) that unchecking the Step 1 box won’t be enough to keep Pix4D Cloud from restarting Step 1. I just asked over there whether it’s possible to upload the Step 1 results in order for Pix4D Cloud to make use of them during Step 2 and 3, but it doesn’t seem like this functionality exists.

Is this where the customization of Pix4D Engine would be useful?

9

Hi Michael,

Yes we got to know that step 1 is reprocessed even if it is unchecked. When you process in cloud, only the raw images and the p4d file are uploaded (and not any results/ output). Thus Pix4D re-processes step 1. Therefore, merging projects in cloud with different step 1 processing options is not possible (for example RGB and thermal, or RGB and multispectral).

1 Like
10

Okay, thanks so much for following up on this. I really appreciate it!

11

Whoops, one more follow up question:

If I process *both* Step 1 and Step 2 locally, is it possible to upload the raw images, the p4d file (with *only* the box for Step 3 checked), and upload the point cloud result generated from local Step 2 into Pix4D for processing of just Step 3?

Or will Steps 1, 2, and 3 be processed anytime I upload a project to the Cloud, even if I upload the results from local processing of Step 2? 

12

Hi Michael,

The uploading is just for sharing purposes. Thus step 2 will be processed. However, for cases when you just want to process step 1 and step 2 (and check only these steps), Pix4D will not perform step 2 at all. It is for cases where you want ortho without getting the dense point cloud (just from the automatic tie points/sparse point cloud): https://support.pix4d.com/hc/en-us/articles/202559109-Is-it-possible-to-obtain-the-DSM-and-or-Orthomosaic-without-processing-Step-2-Point-Cloud-and-Mesh-

13

Copy that. Thanks so much for your help Momtanu!