Hi Anton,
Here is how you should do this:
- horizontal CS: set the output and GCP coordinate system to EPSG 3006. The steps are described here: Select Image / GCP / Output Coordinate System.
-
vertical CS: if you need to select a different vertical datum such as the RH2000, then you should either provide the offset between this datum and the ellipsoid of the selected horizontal coordinate system or to select the option Arbitrary for the vertical coordinate system of the outputs and the GCPs and import GCPs with coordinates in the coordinate system you specified.
In the first case, you should know the shift/offset between the current vertical coordinate system and the target vertical coordinate system (RH2000). This shift can be added in the Geoid Height Above XXX Ellipsoid option. It might be difficult to know this shift. Even if you know it, it will only be an approximation as to convert from one system to the other, just one shift is not enough. Therefore, the vertical coordinates will not have everywhere the same accuracy. More information can be found here: When to use the Geoid Height Above the Ellipsoid Function?
In the second case, the software will respect the GCPs values and will try to fit the model as well as possible to the GCPs. Then, all your outputs will still be in an arbitrary reference system but this system will be based on the system defined by the GCPs. Since you have GCPs in RH2000, this option is the most suitable for you. To check the accuracy of this method and see if it is suitable for your purposes, you could possibly try to run a project setting the vertical coordinate system of both the GCPs and the outputs to Arbitrary and using checkpoints. The checkpoints assess the accuracy of the model.
You could also check these community posts, there are numerous knowledgeable users who published similar questions and could give you more hints:
Converting WGS84 images to NAVD88 output
Vertical Datum of GCP in NAV88 - Which Vertical Coordinate System should I use
Best,
Teodora