Introduction
The following report describes the digitalisation process of the fountain located in Rua da Junqueira in Lisbon, Portugal. The purpose of this exercise was my academic introduction to the photogrammetry workflow, which is linear, and can be approached with different software at different stages but still following the same sequential order. There were four different software involved in developing the final results; both were taught during the course of D3DRVA 2020-2021 of the faculty of Architecture, Urbanism and Design of the University of Lisbon and self-learned during the process.
Workflow
The photogrammetry process of this exercise was the following:
- Image capturing.
- Image correspondence and point clouds recreation.
- Mesh generation and textures.
- Orthophoto generation.
The equipment used for the image capturing was a mobile phone with the following characteristics.
Brand: Xiaomi
Model: Mi A3
Camera resolution: 48 megapixels
Operation system: Android version 11
Lens: f/1.79 26mm
The image capturing was carried out on 3 different days. The first day, February 27, not all of the images obtained during this day were appropriate because of the height of the selected object.
On the third day, March 03, was acquired in advance a 5-meter painting roller extension pole to reach the highest points of the fountain. To accomplish the task of taking pictures at this height, the open-source application Open Camera, developed by Mark Harman (Harman, 20131), was previously installed in the smartphone. Third-party software was used instead of the default camera application because it allowed the setting of a loop auto-timer shooter that could work without using any extra device such as a Bluetooth remote control shooter. Furthermore, this application allowed disabling the auto-focus mode, which was necessary to keep turned off during this process.
Lastly, on March 15, another path of image capturing of the object was added to get more details of certain areas that were left behind in the last session. It is important to notice that to avoid shadow casting pattern problems in the process of image matching by the software, even though the pictures were captured on different days, they were obtained at the same scheduled time,1 pm, when the object received the most direct sunlight projection.
The attachment system of the smartphone to the extension pole was a masking tape, strapped in a manner that allowed to have control of the screen and freed the lens area.
The size of the pictures obtained was 4000×300, in JPG format, and the total size of pictures collected was 696; but not all of them were processed.
The reference measures were taken of the most straight and longest part of the fountain, as shown in figure 5, of 4.22 meters in the horizontal direction and 4.70 meters in the vertical direction. These dimensions would be used in further stages of the post-processing.
2. Image correspondence and point clouds recreation
The first compute carried out by photogrammetry software is the detection of similar features between images and the setting of the relative locations of the cameras (pictures) to the object to later recreate them in a 3D space.
The second step consists of the reconstruction of the model with points in the coordinate system; this a two-stage process: the sparse cloud point cloud, with an automatic representation of the tie points found on the cameras, and lastly, the dense point cloud, with more details, which is later used in the reconstruction of the triangular mesh.
The image correspondence and point clouds recreation processes were carried out with three different software due to the constraints faced during the different phases of reconstruction and post-processing before reaching the orthophoto generation.
The time reported in the tests below corresponds to the following hardware specifications:
Windows 10
32 GB RAM
NVIDIA GeForce GTX 1050 ti
i7 2.2 GHz
Visual SMF
The first test of image processing was made in the open-source software Visual SMF, developed by Changchang Wu during his PhD research, which was part of the workflow taught by the professors of the course (Mateus, 20212).
Number of cameras: 696
Processing time: 0.35 hours
Sparse cloud points:
Problems in the pictures drove to the algorithms of the software to produce 13 different non-related models of the sparse point cloud reconstruction. Despite the usage of the ‘Merge Sparse Models’ option, the reconstruction of the model was still unsuccessful (Wu, 20113).
This erratic reconstruction was caused due to several moving objects and reflecting surfaces surround the object of study that were causing conflict and then hindering proper image matching.
This problem would be later solved with masking methods in another software since this one does not have an in-built option for masking the images, neither read the masks created from third party software. Consequently, the dense cloud reconstruction, which was the following part of the process, was not achieved with Visual SMF.
Meshroom
The second software tested was Meshroom, another open-source software developed by the AliceVision association.
Number of cameras: 328
Processing time: 5.55 hours
Dense cloud points: 1.607.461
Number of triangles in mesh: 3.202.605
It is important to notice that this software does not allow the use of masks for the images; nevertheless, it computed both the sparse and dense point cloud properly. However, there were two main problems with the workflow preventing following through the other parts of the process: the inability to set ground control points and clean the point cloud of extra information.
Ground control points (GCP) are reference points that must be settled in at least three different points that appear in different pictures to scale and orient the model according to real-world measures and the XYZ coordinate system. This feature for Meshroom is still under development and had not been incorporated into the last version used during the post-processing of the images (GitHub / AliceVision / Meshroom, 20204). Therefore, to scale the object properly, the final data produced in Meshroom should be post-processed in a second software: Meshlab; this would be, however, an inaccurate scaling method compared to the GCP method where the information is taken directly from the pictures.
The pattern of the cameras shown above in Figure 10 demonstrates the different angles that were taken to optimise the shape and textures of the study object.
In Figure 11 is shown how the software displays the pipeline of each step of the process in a visual script interface, which can be altered in accordance with the user needs.
The other important step to move forward in the post-processing of the point cloud is deleting extra points that are not part of the study object, leading to extra and non-necessary computing time. Meshroom is not prepared for this post-processing job of point cleaning. Hence, it was necessary to add extra nodes to the script to export a PLY format file to clean the point cloud in Meshlab (Meshroom Manual, 20215); as ABC, the default point cloud format file generated by Meshroom, is not supported by Meshlab.
Nevertheless, there were two subsequently problems arising from the previous methodology applied in this exercise, both related to the textures of the object.
One of them surges in the dense point cloud Meshlab cleaning approach; as there were no images associated with the points, once cleaned, the only way of associating images to the mesh generated in Meshlab from these points; would have been through the importation of the position of the cameras, and the individual alignment of the images used in the reconstruction.
The other texture-related issue in the mesh hand-scaled mesh approach was that once a selection of the mesh was deleted for cleaning, it also lost the texture in the whole mesh. Again, it was not associated in any other way besides the single JPG created automatically by Meshroom.
3DF Zephyr
Lastly, t 3DF Zephyr, a proprietary software, resulted in the best outcomes. In order to use the functionality of adding more than 50 images in the image matching process, was acquired a student license of 30 days, which was later extended for 60 days more.
Number of cameras: 686
Processing time: 4.41 hours
Sparse cloud points: 93.367
Dense cloud points: 1.320.895
Pre-set: Default / Urban
The first advantage in using 3DF Zephyr was the employment of the Masquerade module to mask the conflicting objects of the surroundings of the study object, such as a convex traffic mirror that belongs to the building next to the fountain; tram, buses and cars and their shadows passing by the Junqueira street. After applying the masks to each picture where these objects appeared, 127 pictures in total, the processing time, and the point clouds reconstruction quality improved drastically.
As it can be seen in figures 17 and 18, a reconstruction carried out without masking the conflicting objects led to distortions and misalignments of the model, particularly in the corners of the study object, where were the most problematic areas of the pictures facing the street with reflecting and moving objects.
After the successful reconstruction of the dense point cloud, four ground control points were set in accordance with the measures gathered during the image capture to scale the model and align it with the coordinate system (3D Flow, 20216).
Furthermore, before moving forward with the mesh generation, the dense point cloud was cleaned with two different methods. The first method was the delimitation of a bounding box to remove from the computing of the mesh the farthest and non-related points to the study area. The second method was the application of a colour filter to erase as many points as possible from the ornamental crown on the top of the fountain; in particular, the blue points that were part of the sky and could lead to distortions in the meshing process.
3. Mesh generation and textures
The last part of the photogrammetry workflow is creating a 3D textured mesh that can be later used for different purposes.
Once filtered and cleaned the dense point cloud in 3DF Zephyr, the mesh is generated within the software; it was made in 24 minutes and counted with 2.241.875 triangles.
Furthermore, it was carried out another contextual cleaning of the mesh that was not executed during the point cloud cleaning, since it could have affected areas where edges were blurred with important parts of the study object and were difficult to distinguish in the point cloud cleaning.
Before the generation of the textured mesh, was applied a mesh filter to correct bulges of areas and edges that are detailed, as shown in figures 26 and 27. The filter used was ‘Laplacian smoothing’, with 5 iterations. The final mesh after the cleaning counted with 1.733.825 triangles, and it was computed in 17 minutes.
Lastly, creating the mesh with textures was also made within the 3DF Zephyr environment and exported into an OBJ file with a single texture image file to ease the sharing capabilities.
The settings used in the generation o the mesh were the default pre-sets, Category: General and Default Single texture.
In addition to the final textured mesh, was also created and exported a low poly mesh to be shared on the Sketchfab platform through this link: https://skfb.ly/onZQP
4. Orthophoto generation
One of the architectural uses of photogrammetry is the creation of orthographic projections of the digitalised object. The 3DF Zephyr workflow allows the creation of these images within its environment; however, the method of the orthophoto generation taught in the course was also carried out, since the workflow started with Meshroom was unsuccessful.
There was created a front elevation and a floor plan elevation in JPG files. The processing time of each orthophoto generated in 3DF Zephyr was of 5 minutes; along with the JPG file, was created a PDF file already scaled and with scale references that was later used in AutoCAD.
The OBJ file obtained from 3DF Zephyr was imported into Cloud Compare for the second workflow approach, as this textured mesh was the one that was successfully scaled and clean. It is observed in figure 28 and 29 that the scale factor of both floor plan and elevation projection is 2 cm per pixel, zoom factor 3, and the final output resolution was 4788×2304 pixels.
It can be noticed in figures 30 and 31 that the quality of the orthophotos delivered by 3DF Zephyr are smoother than the ones exported in Cloud Compare, where shadows are more evident in the imperfections of the model, which are the white areas, with a lower detail frequency. To improve the orthophoto detail generated with Cloud Compare, was toggled on the custom light option, while 3DF Zephyr might be using shaders to enhance the picture.
Finally, the orthophotos were imported into AutoCAD to produce a scaled PDF with both projections in a single sheet.
Other experiences
There were three different experiences of scale approach to photogrammetry, all of them urban objects. The first one was a fire hydrant; the second one was a single storey house; and lastly, the fountain, which is part of the public assets of the city council, and was broadly described throughout this report. These two other exercises were also developed in different software to train the image capturing and software usage skills.
Conclusions
As demonstrated throughout the process, the 3DF Zephyr software covers the full photogrammetry workflow, from the image and feature matching to the orthophoto images creation. 3DF Zephyr is a sound software, but it is constrained to the dependency of proprietary software, which is not aimed in the discipline, and for that reason, the workflow is mainly taught with open-source software.
For that reason, it is important to highlight that not having been able to carry out the exercise from the beginning with the VSMF workflow does not mean that it is not a feasible method, simply the environmental conditions of the chosen case study did not provide the ideal static setting, which as it could not be overcome with masks, the software was not able to read.
Furthermore, the open-source software workflow relies on several projects that are under developments, therefore, are constantly changing, and most of them have not been optimised. In this workflow, are several input and output formats to be considered to avoid further problems in the following stages.
After the different tests made, the workflows used in this exercise were summarised in the following diagram:
References
- Harman, M. (2013). Open Camera. Retrieved from Open Camera: https://opencamera.org.uk/ ↩︎
- Mateus, L. (2021, February 25). Workflow fotogramétrico (VisualSFM, CMVS/PMVS, MeshLab, CloudCompare). Retrieved from Youtube: https://www.youtube.com/watch?v=inIphc0iaYI ↩︎
- Wu, C. (2011). VisualSFM : A Visual Structure from Motion System. Retrieved from Documentation: http://ccwu.me/vsfm/doc.html#multiple_models ↩︎
- GitHub / AliceVision / Meshroom. (2020, March 26). Retrieved from CCTAG scaling: https://github.com/alicevision/meshroom/wiki/CCTAG-scaling ↩︎
- Meshroom Manual. (2021). Retrieved from Tutorial: Meshroom for Beginners: https://meshroom-manual.readthedocs.io/en/latest/tutorials/sketchfab/sketchfab.html ↩︎
- 3D Flow. (2021). Obtido de 3DF Zephyr Tutorials: http://www.3dflow.net/it/technology/documents/3df-zephyr-tutorials/ ↩︎
To download the PDF version click here
