Agisoft PhotoScan GPU Acceleration

Introduction

1. Align Photos
2. Build Dense Cloud
3. Build Mesh
4. Build Texture

Of these four steps, only the "Build Dense Cloud" step is able to utilize the video card. However, that one step takes longer than all the others combined so it makes choosing the correct GPU for PhotoScan critical to ensuring that the hardware in your system is correctly optimized. In terms of the time it takes to generate a 3D model, building the dense cloud ranges from being approximately 50% of the total time if you use medium settings, 80% if using high settings, or 96% (!) if you use ultra high (or highest) settings.

In this article we are going to look in detail at a number of different models and quantities of video cards at medium, high, and ultra high. If you would rather simply view our conclusions, feel free to jump ahead to the conclusion section.

Test Setup

For our test system, we used the following hardware:

We learned in our article "Agisoft PhotoScan Multi Core Performance" that how well PhotoScan can use multiple CPU cores improves as you increase the number of GPUs. At the same time, Agisoft recommends disabling one core per GPU (which is done through Preferences -> OpenCL) for best performance. Because of these two factors we wanted to use a pair of Xeon E5 V3 ten core CPUs (for twenty physical cores in total) in order to ensure that the CPU would not bottleneck the video cards to any significant degree.

Originally, we were only planning on testing single and dual GPU configurations but as you will soon see we quickly found the need to bump our testing all the way up to four video cards. The video cards we used in our testing are the following:

You will notice that we are only testing NVIDIA GeForce video cards and not NVIDIA Quadro or AMD Radeon/FirePro cards. PhotoScan should work fine with almost any video card since it uses OpenCL (which is compatible with any modern GPU), but since it does not require dual precision performance or ECC memory the NVIDIA Quadro and AMD FirePro video cards would simply be a waste of money as you would be paying for features that PhotoScan cannot use or need. AMD Radeon cards should work great and would actually probably be a bit better performance per dollar than the NVIDIA GeForce cards, but as we've discussed in the past we feel that in most situations AMD Radeon cards are simply too unreliable historically for a professional environment. Especially if you will be running a job that takes a very long time you really want to have video card(s) that are as reliable as possible.

For our test data we used the Monument sample data that Agisoft has made available on their website. We found that this set of images was a good mix of being hard enough on the hardware to really test the GPU performance while also being relatively quick to finish (since we had to complete a large number of runs to give us the data we need). To ensure that our results will be accurate for larger data sets, we also did some spot testing using other projects including a larger data set provided to us from one of our customers. What we found is that while the resolution of the images doesn't influence the time it takes to align the photos, it does change the amount of time it takes to complete the other three steps at close to a 1:1 ratio. In addition, the number of images included in the data set also causes a linear increase in the amount of time it takes to complete all four steps (including aligning the photos). Since PhotoScan only uses the GPU for the "build dense cloud" step, this means that a data set with either twice the number of images or twice the MP count will take approximately twice the amount of time to complete for that step. Further, if you have a data set with twice the number of images and twice the MP count it will have a roughly 4x longer build time.

What that means is that we have found the performance of PhotoScan to be roughly linear so the CPU and GPU that is best for a large number of high res images will also be best for a smaller number of lower res images. The amount of actual build time will of course be different - and you will need more RAM for larger data sets - but a configuration that gives a 50% performance increase for a small data set should give a roughly 50% performance increase for a larger data set.

Medium Quality Setting

Starting with the quality setting for "Build Dense Cloud" on medium we already see some very interesting results. With a single video card, there is a nice decrease in the build time of about 10% each time the GPU was upgraded to a faster model. For the dual, triple, and quad GPU configurations we saw an even better improvement (about 15-20%) between the GTX 960 and the GTX 970, but almost nothing when upgrading to either a GTX 980 or GTX Titan X.

The really useful information here is the amount of improvement we saw each time we increased the number of video cards. Depending on the model of video card, going from one GPU to two decreased the build time between 25-35%. Going from two to three was an additional 20-25% decrease in build time and going from three to four was another 10-15% decrease in build time on top of that. 

What this means is that (assuming you are not CPU limited) it is much better to have more GTX 960 or GTX 970 video cards than it is to have fewer, high-end video cards if you will be using the medium quality setting. However, there are some additional factors that come into play that we will discuss in the conclusion section that depending on your budget may prevent you from using three or four mid-range video cards instead of one or two high-end video cards.

High Quality Setting

Upping the quality setting to high gives us fairly similar results to the medium settings except that we see a benefit of using faster GPUs even at the higher video card counts.

On average, upgrading to a GTX 970 from a GXT 960 gives a great 20-25% decrease in build times while upgrading to a GTX 980 or GTX Titan X results in a further 5-10% decrease in the time it takes to build the dense cloud. Again, however, this is somewhat overshadowed by how much of a performance benefit there is to having multiple video cards.

Going from one GPU to two results in about a 35-40% decrease in build time depending on the GPU which is 5-10% better than what we saw on the medium settings. Going form two to three video cards results in another 20-30% decrease in build time while going from three to four video cards results in a further 10-20% decrease in build time.

Ultra High Quality Setting

While using the ultra high quality setting significantly increases the time to takes to build the dense cloud, the relative performance difference between the model and number of video cards is actually very similar to what we saw with the high quality setting.

The only oddity is that with only one or two video cards the GTX Titan X isn't as much of a performance improvement over the GTX 980 as it was on either the medium or high quality setting. It is certainly still faster than the GTX 980, but only by about 3-4%. Once you get up to triple or quad GPU configurations, however, it is around 7-13% faster than the GTX 980.

Overall, we saw about a 20-25% decrease in build time going from the GTX 960 to the GTX 970 depending on the number of video cards and anywhere from a 5-10% further decrease in build time upgrading to the GTX 980 or GTX Titan X.

Upgrading the number of video cards from one to two results in a 35-40% decrease in built time, while going from two to three is about a 20-25% further decrease in build time. Increasing the GPU count to four is a bit less effective, but still results in a 15-20% decrease in build time compared to the triple GPU configurations.

Conclusion

At first glance, it looks for choosing the right video card for PhotoScan follows the classic "more is better" approach. If you can buy four GTX 960 video cards for the same price as a GTX Titan X but the four GTX 960's result in a build time that is twice as fast (or even more at high and ultra high settings) why wouldn't you do that? Yes, the power draw of four GTX 960s is twice that of a single GTX Titan X, but if they can cut the build time in half or more that actually makes the four GTX 960's both cheaper and more power efficient than a single GTX Titan X.

The problem really comes about when you factor in the rest of the system - most notably the CPU. If you read our Agisoft PhotoScan Multi Core Performance article, there are a couple of very interesting CPU considerations when it comes to using multiple video cards in PhotoScan:

If you are configuring a system for PhotoScan, we recommend also reading our other articles regarding the hardware requirements for PhotoScan:

• Agisoft PhotoScan Multi Core Performance
• Recommended Hardware for Agisoft PhotoScan

If you don't want to wade through all those different results, we also came up with three different recommended systems based on whether you have the budget for a dual, triple, or quad GPU system. These systems have a few CPU and video card options but they were designed so that you can't possibly make a bad decision. If you choose a more expensive CPU or GPU option, you will see a decrease in the time it takes to build a 3D model in PhotoScan: