Agisoft PhotoScan GPU AccelerationWritten on September 11, 2015 by Matt Bach
When designing a computer there are literally thousands of different hardware components to choose from and each one will have an impact on the overall performance of your system in some shape or form. Depending on the software you will be using, however, some components will simply be more important than others. In the case of Agisoft PhotoScan, the two components that most directly affect performance are the CPU and the video card. In this article we want to look at how the performance and number of video cards in a system affects the time it takes to generate a 3D model in PhotoScan. If you are interested in how the CPU affects PhotoScan performance, we recommend reading our related article Agisoft PhotoScan Multi Core Performance.
While PhotoScan can do a number of different tasks, for this article we are going to focus on one of the four basic steps PhotoScan goes through to convert a series of photographs into a 3D model:
- Align Photos
- Build Dense Cloud
- Build Mesh
- Build Texture
Of these four steps, only the "Build Dense Cloud" step is able to utilize the video card. However, that one step takes longer than all the others combined so it makes choosing the correct GPU for PhotoScan critical to ensuring that the hardware in your system is correctly optimized. In terms of the time it takes to generate a 3D model, building the dense cloud ranges from being approximately 50% of the total time if you use medium settings, 80% if using high settings, or 96% (!) if you use ultra high (or highest) settings.
In this article we are going to look in detail at a number of different models and quantities of video cards at medium, high, and ultra high. If you would rather simply view our conclusions, feel free to jump ahead to the conclusion section.
For our test system, we used the following hardware:
|Motherboard:||Asus Z10PE-D8 WS|
|CPU:||2x Intel Xeon E5-2687W V3 3.1GHz Ten Core|
|RAM:||8x Kingston DDR4-2133 8GB ECC Reg.|
|Hard Drive:||Samsung 850 Pro 512GB SATA 6Gb/s SSD|
|OS:||Windows 8.1 Pro 64-bit|
|PSU:||EVGA SuperNOVA 1600W P2 Power Supply|
|Software:||Agisoft PhotoScan 1.1.6 build 2038 (64-bit)|
We learned in our article "Agisoft PhotoScan Multi Core Performance" that how well PhotoScan can use multiple CPU cores improves as you increase the number of GPUs. At the same time, Agisoft recommends disabling one core per GPU (which is done through Preferences -> OpenCL) for best performance. Because of these two factors we wanted to use a pair of Xeon E5 V3 ten core CPUs (for twenty physical cores in total) in order to ensure that the CPU would not bottleneck the video cards to any significant degree.
Originally, we were only planning on testing single and dual GPU configurations but as you will soon see we quickly found the need to bump our testing all the way up to four video cards. The video cards we used in our testing are the following:
|Test Video Cards|
|1-4x NVIDIA GeForce GTX 960 2GB|
|1-4x NVIDIA GeForce GTX 970 4GB|
|1-4x NVIDIA GeForce GTX 980 4GB|
|1-4x NVIDIA GeForce GTX Titan X 12GB|
You will notice that we are only testing NVIDIA GeForce video cards and not NVIDIA Quadro or AMD Radeon/FirePro cards. PhotoScan should work fine with almost any video card since it uses OpenCL (which is compatible with any modern GPU), but since it does not require dual precision performance or ECC memory the NVIDIA Quadro and AMD FirePro video cards would simply be a waste of money as you would be paying for features that PhotoScan cannot use or need. AMD Radeon cards should work great and would actually probably be a bit better performance per dollar than the NVIDIA GeForce cards, but as we've discussed in the past we feel that in most situations AMD Radeon cards are simply too unreliable historically for a professional environment. Especially if you will be running a job that takes a very long time you really want to have video card(s) that are as reliable as possible.
For our test data we used the Monument sample data that Agisoft has made available on their website. We found that this set of images was a good mix of being hard enough on the hardware to really test the GPU performance while also being relatively quick to finish (since we had to complete a large number of runs to give us the data we need). To ensure that our results will be accurate for larger data sets, we also did some spot testing using other projects including a larger data set provided to us from one of our customers. What we found is that while the resolution of the images doesn't influence the time it takes to align the photos, it does change the amount of time it takes to complete the other three steps at close to a 1:1 ratio. In addition, the number of images included in the data set also causes a linear increase in the amount of time it takes to complete all four steps (including aligning the photos). Since PhotoScan only uses the GPU for the "build dense cloud" step, this means that a data set with either twice the number of images or twice the MP count will take approximately twice the amount of time to complete for that step. Further, if you have a data set with twice the number of images and twice the MP count it will have a roughly 4x longer build time.
What that means is that we have found the performance of PhotoScan to be roughly linear so the CPU and GPU that is best for a large number of high res images will also be best for a smaller number of lower res images. The amount of actual build time will of course be different - and you will need more RAM for larger data sets - but a configuration that gives a 50% performance increase for a small data set should give a roughly 50% performance increase for a larger data set.
Medium Quality Setting
Starting with the quality setting for "Build Dense Cloud" on medium we already see some very interesting results. With a single video card, there is a nice decrease in the build time of about 10% each time the GPU was upgraded to a faster model. For the dual, triple, and quad GPU configurations we saw an even better improvement (about 15-20%) between the GTX 960 and the GTX 970, but almost nothing when upgrading to either a GTX 980 or GTX Titan X.
The really useful information here is the amount of improvement we saw each time we increased the number of video cards. Depending on the model of video card, going from one GPU to two decreased the build time between 25-35%. Going from two to three was an additional 20-25% decrease in build time and going from three to four was another 10-15% decrease in build time on top of that.
What this means is that (assuming you are not CPU limited) it is much better to have more GTX 960 or GTX 970 video cards than it is to have fewer, high-end video cards if you will be using the medium quality setting. However, there are some additional factors that come into play that we will discuss in the conclusion section that depending on your budget may prevent you from using three or four mid-range video cards instead of one or two high-end video cards.
High Quality Setting
Upping the quality setting to high gives us fairly similar results to the medium settings except that we see a benefit of using faster GPUs even at the higher video card counts.
On average, upgrading to a GTX 970 from a GXT 960 gives a great 20-25% decrease in build times while upgrading to a GTX 980 or GTX Titan X results in a further 5-10% decrease in the time it takes to build the dense cloud. Again, however, this is somewhat overshadowed by how much of a performance benefit there is to having multiple video cards.
Going from one GPU to two results in about a 35-40% decrease in build time depending on the GPU which is 5-10% better than what we saw on the medium settings. Going form two to three video cards results in another 20-30% decrease in build time while going from three to four video cards results in a further 10-20% decrease in build time.
Ultra High Quality Setting
While using the ultra high quality setting significantly increases the time to takes to build the dense cloud, the relative performance difference between the model and number of video cards is actually very similar to what we saw with the high quality setting.
The only oddity is that with only one or two video cards the GTX Titan X isn't as much of a performance improvement over the GTX 980 as it was on either the medium or high quality setting. It is certainly still faster than the GTX 980, but only by about 3-4%. Once you get up to triple or quad GPU configurations, however, it is around 7-13% faster than the GTX 980.
Overall, we saw about a 20-25% decrease in build time going from the GTX 960 to the GTX 970 depending on the number of video cards and anywhere from a 5-10% further decrease in build time upgrading to the GTX 980 or GTX Titan X.
Upgrading the number of video cards from one to two results in a 35-40% decrease in built time, while going from two to three is about a 20-25% further decrease in build time. Increasing the GPU count to four is a bit less effective, but still results in a 15-20% decrease in build time compared to the triple GPU configurations.
At first glance, it looks for choosing the right video card for PhotoScan follows the classic "more is better" approach. If you can buy four GTX 960 video cards for the same price as a GTX Titan X but the four GTX 960's result in a build time that is twice as fast (or even more at high and ultra high settings) why wouldn't you do that? Yes, the power draw of four GTX 960s is twice that of a single GTX Titan X, but if they can cut the build time in half or more that actually makes the four GTX 960's both cheaper and more power efficient than a single GTX Titan X.
The problem really comes about when you factor in the rest of the system - most notably the CPU. If you read our Agisoft PhotoScan Multi Core Performance article, there are a couple of very interesting CPU considerations when it comes to using multiple video cards in PhotoScan:
- As you increase the number of GPUs, the multi core efficiency of the "Build Dense Cloud" step increases by about 5% per GPU
- For every physical GPU in the system, Agisoft recommends you disable one CPU core (done through Preferences -> OpenCL)
Since you need one core reserved per GPU and the multi core efficiency increases as you add more GPUs to the system this means that as you add video cards you also have to have a CPU with a higher core count as well. Since CPUs with higher core counts are more expensive, this means you have to balance purchasing multiple cheaper video cards with a more expensive high core count CPU to ensure that you properly allocating your budget. Unfortunately, this makes it very difficult to recommend one GPU configuration over another.
We came up with two ways to help you make sure that you have the correct combination of CPU and GPU to give you the absolute best performance in PhotoScan for your budget. The first is a Google doc where we used both the results from this testing and our CPU article to come up with an estimate of approximately how long it should take different combinations of CPU and video cards to build a theoretical 3D model. If you want to play around with that, it is available at:
You will need to make a copy of the sheet (through File -> Make a Copy) before you can edit anything, but this gives you the freedom to input different CPU options (although the ones we have in there should be among the best choices) as well as what your budget is. Based on that, it will highlight the CPU/GPU combinations that fall within your budget so that you can look through them to find the one with the lowest build time.
If you are configuring a system for PhotoScan, we recommend also reading our other articles regarding the hardware requirements for PhotoScan:
If you don't want to wade through all those different results, we also came up with three different recommended systems based on whether you have the budget for a dual, triple, or quad GPU system. These systems have a few CPU and video card options but they were designed so that you can't possibly make a bad decision. If you choose a more expensive CPU or GPU option, you will see a decrease in the time it takes to build a 3D model in PhotoScan:
Recommended Systems for Agisoft PhotoScan