Article Thumbnail

Agisoft PhotoScan GPU Acceleration

Written on September 11, 2015 by Matt Bach
Share:
Table of Contents:
  1. Introduction
  2. Test Setup
  3. Medium Quality Setting
  4. High Quality Setting
  5. Ultra High Quality Setting
  6. Conclusion
  7. Recommended Systems for Agisoft PhotoScan

Introduction

When designing a computer there are literally thousands of different hardware components to choose from and each one will have an impact on the overall performance of your system in some shape or form. Depending on the software you will be using, however, some components will simply be more important than others. In the case of Agisoft PhotoScan, the two components that most directly affect performance are the CPU and the video card. In this article we want to look at how the performance and number of video cards in a system affects the time it takes to generate a 3D model in PhotoScan. If you are interested in how the CPU affects PhotoScan performance, we recommend reading our related article Agisoft PhotoScan Multi Core Performance.

While PhotoScan can do a number of different tasks, for this article we are going to focus on one of the four basic steps PhotoScan goes through to convert a series of photographs into a 3D model:

  1. Align Photos
  2. Build Dense Cloud
  3. Build Mesh
  4. Build Texture

Of these four steps, only the "Build Dense Cloud" step is able to utilize the video card. However, that one step takes longer than all the others combined so it makes choosing the correct GPU for PhotoScan critical to ensuring that the hardware in your system is correctly optimized. In terms of the time it takes to generate a 3D model, building the dense cloud ranges from being approximately 50% of the total time if you use medium settings, 80% if using high settings, or 96% (!) if you use ultra high (or highest) settings.

In this article we are going to look in detail at a number of different models and quantities of video cards at medium, high, and ultra high. If you would rather simply view our conclusions, feel free to jump ahead to the conclusion section.

Test Setup

For our test system, we used the following hardware:

Testing Hardware
Motherboard: Asus Z10PE-D8 WS
CPU: 2x Intel Xeon E5-2687W V3 3.1GHz Ten Core
RAM: 8x Kingston DDR4-2133 8GB ECC Reg.
Hard Drive: Samsung 850 Pro 512GB SATA 6Gb/s SSD
OS: Windows 8.1 Pro 64-bit
PSU: EVGA SuperNOVA 1600W P2 Power Supply
Software: Agisoft PhotoScan 1.1.6 build 2038 (64-bit)

We learned in our article "Agisoft PhotoScan Multi Core Performance" that how well PhotoScan can use multiple CPU cores improves as you increase the number of GPUs. At the same time, Agisoft recommends disabling one core per GPU (which is done through Preferences -> OpenCL) for best performance. Because of these two factors we wanted to use a pair of Xeon E5 V3 ten core CPUs (for twenty physical cores in total) in order to ensure that the CPU would not bottleneck the video cards to any significant degree.

Originally, we were only planning on testing single and dual GPU configurations but as you will soon see we quickly found the need to bump our testing all the way up to four video cards. The video cards we used in our testing are the following:

You will notice that we are only testing NVIDIA GeForce video cards and not NVIDIA Quadro or AMD Radeon/FirePro cards. PhotoScan should work fine with almost any video card since it uses OpenCL (which is compatible with any modern GPU), but since it does not require dual precision performance or ECC memory the NVIDIA Quadro and AMD FirePro video cards would simply be a waste of money as you would be paying for features that PhotoScan cannot use or need. AMD Radeon cards should work great and would actually probably be a bit better performance per dollar than the NVIDIA GeForce cards, but as we've discussed in the past we feel that in most situations AMD Radeon cards are simply too unreliable historically for a professional environment. Especially if you will be running a job that takes a very long time you really want to have video card(s) that are as reliable as possible.

For our test data we used the Monument sample data that Agisoft has made available on their website. We found that this set of images was a good mix of being hard enough on the hardware to really test the GPU performance while also being relatively quick to finish (since we had to complete a large number of runs to give us the data we need). To ensure that our results will be accurate for larger data sets, we also did some spot testing using other projects including a larger data set provided to us from one of our customers. What we found is that while the resolution of the images doesn't influence the time it takes to align the photos, it does change the amount of time it takes to complete the other three steps at close to a 1:1 ratio. In addition, the number of images included in the data set also causes a linear increase in the amount of time it takes to complete all four steps (including aligning the photos). Since PhotoScan only uses the GPU for the "build dense cloud" step, this means that a data set with either twice the number of images or twice the MP count will take approximately twice the amount of time to complete for that step. Further, if you have a data set with twice the number of images and twice the MP count it will have a roughly 4x longer build time.

What that means is that we have found the performance of PhotoScan to be roughly linear so the CPU and GPU that is best for a large number of high res images will also be best for a smaller number of lower res images. The amount of actual build time will of course be different - and you will need more RAM for larger data sets - but a configuration that gives a 50% performance increase for a small data set should give a roughly 50% performance increase for a larger data set.

Medium Quality Setting

Starting with the quality setting for "Build Dense Cloud" on medium we already see some very interesting results. With a single video card, there is a nice decrease in the build time of about 10% each time the GPU was upgraded to a faster model. For the dual, triple, and quad GPU configurations we saw an even better improvement (about 15-20%) between the GTX 960 and the GTX 970, but almost nothing when upgrading to either a GTX 980 or GTX Titan X.

The really useful information here is the amount of improvement we saw each time we increased the number of video cards. Depending on the model of video card, going from one GPU to two decreased the build time between 25-35%. Going from two to three was an additional 20-25% decrease in build time and going from three to four was another 10-15% decrease in build time on top of that. 

What this means is that (assuming you are not CPU limited) it is much better to have more GTX 960 or GTX 970 video cards than it is to have fewer, high-end video cards if you will be using the medium quality setting. However, there are some additional factors that come into play that we will discuss in the conclusion section that depending on your budget may prevent you from using three or four mid-range video cards instead of one or two high-end video cards.

High Quality Setting

Upping the quality setting to high gives us fairly similar results to the medium settings except that we see a benefit of using faster GPUs even at the higher video card counts.

On average, upgrading to a GTX 970 from a GXT 960 gives a great 20-25% decrease in build times while upgrading to a GTX 980 or GTX Titan X results in a further 5-10% decrease in the time it takes to build the dense cloud. Again, however, this is somewhat overshadowed by how much of a performance benefit there is to having multiple video cards.

Going from one GPU to two results in about a 35-40% decrease in build time depending on the GPU which is 5-10% better than what we saw on the medium settings. Going form two to three video cards results in another 20-30% decrease in build time while going from three to four video cards results in a further 10-20% decrease in build time.

Ultra High Quality Setting

While using the ultra high quality setting significantly increases the time to takes to build the dense cloud, the relative performance difference between the model and number of video cards is actually very similar to what we saw with the high quality setting.

The only oddity is that with only one or two video cards the GTX Titan X isn't as much of a performance improvement over the GTX 980 as it was on either the medium or high quality setting. It is certainly still faster than the GTX 980, but only by about 3-4%. Once you get up to triple or quad GPU configurations, however, it is around 7-13% faster than the GTX 980.

Overall, we saw about a 20-25% decrease in build time going from the GTX 960 to the GTX 970 depending on the number of video cards and anywhere from a 5-10% further decrease in build time upgrading to the GTX 980 or GTX Titan X.

Upgrading the number of video cards from one to two results in a 35-40% decrease in built time, while going from two to three is about a 20-25% further decrease in build time. Increasing the GPU count to four is a bit less effective, but still results in a 15-20% decrease in build time compared to the triple GPU configurations.

Conclusion

At first glance, it looks for choosing the right video card for PhotoScan follows the classic "more is better" approach. If you can buy four GTX 960 video cards for the same price as a GTX Titan X but the four GTX 960's result in a build time that is twice as fast (or even more at high and ultra high settings) why wouldn't you do that? Yes, the power draw of four GTX 960s is twice that of a single GTX Titan X, but if they can cut the build time in half or more that actually makes the four GTX 960's both cheaper and more power efficient than a single GTX Titan X.

The problem really comes about when you factor in the rest of the system - most notably the CPU. If you read our Agisoft PhotoScan Multi Core Performance article, there are a couple of very interesting CPU considerations when it comes to using multiple video cards in PhotoScan:

  1. As you increase the number of GPUs, the multi core efficiency of the "Build Dense Cloud" step increases by about 5% per GPU
  2. For every physical GPU in the system, Agisoft recommends you disable one CPU core (done through Preferences -> OpenCL)

Since you need one core reserved per GPU and the multi core efficiency increases as you add more GPUs to the system this means that as you add video cards you also have to have a CPU with a higher core count as well. Since CPUs with higher core counts are more expensive, this means you have to balance purchasing multiple cheaper video cards with a more expensive high core count CPU to ensure that you properly allocating your budget. Unfortunately, this makes it very difficult to recommend one GPU configuration over another.

We came up with two ways to help you make sure that you have the correct combination of CPU and GPU to give you the absolute best performance in PhotoScan for your budget. The first is a Google doc where we used both the results from this testing and our CPU article to come up with an estimate of approximately how long it should take different combinations of CPU and video cards to build a theoretical 3D model. If you want to play around with that, it is available at:

You will need to make a copy of the sheet (through File -> Make a Copy) before you can edit anything, but this gives you the freedom to input different CPU options (although the ones we have in there should be among the best choices) as well as what your budget is. Based on that, it will highlight the CPU/GPU combinations that fall within your budget so that you can look through them to find the one with the lowest build time.

If you are configuring a system for PhotoScan, we recommend also reading our other articles regarding the hardware requirements for PhotoScan:

If you don't want to wade through all those different results, we also came up with three different recommended systems based on whether you have the budget for a dual, triple, or quad GPU system. These systems have a few CPU and video card options but they were designed so that you can't possibly make a bad decision. If you choose a more expensive CPU or GPU option, you will see a decrease in the time it takes to build a 3D model in PhotoScan:

Recommended Systems for Agisoft PhotoScan

Compact Dual GPU

Purchase

Ideal for budgets of $3,500+ 

  • Intel Core i7 6850K or 6900K
  • Dual GTX 1070 or 1080
  • Up to 128GB of RAM

If you are looking for an affordable entry point into using PhotoScan, or want the best performance per dollar, dual GPUs is a great way to go. The options allow for up to 20% performance boost over the base CPU / GPU combination, and a wide range of drive choices.

Mid Tower Triple GPU

Purchase

Increased performance for $6,500+

  • Intel Core i7 or Xeon E5
  • Triple GTX 1070 or GTX 1080
  • Up to 512GB of RAM

Moving up to triple GPUs and a higher core count processor can increase PhotoScan performance 30-70% over the base dual GPU configuration. This design is also in a larger tower with room for more drives and support for twice the amount of RAM.

Full Tower Quad GPU

Purchase

Maximum performance for $10,000+

  • Dual Intel Xeon E5 CPUs
  • Quad GTX 1080
  • Up to 512GB of RAM

Going to a quad GPU setup provides the best possible performance in PhotoScan, with 130-150% faster results than the base dual GPU design. Because adding more GPUs requires more CPU cores to be effective, this setup uses a pair of Xeon processors.

Tags: Agisoft, PhotoScan, GPU, Video Card

I wanted to add a quick note for folks reading this, who might wonder why the top-end GPU we tested with was the GeForce GTX Titan X but then we use the GTX 980 Ti in our recommended systems. That is because the 980 Ti is almost as fast as the Titan X, but about $400 less expensive. The biggest difference is that the Titan X has 12GB of memory vs 6GB on the 980 Ti, but in our testing the video card memory had no impact (only a small amount of it was being used). As such, the 980 Ti gives the best performance / value combination on the high end, and the 970 is a solid cost savings option for lower priced systems.

Posted on 2015-09-11 21:52:44
stephen long

Hi William im looking at doing large environment landscapes using 3000 ish pictures according to the documents agisoft recommends 320gb of memory im wondering is that memory helped out with video card memory or does video card memory have little overall effect on processing ? so 2x Intel Xeon E5-2637 V3 chips , 4x 980ti's and 320gb (10xSamsung 32GB (1 x 32GB) Registered DDR4-2133 Memoy)

resting in a

Asus Z10PE-D16 WS
board overkill ? or just right for the job ?

Posted on 2015-10-15 01:47:43
Eyal Saiet

This is a great article. Something I think worth adding, Agisoft files (after done processing) tend to be several GB to tens of GB. Software such as Quick Terrain Modeller, depend on huge memory in the GPU to work with these massive files. Therefore I think there is significant value that one of the four graphic cards has the largest memory you can afford.

Posted on 2015-10-07 00:18:05

The two cards we are currently listing on our recommended systems are the GTX 970 4GB and the GTX 980 Ti 6GB. Is 6GB of VAM enough or do you think we should add the GTX Titan X 12GB to that lineup? It doesn't give much performance over the GTX 980 Ti, but we could add it with a note about the additional VRAM being needed for certain post-processing software.

Posted on 2015-10-07 00:55:02
voojoo

How about NVIDIA Tesla

Posted on 2016-04-05 01:16:07