Puget Systems print logo

https://www.pugetsystems.com

Read this article at https://www.pugetsystems.com/guides/1458
Article Thumbnail

RealityCapture Benchmark

Written on May 3, 2019 by William George
Share:

CapturingReality Logo (all rights to this image belong to CapturingReality, makers of RealityCapture)

Here at Puget Systems, we have put together a benchmark utility for RealityCapture which measures system performance by running two small projects - a 3D Model and a 3D Map - and tracking the time taken to process each step. This benchmark is freely available to download, below, though running it requires an existing installation and valid license of RealityCapture on a 64-bit Windows computer.

We built and tested this benchmark using the full version of RealityCapture, for which a free demo is available with registration, and it is not known if it will work with RealityCapture Promo or PGM variants. If you try it on one of those, please let us know in the comments!

How to Use the RealityCapture Benchmark

Using this benchmark is quite simple, though you do need to have RealityCapture installed and logged in prior to starting.

  1. Download the RealityCapture Benchmark file by clicking on the blue button above
  2. Once downloaded, unzip "Puget Systems RealityCapture Benchmark.zip" to a location where you have write permission
  3. After it has fully extracted, open the Puget Systems RealityCapture Benchmark folder
  4. Run the RealityCaptureBenchmark.exe file
  5. Wait for it to detect the version of RealityCapture that is installed
  6. At the intro screen, read the description and then click on Start to begin
  7. Avoid using the mouse or keyboard during the benchmark execution - changing focus at the wrong time will interfere with it
  8. When the benchmark is complete, a summary screen will be displayed with the time (in seconds) for processing each step
  9. That information will also be saved to a results file in the same folder, along with some basic system specs, for future reference

You can compare the results from running this benchmark on your system to those we show in future RealityCapture articles.

Please also share your results and system specs in the comments below!

Update Log

5-3-2019: Initial public release
5-23-2019: Update to benchmark, fixing a bug that could cause the script to get stuck locating photos when opening a project file

Tags: RealityCapture, Benchmark, CPU, GPU, Performance, photogrammetry
marik c

Benchmark Started at 05:47 on 05-21-2019
RealityCapture Version: 1.0.3.6310
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
GPU: NVIDIA GeForce RTX 2080 Ti
Project: Rock Model
Align Images: 24.6
Calculate Model: 98.9
Colorize: 20.9
Texture: 371.7
Total: 516.1 seconds
Project: School Map
Align Images: 34.5
Calculate Model: 193.4
Colorize: 29.3
Texture: 204.8
Ortho Projection: 140.2
Total: 602.2 seconds
Benchmark Completed at 06:10

Posted on 2019-05-21 19:57:28

Thank you for sharing your results! :)

Posted on 2019-05-21 20:01:29
Milos Lukac

Hi Villliam
Would it be possible that you make some sort of Google DOC where can people post the results, so you get much bigger test base and result page ?

If YES i can post it in our 3D scanning forum + few other with approx 50k+ user base...

Posted on 2019-06-02 12:04:21

At some point I would like to build a section on our website where folks could upload results and compare them, but that is probably a long way off at this point. I could see a Google doc as a stop-gap measure until such a portal was ready... I will give that some more consideration.

Posted on 2019-06-03 16:37:16
Lorenz

Hi, I created a Google Form to collect the benchmark data and I'm working now on the automatic evaluation of them. Can I post the link to the form here? and in the forum of the different communities? All the data will be published later when I nicely organized it an will be public available. Is that fine? @William contact me otherwise on Twitter

Posted on 2019-06-17 08:58:17

I am fine with that, but just FYI Milos Lukac has also talked about setting up something similar. Anyone is welcome to use our benchmark tests & results, as long as you comply with the licensing terms... which basically just ask that you don't change the benchmark and that you cite us as the source and link to us. Should be pretty straightforward, but feel free to ask questions here, email me (william [at] pugetsystems [dot] com) or message me on Twitter (@PSWilliamGeorge).

Posted on 2019-06-17 16:37:48
Lorenz

Hi William, Thank you for the answer! Then I will post the Link here. Didn't ask Milos you for setting up a Google Docs? :D
With the citing I do for sure! Thank you very much!

Posted on 2019-06-17 20:23:38
marik c

Benchmark Started at 13:02 on 05-21-2019
RealityCapture Version: 1.0.3.6310
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
GPU: NVIDIA GeForce RTX 2080 Ti
GPU: NVIDIA GeForce GTX 1080 Ti
Project: Rock Model
Align Images: 12.8
Calculate Model: 44.4
Colorize: 15
Texture: 376.7
Total: 448.9 seconds
Project: School Map
Align Images: 17.8
Calculate Model: 80.3
Colorize: 19
Texture: 199.6
Ortho Projection: 139.4
Total: 456.1 seconds
Benchmark Completed at 13:22

Posted on 2019-05-21 20:25:23

Nice - so it looks like adding the second GPU substantially sped up image alignment, model calculation, and colorization - but not making the texture or ortho projection. That is good to know ahead of my planned multi-GPU testing :)

Posted on 2019-05-22 20:05:17

I'm starting to do multi-GPU testing in RealityCapture, and when going from one RTX 2080 Ti to two I did not see as big of an improvement in performance as you got in the results above. I only got a 5-10% reduction in processing time, while your results look like 10-20% or maybe even a little more.

Is there somewhere in RealityCapture's settings that I need to ensure both GPUs are enabled / selected? Or anything else you know of that I could be overlooking?

Posted on 2019-05-28 19:49:13
marik c

You need to enable the 2nd GPU in the reconstruction settings, but I think that's the only place such an option shows up strangely enough. I'd keep an eye on performance monitor CUDA to make sure it runs on both cards at the same time. My CPU is also at 5ghz all cores if that helps with overhead,

EDIT: Also I am kind of "cheating" extra pcie lanes. I have the full x16 for the 2080ti, and then up to 4x lanes from the PCH as the 1080ti is plugged in egpu thunderbolt style. RC seems to like very fast nvme A LOT though, so I'm not sure if competing with those PCH lanes is good or bad for me. I'll try a few more weird setups and post results.

Posted on 2019-05-29 01:52:05

Okay, I did enable that one setting for the second GPU - I just wasn't sure if there were other places it needed to be turned on as well, etc. Thank you for confirming :)

Thinking through the PCI-E lanes and such, the system I have been using would be x8 / x8 with two video cards installed, both coming from the CPU, while the NVMe is x4 coming (I think) from the PCH. The CPU isn't quite as fast as yours, topping out at 4.7GHz all-core (i9 9900K at stock speeds). Maybe that extra headroom lets the GPUs flex more? Or maybe somehow the 1080 Ti is behaving differently / better than a second 2080 Ti? Hmm, I'll do some more testing and see what I can find.

Posted on 2019-05-29 16:04:56
marik c

Ok silly me, I did not clear the RC's cache between runs which is what caused the performance jumps. It seems an egpu actually slows the benchmark probably due to overhead:

Benchmark Started at 09:14 on 05-29-2019
RealityCapture Version: 1.0.3.6310
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (5ghz)
GPU: NVIDIA GeForce RTX 2080 Ti (internal x16 to cpu)

GPU: NVIDIA GeForce GTX 1080 Ti (thunderbolt x4 to pch)

Project: Rock Model
Align Images: 24.7
Calculate Model: 93
Colorize: 21.2
Texture: 379.7
Total: 518.6 seconds
Project: School Map
Align Images: 34.7
Calculate Model: 171.5
Colorize: 29.6
Texture: 210
Ortho Projection: 167.2
Total: 613 seconds
Benchmark Completed at 09:37

RC's performance is still a mystery to many (if not all?!). I am pretty confident that the bulk of gpu impact is seen during the model reconstruction, however even then it is not much until you get to large models in highest quality. I notice during calculate model the gpu's CUDA work shoots up to 100%, stays there until the step is around halfway done, then the gpu does some copy work and the latter half is done all by the CPU.

I think to test multi GPU benefit you would need to bias a benchmark solely towards the model reconstruction, on high quality, with settings tuned for higher quality still. The CUDA portion of the work is done so quickly, there has to be a lot of it to do to benefit from multi gpu.

Posted on 2019-05-29 19:34:35

Fascinating... though now you have me worried, since I wasn't aware that RC cached data that wasn't cleared by simply exiting the program and restarting it (which should happen between every processing step in the benchmark). How is the cache manually cleared in RC? I should do some testing to make sure my own results are not being impacted by that :/

Speaking of my own testing, internally I do use a couple of larger image sets that wouldn't be feasible to include in the public benchmark (too large and too time consuming). They are still run at the same quality settings as everything in the public tests, just with many more photos. If you are interested, check out the article I published yesterday and feel free to leave feedback :)

https://www.pugetsystems.co...

Posted on 2019-05-29 20:05:08
marik c

Great article, makes the mystery deeper for me; so RC doesn't care much for extra cpu cores, uses CUDA for only a few steps in the process, and it barely even uses RAM! Yet it is hard to deny the performance is exceptional.

I did optimize my RC in the past so the caching may be off by default. It's in Workflow > Settings > Clear Cache. Also since a lot of work is done out of memory, I noticed a good speed boost by setting my cache location to a fast nvme that is external from OS.

Posted on 2019-05-30 05:39:14

Hmm, it looks like the "Clear cache on exit" option is set to False by default... I wonder if that may have impacted my results at all. I'm going to do a quick single-pass through the full RC benchmark with that on False and then again with it on True, and see if that makes any difference. If it does, I may need to redo the testing for this article :/

Posted on 2019-05-31 17:12:40

Okay, here are results from runs with "Clear cache on exit" off (false) and then on (true):

OFF
-----System Specs-----
Computer Name: DESKTOP-10G9R51
CPU: Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz
GPU: NVIDIA GeForce RTX 2080 Ti (26.21.14.3039)
RAM: 128GB
Storage: NVMe Samsung SSD 960 SCSI Disk Device (C:)
OS Version: Windows 10 Pro (Version 1809)
----------
RealityCapture 1.0.3.6310,Rock Model,Align Images,14.4,seconds
RealityCapture 1.0.3.6310,Rock Model,Calculate Model,132.4,seconds
RealityCapture 1.0.3.6310,Rock Model,Colorize,11.6,seconds
RealityCapture 1.0.3.6310,Rock Model,Texture,117,seconds
RealityCapture 1.0.3.6310,Rock Model,Total,275.4,seconds
RealityCapture 1.0.3.6310,School Map,Align Images,15.6,seconds
RealityCapture 1.0.3.6310,School Map,Calculate Model,236.3,seconds
RealityCapture 1.0.3.6310,School Map,Colorize,17.9,seconds
RealityCapture 1.0.3.6310,School Map,Texture,166.2,seconds
RealityCapture 1.0.3.6310,School Map,Ortho Projection,46.5,seconds
RealityCapture 1.0.3.6310,School Map,Total,482.5,seconds
RealityCapture 1.0.3.6310,School Model,Align Images,99.4,seconds
RealityCapture 1.0.3.6310,School Model,Calculate Model,1324.8,seconds
RealityCapture 1.0.3.6310,School Model,Colorize,99.9,seconds
RealityCapture 1.0.3.6310,School Model,Texture,618.5,seconds
RealityCapture 1.0.3.6310,School Model,Total,2142.6,seconds
RealityCapture 1.0.3.6310,Park Map,Align Images,315.8,seconds
RealityCapture 1.0.3.6310,Park Map,Calculate Model,2571,seconds
RealityCapture 1.0.3.6310,Park Map,Colorize,240.8,seconds
RealityCapture 1.0.3.6310,Park Map,Texture,1638.8,seconds
RealityCapture 1.0.3.6310,Park Map,Ortho Projection,166.9,seconds
RealityCapture 1.0.3.6310,Park Map,Total,4933.3,seconds

ON (and manually cleared the cache prior to starting, just to be safe)
-----System Specs-----
Computer Name: DESKTOP-10G9R51
CPU: Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz
GPU: NVIDIA GeForce RTX 2080 Ti (26.21.14.3039)
RAM: 128GB
Storage: NVMe Samsung SSD 960 SCSI Disk Device (C:)
OS Version: Windows 10 Pro (Version 1809)
----------
RealityCapture 1.0.3.6310,Rock Model,Align Images,15.3,seconds
RealityCapture 1.0.3.6310,Rock Model,Calculate Model,133.4,seconds
RealityCapture 1.0.3.6310,Rock Model,Colorize,11.6,seconds
RealityCapture 1.0.3.6310,Rock Model,Texture,117.4,seconds
RealityCapture 1.0.3.6310,Rock Model,Total,277.7,seconds
RealityCapture 1.0.3.6310,School Map,Align Images,15.7,seconds
RealityCapture 1.0.3.6310,School Map,Calculate Model,238.4,seconds
RealityCapture 1.0.3.6310,School Map,Colorize,17.8,seconds
RealityCapture 1.0.3.6310,School Map,Texture,166.7,seconds
RealityCapture 1.0.3.6310,School Map,Ortho Projection,46,seconds
RealityCapture 1.0.3.6310,School Map,Total,484.6,seconds
RealityCapture 1.0.3.6310,School Model,Align Images,95,seconds
RealityCapture 1.0.3.6310,School Model,Calculate Model,1318.4,seconds
RealityCapture 1.0.3.6310,School Model,Colorize,99,seconds
RealityCapture 1.0.3.6310,School Model,Texture,620,seconds
RealityCapture 1.0.3.6310,School Model,Total,2132.4,seconds
RealityCapture 1.0.3.6310,Park Map,Align Images,305,seconds
RealityCapture 1.0.3.6310,Park Map,Calculate Model,2565.7,seconds
RealityCapture 1.0.3.6310,Park Map,Colorize,239.3,seconds
RealityCapture 1.0.3.6310,Park Map,Texture,1641.6,seconds
RealityCapture 1.0.3.6310,Park Map,Ortho Projection,166.2,seconds
RealityCapture 1.0.3.6310,Park Map,Total,4917.8,seconds

So basically the same performance both ways, within less than one percent (margin of error). It looks like, at least for my testing here, having that setting in place did not matter. Whew! Glad I don't have to re-test everything :)

Posted on 2019-05-31 22:07:56
marik c

Interesting, and your cache location is set to a persistent disk (not a ram drive)? For me if I run the benchmark twice in a row with cache saving I get the performance boosts as seen earlier, however clearing cache makes the next run results the same as my original single card run.

Posted on 2019-05-31 22:28:48

I didn't alter the location of the cache as far as I can recall, and it looks like it is set to the Windows temp folder.

OOOHHH! I think I know what is going on. Did you move your cache? I'd bet $20 that you moved it from its default location in the Windows temp folder :)

See, I didn't realize there were settings in RealityCapture for cache... but I did notice that it was writing files to the temp folder, so in my benchmark it deletes the folder that RC makes in there at the end of every run. So I am manually clearing the cache, in effect, but only if the user hasn't moved it. Since you did (I assume) move yours, that is why it is impacting the performance if the cache is not cleared between runs from within RC.

Posted on 2019-05-31 22:45:00
marik c

Indeed, I moved my cache to a nvme that is dedicated just for that purpose.

Posted on 2019-05-31 23:38:40

Okay, that all makes sense now :)

I will try and see if I can automate a better way of clearing the cache regardless of its location, so that it makes for a more fair comparison when other people run the benchmark. Thank you so much for your input on this!

Posted on 2019-06-01 04:09:31
Milos Lukac

Have you done the testing even with the W10 scheduler "fix" on the AMD system ? that can dramatically improve the speed..

https://hexus.net/tech/news...

https://www.phoronix.com/sc...

Posted on 2019-06-02 12:15:14

In my recent CPU performance article (https://www.pugetsystems.co... ) I was using AMD's Ryzen Master software and made sure that Dynamic Local Mode was enabled, and all Windows updates were applied, but I was *not* using any additional 3rd-party software to try and improve the scheduling on those chips. Given that higher core count / slower clock speed chips from both AMD and Intel did poorly in RealityCapture, I don't think that improving the behavior of Windows' scheduler would help much. RC seems to really like high clock speed without so much concern for core count.

Posted on 2019-06-03 16:17:55
Milos Lukac

Try it, as i send you some additional info on what is going on there

https://bitsum.com/portfoli...

Posted on 2019-06-06 10:56:50

Unfortunately, it doesn't look like CorePrio helps in photogrammetry applications: https://www.pugetsystems.co...

Posted on 2019-06-11 22:33:12
Milos Lukac

I was in the DEV of RC from ( Not anymore ) the beginning so if you need expertise on how to set up benchmark and etc feel free to PM me over Facebook, i send you PM there already today...

Posted on 2019-06-02 12:39:27

I have updated my benchmark tool to enable the "Clear cache on exit" option within RealityCapture, rather than relying on manually deleting the RC subfolder in the Windows temp folder. Before I put it in the public benchmark .zip file, would you be able to give it a try and make sure it sets that option properly and that performance is (fairly) consistent between runs? If so, drop me an email at william [at] pugetsystems [dot] com and I can send you the .exe file as either an attachment or a Google Drive link.

Posted on 2019-06-10 21:37:27

I've been waiting forever for some benchmarks for RC. I'm curious to know if the GPU is the primary difference in the results shown here or if it's the Intel chip. Here's my results from 2 tests:

Benchmark Started at 10:18 on 05-30-2019
RealityCapture Version: 1.0.3.6310
CPU: AMD Ryzen 7 1700X Eight-Core Processor @ 3.5GHz
GPU: NVIDIA GeForce GTX 1070
Project: Rock Model
Align Images: 30.5
Calculate Model: 168.9
Colorize: 14.7
Texture: 139.7
Total: 353.8 seconds
Project: School Map
Align Images: 38.5
Calculate Model: 347.8
Colorize: 22.3
Texture: 191.2
Ortho Projection: 9.4
Total: 609.2 seconds
Benchmark Completed at 10:39

Benchmark Started at 10:38 on 05-31-2019

RealityCapture Version: 1.0.3.6310
CPU: AMD Ryzen 7 1700X Eight-Core Processor @3.8GHz
GPU: NVIDIA GeForce GTX 1070
Project: Rock Model
Align Images: 29.9
Calculate Model: 165.6
Colorize: 14
Texture: 137.1
Total: 346.6 seconds
Project: School Map
Align Images: 38.8
Calculate Model: 320.2
Colorize: 21.2
Texture: 188.5
Ortho Projection: 9.3
Total: 578 seconds
Benchmark Completed at 10:58

Posted on 2019-06-03 21:46:19

We are putting out some articles looking at CPU and GPU performance using this benchmark (and a couple of larger image sets too, which were too big / time consuming to include in this public test). The first article is out now, looking at some CPUs, and another looking at GPUs will be coming later this week:

https://www.pugetsystems.co...

Also, just a side note, I have observed variance of up to about 10% between benchmark runs on the same hardware - so I usually do 2 or 3 runs and take the fastest result for my articles. However, another reader also helped me uncover the fact that if you have moved the RC cache away from its default location (in the Windows temp folder) then my benchmark does not properly clear the cache between runs, so any after the first run will be artificially faster. I am hoping to address that in the next update.

Posted on 2019-06-03 21:52:00

I saw the same results when moving to the slower clock speed, read the discussion and these results are after clearing the cache. Thx!

Posted on 2019-06-03 21:54:00
aya:Tinkering-man:100%UnOffici

Benchmark Started at 23:43 on 06-06-2019
RealityCapture Version: 1.0.3.6310
CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
GPU: NVIDIA GeForce GTX 1080
Project: Rock Model
Align Images: 39.2
Calculate Model: 760.6
Colorize: 17.3
Texture: 44.2
Total: 861.3 seconds
Project: School Map
Align Images: 59.3
Calculate Model: 2197.6
Colorize: 25.2
Texture: 87.9
Ortho Projection: 9.3
Total: 2379.3 seconds
Benchmark Completed at 00:43

Posted on 2019-06-06 15:48:01
Lorenz

Hi, what about RAM configuration, motherboard, strorage type etc. It would be really nice to include them into the bemchmakr for further investigation.
What do you think?

Posted on 2019-06-10 14:26:52

As I have gotten to better understand RealityCapture, I do think that perhaps recording storage and RAM would be nice... but it gets tricky. With storage in particular, the most applicable parts would be the drive where the projects / images are kept and then also where the cache is located. Here in our Labs department, it is generally all done on a single, very fast NMVe SSD... but in the real world, I bet a number of people have multiple drives in their workstations - and I am not sure how to accurately determine which drives (if there are multiple) are being used for which roles. Without that info, simply having a list of drives would not be very informative.

RAM would be a bit easier to record, though. I am planning on doing some testing soon across different amounts of RAM, and if I do see any difference between various amounts of memory (with all other things being equal) then I will likely add that to the system data that is recorded. If it makes no difference, perhaps I'll skip over it. I guess RAM speed could also be a factor, in some systems... hmm...

I am very much open to suggestions, though, so if you think there are easier ways of recording this info or some aspect of it that I am not considering then please let me know :)

Posted on 2019-06-10 15:43:17
Lorenz

That with the different storage configuration is a good point and probably also really hard to evaluate. Esp. depends also on the free disk space, the defragmentation level etc. So a speedtest upfront would be needed anyway.

But for the RAM it would be definitely interesting what is more beneficial; more RAM or faster and what's the balance of the two w.r.t. efficiency and price (better get more slower RAM vs less fast RAM) - so really looking forward to you RAM test!

And probably you totaly aware of this:https://www.hwinfo.com/down... has a library for in-depth pc components assessment, so if possible would be cool to include this! - esp. for the folks which have there CPU OC ;D

Posted on 2019-06-14 12:16:16
Lorenz

Hi All! For consolidating the results of the benchmark of Puget Systems (! thank you for that !!) I created a google docs and there you can post your results. I will try to make a summary public as fast as possible and if a critical amount submit their results ;D - All the data will be public asap!

Thank you very much!! https://forms.gle/9AAndxszt...

Posted on 2019-06-17 20:26:36
Nino Skupnjak

Little older system :)

Benchmark Started at 09:19 on 06-26-2019
RealityCapture Version: 1.0.3.6310
CPU: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
GPU: Intel(R) HD Graphics 4600 (GPU is actually 1060 6gb)

Project: Rock Model
Align Images: 38
Calculate Model: 746.4
Colorize: 20.5
Texture: 233.8
Total: 1038.7 seconds
Project: School Map
Align Images: 46.5
Calculate Model: 1805.6
Colorize: 29.8
Texture: 305.1
Ortho Projection: 13.9
Total: 2200.9 seconds
Benchmark Completed at 10:20

Posted on 2019-06-26 08:44:38

Thank you for sharing your results :)

Posted on 2019-07-01 16:52:24