PugetBench for Premiere Pro

At Puget Systems, one of our primary goals is to provide fast and reliable workstations that are perfectly tailored to each customer’s unique workflow. As a part of this, we publish an ongoing series of hardware articles for Premiere Pro – as well as numerous other software packages – in order to discover what hardware configurations are optimal for these specific applications.

However, while this ensures that we are selling the right hardware, it does not give our customers a great idea of how much faster a new workstation might be compared to their existing system. In addition, with the amount of hardware that is currently available, it is impossible for us to test everything on our own, so we want to provide tools for other hardware reviewers to perform real world benchmarking in professional applications like Premiere Pro.

To address both of these issues, we are making our Premiere Pro benchmark available for download so that anyone can can perform the same testing we do in our hardware articles. Even better, this benchmark is compatible with both Windows and Mac-based systems which allows for cross-platform performance comparisons.

Be sure to check out the full list of benchmarks we have available!

System Requirements

• Adobe Premiere Pro version 14.x, 15.x, 22.x, 23.x
• 32GB of memory
• GPU with >4GB of VRAM
• 100GB free space on your drive
• Compatible with both Windows 10/11 and MacOS
• Premiere Pro/OS language must be set to English
• Must be run from a local drive (no network attached storage)

Note that this benchmark is still in BETA. Plug-ins and customized preferences in particular may prevent the benchmark from running properly. If you run into any issues, we recommend resetting your preferences either by following Adobe’s guide or by using our Adobe CC Preference Backup/Restore/Clear utility (Windows only).

How to run the benchmark

1. Add the benchmark plugin to Creative Cloud from the Adobe Exchange
2. Follow the directions to install the plugin on your system
3. Download test assets
4. Launch Premiere Pro and open the appropriate project file for the version of Premiere Pro you are using
   1. Premiere Pro 14.x – “Benchmark_Project.prproj”
   2. Premiere Pro 15.x – “Benchmark_Project_15.prproj”
   3. Premiere Pro 22.x – “Benchmark_Project_22.prproj”
   4. Premiere Pro 23.x – “Benchmark_Project_23.prproj”
5. Follow the setup instructions as shown in the Benchmark Setup sequence.
   1. Windows:
      1. Make sure the Program Window is in the upper-right quadrant of the application
   2. MacOS
      1. Add Premiere Pro to “Security & Privacy -> Accessibility”
      2. When running the benchmark, click “OK” to the window asking to allow Premiere Pro to access “System Events”
6. Start the plugin from “Window->Extensions->PugetBench for Premiere Pro”
7. Configure any settings you want to change (especially if you have a commercial license).
8. Run the benchmark!

While a there are no configuration options in Premiere Pro that need to be set, there are a few things you need to do before running the benchmark depending on your OS.

• Windows: make sure that the program window is in the upper-right quadrant of the Premiere Pro application. We need to use an external app that clicks in this are in order to change the playback resolution.
• MacOS: you need to go to “Security & Privacy” and add Premiere Pro to the Accessibility list. In addition, a few seconds or minutes after the benchmark starts, a popup will come up that asks you to allow Premiere Pro access System Events. Again, this is so that we can change the playback resolution (via the top menu bar).

When the benchmark is complete, it will give you an “Overall Score” and upload the results to to our online database (unless told not to with the commercial version). From the uploaded results, you can view the sub-scores and individual test results.

Troubleshooting

If you have any issues with the benchmark not properly running, first verify that your system meets the System Requirements and that the version of Premiere Pro you are using is supported. If there are no issues with your system or software, there are a number of troubleshooting steps you can try:

• Make sure you are following each step listed in the How to run the benchmark section
• Do not attempt to use the system while the benchmark is running. Defocusing the application window can break the benchmark in many situations.
• If you are on MacOS and have renamed your drive, the benchmark will not be able to get your system specs. The only work-around at the moment is to rename your drive back to “Macintosh HD”, or you can continue to run the benchmark without it being able to pull specs.
• Some MacOS systems will not open the main benchmark project properly, complaining of an unsupported codec. If this happens, make a new empty Premiere Pro project and save it over the “Benchmark_Project_##.prproj” project file.
• Clear the application preferences and reset them to default.​​​​​​ by either using Adobe’s guide or our Adobe CC Preference Backup/Restore/Clear utility (Windows only).
• Disable Windows Defender or any real-time virus scanning programs when you run the benchmark.
• Ensure that your OS and Premiere Pro is set to English and that the path to the benchmark script does not have any special or non-English characters.
• The benchmark does not work over an unmapped network drive (I.E. //192.168.1.2/Benchmark/). Mapped drives should work, but we highly recommend having the benchmark on a local drive as storage performance can affect the results.
• Re-download the benchmark and use a unzipping tool like 7zip to unzip the files rather than the tool that is built into your OS.
• Uninstall and reinstall the benchmark plugin.

Lower than expected performance can be caused by a number of factors, but the most common are:

• Outdated BIOS/drivers/software
• Running background processes (Antivirus, Windows Updates, etc.)
• Windows power profile (High Performance is recommended for the best results)
• Adobe application settings are not correct (GPU acceleration not enabled, etc.)

If you continue to have issues, we recommend posting in the comment section below – ideally with a screenshot of the problem and a copy of any log files that are generated by the benchmark.

What is tested?

Our Premiere Pro benchmark looks at both live playback and export performance with a wide range of codecs at 4K and 8K resolutions. In addition, there are dedicated “Heavy GPU Effects” and “Heavy CPU Effects” sequences that are designed to individually stress the GPU and CPU beyond what a typical Premiere Pro user will do.

Our test media consists of the following resolutions and codecs:

For each type of test media, we look at four test sequences:

1. Standard – Two 59.94FPS clips in series with a Lumetri Color effect applied
2. 2x Forward – Four 59.94FPS clips in series with Lumetri color set to 200% speed in a 119.88FPS sequence to simulate performance when playing at 2x speed.
3. 4x Forward – Eight 59.94FPS clips in series with Lumetri color set to 400% speed in a 239.76FPS sequence to simulate performance when playing at 4x speed.
4. MultiCam – multiple clips across four-six tracks in a multicam sequence. Playback is tested in “Multi-Camera” display mode.

These tests are all used for live playback performance testing at Full playback resolution. The Standard sequence is also used to test export performance using the “Youtube 2160p 4K Ultra HD” preset (H.264, 4K, 40mbps) as well as exporting to 4K ProRes 422HQ 8-bpc.

In addition, we have a “Heavy GPU Effects” sequence using:

• Two ProRes 422 clips in series with a Cross Dissolve between each
• Advanced effects: Lumetri Color, Ultra Key, Sharpen, Gaussian Blur, Basic 3D, Directional Blur, and VR Digital Glitch.
• Extreme effects: Lumetri Color, Ultra Key, Sharpen, Gaussian Blur, Basic 3D, Directional Blur, VR Digital Glitch, and VR De-Noise.
• Performance is measured by exporting to ProRes 422HQ.

and a “CPU Heavy Effects” sequence using:

• Two ProRes 422 clips in series
• Advanced effects: Text layer with keyframed position and scale, and Lens Flare.
• Extreme Effects: Text layer with keyframed position and scale, Dust & Scratches, Noise, and Lens Flare.

• Performance is measured by exporting to ProRes 422HQ.

How does the scoring work?

The scoring system used in our benchmark is based on the performance relative to a reference system with the following specifications:

• AMD Ryzen 5900X 12 Core
• NVIDIA GeForce RTX 3080 10GB
• 64GB of RAM
• Samsung 960 Pro 1TB
• Windows 10 Pro
• Adobe Premiere Pro 2020 (ver. 14.8)

The performance in FPS for each individual task is compared to the reference result and used to calculate the Live Playback, Export, GPU, and Effects scores. For example, for these sub-scores, a score of “100” would match the reference system, while a score of “80” would indicate performance roughly equal to 80% of the reference system.

The Live Playback and Export scores are all combined into the “Standard”, and “Extended” Overall scores depending on the benchmark preset you chose. Here, we multiply the scores by 10 in order to make sure the Overall scores are larger than the sub-scores (since bigger number = more important). Due to this, a score of “1000” would match the reference system, while a score of “800” would indicate performance roughly equal to 80% of the reference system.

The GPU Score is the average of the results for the GPU Effects tests as well as the H.264 export tests.

Update Log

Version 0.95.7 BETA

• Updated CLI utility to support changes to plugin paths when installing via Creative Cloud

Version 0.95.6 BETA

• Added plugin and CLI support for Premiere Pro 23.0
• Updated project files to support Premiere Pro 23.0
• Updated benchmark upload/view URLs to match web hosting changes

Version 0.95.5 BETA

• Added filter to remove “(C)”, “(R)”, and “(TM)” strings from the systems specifications
• Added additional granularity to the “Custom” preset. You can now select individual tests to run rather than only being limited to specific groups of tests.
• Added the ability to specify which tests to run with the “Custom” preset when using the CLI
  • Include the “/custom_tests” argument when launching the CLI along with a comma-separated list of test numbers.
  • The test number associated with each test can be seen if you launch the benchmark plugin manually. It will be shown before the name of each selectable test.
  • Example: /custom_tests “8,21”
• Added “/cooldown” CLI argument to specify how many seconds to wait at the end of the benchmark run to allow the system to cool down. By default, the CLI waits for 20 seconds
  • Example: /cooldown 60
• Added the ability to launch custom .bat files at various points during the benchmark run. This was requested multiple times in order to automate additional logging using utilities such as Windows Performance Recorder.
  • If running the benchmark manually, there is now a “Run custom commands” checkbox in the automation settings
  • If using the CLI, include the command line argument of “/custom_tests”
  • Four different .bat files are launched at the start/end of the benchmark, and at the start/end of each individual test.
  • More information on how to make the .bat files, where to save them, and a sample set of files is available in the user guide
  • Note that if the commands require admin privileges, you will need to launch the application or CLI as admin.
• Required application information for the CLI moved to an .ini file that resides alongside the CLI in the plugin folder. This information was previously baked into the CLI itself, but by having it in an editable file, end users can add support for things like beta and pre-release versions of Premiere Pro. The .ini requires the following entries:
  • Main section title: The major version of Premiere Pro. This is used by the CLI for the “/app_version” argument
  • prefLoc: Location of the application preferences folder
  • prefFile: In many cases, this is the same as prefLoc, but some benchmarks need a specific preferences file defined
  • appLoc: Path to the Premiere Pro application
  • appEXE: Name of the Premiere Pro .exe when it has been launched

Version 0.95.4 BETA

• Remade Benchmark_Project_22.prproj file in order to fix bug with Apple M1 devices.
• Fixed issue with “Benchmark Complete” dialog box showing the Standard Overall Score when the Extended preset is run.
• Updated spec gathering to include GPU core count for M1 devices.
• Fixed bug that prevented the benchmark from being run multiple times (we still highly recommend restarting Premiere Pro between benchmark runs, however).
• Switching to “qe.project.getActiveSequence().player.totalFrames” to get the number of total frames processed for Live Playback tests. Previously we just used the total number of frames in the sequence, but Pr 22.0 at times will process a frame multiple times. Especially on low-end systems, this can throw off the results since it makes it possible for Pr to drop a single frame more than once.

Version 0.95.3 BETA

• Added project files and CLI support for Premiere Pro version 22.x
• In order to run the benchmark, you must have both the latest plugin version, and download the latest test assets so that the project files for Premiere Pro 22.x are present.

Version 0.95.1 BETA

• Added project files and CLI support for Premiere Pro 2021 (15.x)
• In order to run the benchmark, you must have both the latest plugin version, and download the latest test assets so that the project files for Premiere Pro 2021 are present.

Version 0.95 BETA (Major Update)

• Major update – Results cannot be compared to previous benchmark versions
• Requires new project files due to significant changes to what is tested
• Added additional failsafes for when gathering of system specs fails
• Added spec gathering support for Apple M1 systems
• In some cases, playback can report more dropped frames than exist in the sequence. When this happens, the playback FPS will be reported as “0”
• Standard and Extended tests adjusted. Old test sets were too easy given recent hardware/application updates, resulting in many of the live playback tests in particular running at full FPS on most systems
• Test changes also significantly reduced the amount of time it takes to run the benchmark
• Scoring changed to be relative to a reference system (AMD Ryzen 5900X w/ GeForce RTX 3080)
• Test changes:
  • Dropped half resolution Live Playback tests
  • Added 2x and 4x speed forward playback tests
  • Increased ProRes multicam test to 6 streams
  • Shortened all test sequences from 30 sec to 20 sec in length
  • Export tests now use in/out points so that we can shorten some tests (CPU effects test primarily)
  • H.264 export tests now count towards the GPU score since it uses GPU encoding by default
  • CPU & GPU Effects sequences no longer do live playback tests, and only export to ProRes. Live Playback results are often not consistent with this focused of a load, and this change should make the scoring more accurate
  • Shortened CPU Effects export test to 1/3 the length since this test can take a long time
  • GPU/CPU effects tests now have their own “Effects” category and sub-score
  • Overall score is now the average of the “Live Playback”, “Export”, and “Effects” sub-scores
  • Scoring now based on performance relative to a reference system (Ryzen 5900X w/ RTX 3080) rather than relative to the media FPS
  • Dropping 23.976 FPS media from Enhanced Extended tests. Extended will now only adds sequences with 8K 59.94FPS H.265 and R3D media
• “Custom” preset is now a licensed feature and allows you to select individual tests to run. Custom preset can still be set via the CLI utility, but requires you to manually configure and save the settings at least once before running the benchmark from the command line.

Version 0.92 BETA

• Moved to using node-wmi to gather system specs on Windows systems. This should be significantly faster and more reliable than the previous command line method
• Added GPU driver and motherboard BIOS to the system specs for Windows systems
• CLI utility now updates the Pr preferences so that workspaces are imported when opening a project (default behavior was changed in Pr 14.3.1)
• Misc bug fixes

Version 0.91 BETA

• Improved system spec gathering reliability
• System specs on MacOS now gather properly even if the system drive’s name has been changed from the default “Macintosh HD”

Version 0.9 BETA (Major Update)

• Results are now uploaded to our online database. This is required for the free version, but opt-in if you have a commercial license.
• Removed the temporary .csv logging from the free version now that the full results can be viewed on our benchmark listing page.
• Complete plugin UI revamp. Added status bar/text, and improved the configuration options.
• Improved method used to gather system specs. This should break less often on unusual or older system configurations.
• License validation moved from the CLI utility to the plugin itself.
• Added the ability to do a “Custom” test which does the same tests as normal, only on whatever sequences are on the root-level of the currently opened project.
• Added ability to set the location for any disk-based tests (exporting).
• Added tooltips for the various settings that can now be configured.
• Status logs and configuration settings moved to “~Documents⧵PugetBench⧵Premiere Pro⧵TIMESTAMP” since we cannot log directly to the plugin folder.
• Dropped 8K H.265 tests. This test was simply too difficult for most systems and resulted in Premiere Pro hanging more often than not.
• Dropped ability to run tests in individual codecs. The intention is that the “Custom” test type will replace this.
• Improved CPU and GPU effects test. Added “Advanced” and “Extreme” tests to differentiate between what someone might be doing in their own projects versus a worst-case scenario. Also dropped exporting to H.264 for these tests since we want to focus on the system’s performance for processing CPU and GPU effects
• Scoring has been adjusted based on the test changes. Due to this, the Overall Score will not be consistent with previous versions.
• General bug fixes and stability improvements.

Version 0.88 BETA

• Slight change in benchmark plugin folder name. Be sure to uninstall any existing “PugetBench for Premiere Pro” plugins before install the new 0.88 version.
• Fixed issue where the benchmark would not run due to a change in the API when using confirm boxes.
• Added “GPU Score” that is the average of the results from the “GPU Effects” tests.

Version 0.86 BETA

• Dropped the “Benchmark Results” score screen due to Premiere Pro 2020 breaking MGRT scripting functionality
• Adding back .csv log file support until we can get the “Benchmark Results” screen working
• No longer generating the test sequences on the fly – switching to pre-made projects for each test codec. This greatly increases the overall stability of the benchmark
• Due to using pre-made projects, Premiere Pro 2019 is no longer supported
• Improved system information gathering methods. This should make it much more reliable on MacOS
• Added option to continue the benchmark if the system information gathering does fail for any reason
• Miscellaneous bug fixes and improvements

Version 0.8 BETA

• Renamed benchmark to “PugetBench for Premiere Pro”
• Added support for Premiere Pro 2020
• Dropping the 29.97 FPS CPU and GPU effects tests. These are already somewhat artifical, so no reason to run them at both 29.97 and 59.94 FPS. This means that the Extended test scores will be slightly different than previous versions, but it shouldn’t be by a huge amount
• There is now a “Benchmark Results” screen that comes up at the end of the benchmark that displays a bunch of useful information including: benchmark version, cores, results for each individual test, and system information like CPU, RAM, OS, GPU, and Premiere Pro version
• The benchmark also makes a PNG of the “Benchmark Results” screen for easy sharing
• Removed .csv log file support in the free edition (log files will be a feature in the commercial use version)
• Removed “start” .exe applications (automation will be a feature in the commercial use version)
• More code to ensure consistency between benchmark runs
• Miscellaneous bug fixes and improvements

Version 0.3 BETA

• Replaced H.264 media with footage straight from a Panasonic GH5 (Thanks Neil Purcell!)
• Replaced “Quick” preset with “Standard” preset. Changed tested media to 59.94FPS instead of 29.97FPS.
  • This was done with the intention of having a benchmark preset that is geared towards measuring the overall performance of a system in as little time as possible.
• Separated media files into two groups: a “Standard” set and a “Full” set to make the download much more manageable for people just doing the Standard preset.
• Due to the change in the test media, the overall scores are not interchangeable with previous benchmark versions

Version 0.2 BETA

• First release.