Introduction
PugetBench is a suite of performance benchmarks developed by Puget Systems, designed to evaluate hardware using the same real-world creative applications (including Premiere Pro) that professionals rely on every day. The primary focus is on real-world workflows to give relevant, accurate, and reflective results of how your system will perform in production.
In the case of Premiere Pro, this includes testing with a wide range of source codecs (H.264, HEVC, ProRes, RAW, etc.) and with an array of export presets. Since our benchmarks run on top of the host application, we are sometimes limited in what can be tested, depending on what can be executed and recorded through the software’s scripting or plugin API capabilities.
Typically, new versions of the software expand what we can test, but in the upcoming Premiere Pro 25.3 version (currently in beta), a change was made that prevents us from being able to run any HEVC export tests. This results in PugetBench for Premiere Pro failing when running in the recent Premiere Pro betas – 25.3 and 25.4.
Benchmark Failing in Premiere Pro 25.3 and 25.4
Starting in Premiere Pro 25.3, PugetBench for Premiere Pro will fail with a message of “Error: -1” on all of the “Encoding: HEVC […]” tests, usually the second test of a benchmark run. This is the exact error that Premiere Pro is passing to our benchmark application, and does not include any additional details.
Image
Adobe has informed us that this error is caused by a change to Premiere Pro, which will no longer allow scripts to export to HEVC. This change comes into effect with Premiere Pro 25.3 and will be in place for the foreseeable future (including the current 25.4 beta and all future releases).
To be clear, this doesn’t mean that users cannot export HEVC using Premiere Pro – manually exporting to HEVC works the same way it always has. However, this does mean that our benchmark (or any other plugin for Premiere Pro) will no longer be able to trigger an HEVC export in any capacity.
Our Plan Moving Forward
Across all our benchmarks, once a 1.x version is released, we commit to not changing any of the core tests or scoring. While we regularly release updates to fix bugs and add support for new host application versions, all 1.x versions maintain the same core functionality. This approach has held for our Premiere Pro benchmark for roughly 1.5 years (since November 2023). However, this change by Adobe will require an adjustment to the benchmark that impacts its core functionality and output. As a result, for upcoming benchmark versions, we will be incrementing the major version number to clearly differentiate results from previous versions.
To address this limitation, we will soon release a PugetBench for Premiere Pro 2.0.0-beta build that removes all “Encoding—HEVC […]” tests. This version will be available to all licensed users while in a beta state and to free users once it is fully developed. Please note that the LongGOP Score in this beta build will be heavily skewed toward the performance of the remaining H.264 encoding test, since the codec type scores are currently based on a 50/50 split between encoding and processing. We do not believe this accurately reflects real-world performance for users working with LongGOP media, and we are actively exploring solutions that will be implemented in a future update.
Moving to a new major version also gives us the opportunity to make other adjustments to the benchmark that would normally be outside of what we can do as an incremental benchmark update. The full scope of those changes is still in development, but will include a change to the scoring multiplier to help discourage improper comparisons between major benchmark versions.
We understand that many professionals work with HEVC daily, and that this change may especially impact hardware reviewers evaluating HEVC encoding performance. While we’re disappointed to lose this aspect of the Premiere Pro benchmark, users interested in HEVC performance can use our PugetBench for DaVinci Resolve benchmark, which continues to include both 8-bit and 10-bit HEVC encoding tests.
If you want to stay informed of upcoming changes to PugetBench, you can sign up for our newsletter below:
Looking for a Content Creation workstation?
We build computers tailor-made for your workflow.
Don’t know where to start?
We can help!
Get in touch with one of our technical consultants today.