Skip to content

Main Navigation

Puget Systems Logo
  • Solutions
    • Recommended Systems For:
    • Content Creation
      • Photo Editing
        • Recommended Systems For:
        • Adobe Lightroom Classic
        • Adobe Photoshop
      • Video Editing
        • Recommended Systems For:
        • Adobe After Effects
        • Adobe Premiere Pro
        • DaVinci Resolve
        • Foundry Nuke
      • 3D Design & Animation
        • Recommended Systems For:
        • Autodesk 3ds Max
        • Autodesk Maya
        • Blender
        • Cinema 4D
        • Houdini
        • ZBrush
      • Real-Time Engines
        • Recommended Systems For:
        • Game Development
        • Unity
        • Unreal Engine
        • Virtual Production
      • Rendering
        • Recommended Systems For:
        • Keyshot
        • OctaneRender
        • Redshift
        • V-Ray
      • Digital Audio
        • Recommended Systems For:
        • Ableton Live
        • FL Studio
        • Pro Tools
    • Engineering
      • Architecture & CAD
        • Recommended Systems For:
        • Autodesk AutoCAD
        • Autodesk Inventor
        • Autodesk Revit
        • SOLIDWORKS
      • Visualization
        • Recommended Systems For:
        • Enscape
        • Lumion
        • Twinmotion
      • Photogrammetry & GIS
        • Recommended Systems For:
        • ArcGIS Pro
        • Agisoft Metashape
        • Pix4D
        • RealityCapture
    • Scientific Computing
      • Recommended Systems For:
      • Data Science
      • Machine Learning / AI
      • Scientific Computing
    • More
      • Recommended Systems For:
      • Compact Size
      • Live Streaming
      • NVIDIA RTX Studio
      • Quiet Operation
      • Virtual Reality
    • Business & Enterprise
      We can empower your company
    • Government & Education
      Services tailored for your organization
  • Products
    • Intel Core i7 & i9
      Workstations with 13th Gen Intel Core i7 & i9 processors on Z690 and Z790 chipsets
    • AMD Ryzen 7 & 9
      Workstations with AMD Ryzen 7000 Series processors on B650 and X670 chipsets
    • AMD Threadripper PRO
      Workstations with AMD Threadripper PRO 5000 WX processors on the WRX80 chipset
    • Intel Xeon W
      Workstations with Intel Xeon W 2400 and 3400 processors on the W790 chipset
    • Rackmount & Server
      Servers and workstations in rackmount chassis
    • Custom Computers
      Customize a desktop workstation from scratch
    • Custom Servers
      Customize a rackmount server from scratch
    • QNAP Network Attached Storage
      Check out our external storage options as an authorized reseller for QNAP
    • Recommended Third Party Peripherals
      View our list of recommended peripherals to use with your new PC
  • Publications
    • Articles
    • HPC Blog
    • Blog Posts
    • Case Studies
    • Podcasts
    • Press
  • Support
    • Contact Support
    • Support Articles
    • Warranty Details
    • Onsite Services
  • About Us
    • About Us
    • Contact Us
    • Enterprise
    • Gov & Edu
    • Press Kit
    • Testimonials
    • Careers
  • Talk to an Expert
  • My Account
  1. Home
  2. /
  3. Hardware Articles
  4. /
  5. Stable Diffusion Benchmark Testing Methodology
Stable Diffusion Automatic 1111 web interface

Stable Diffusion Benchmark Testing Methodology

Posted on July 31, 2023 (August 10, 2023) by Matt Bach

Table of Contents

  • Introduction
  • Stable Diffusion Implementations
  • Settings & Models
  • Recommended Prompts for Benchmarking
    • Text to Image
    • Image to Image
  • Measuring Performance
  • Automation

Introduction

Stable Diffusion (most commonly used to convert text into images) is a growing application of AI technology in the content creation industry. Unlike many workflows that utilize commercially-developed software (Photoshop, Premiere Pro, DaVinci Resolve, etc.), many commonly used Stable Diffusion applications are open source and constantly evolving.

Stable Diffusion Automatic 1111 web interface
Stable Diffusion Automatic 1111 Web UI
Image
Open Full Resolution

To drive Stable Diffusion on your local system, you need a powerful GPU in your computer that is capable of handling its heavy requirements. Not only will a more powerful card allow you to generate images more quickly, but you also need a card with plenty of VRAM if you want to create larger-resolution images.

In this article, we will present our methodology for benchmarking various GPUs for Stable Diffusion. This includes what implementations of Stable Diffusion we recommend, system configuration, testing prompts, automation, and scoring calculation. With a wide technology like this, there are effectively infinite ways to measure performance. These guidelines are not intended to be definitive or absolute, but rather to act as a common baseline.

Stable Diffusion Implementations

While it is possible to directly use the version of Stable Diffusion that Stability AI and Runway developed, most people use one of the web-based graphical UI created by third parties. The most common are:

  • Automatic 1111 – Automatic 1111 is most commonly used with NVIDIA GPUs, although forks are available for AMD and Apple Silicon. It allows for the use of xformers which can significantly increase performance on NVIDIA GPUs. There is also OpenVINO support through a custom script to run it on Intel GPUs.
  • SHARK – An alternative to Automatic 1111, SHARK works natively with NVIDIA and AMD GPUs. However, performance with AMD GPUs tends to be higher than Automatic 1111, while performance with NVIDIA is typically lower.
  • Custom – Since Stable Diffusion is available for anyone to utilize directly, some opt to create their own application with the exact features they need. In fact, we are in the process of developing an easy-to-run application focused on benchmarking rather than actual image generation.

Each of these implementations (as well as many others that are available) has unique pros and cons regarding features and usability. However, from a performance and benchmarking standpoint, Automatic 1111 and SHARK are the two we currently recommend using. In addition, depending on what GPUs you plan to test, we advise using both in tandem. Automatic 1111 for NVIDIA GPUs, and SHARK for AMD GPUs.

One thing to be aware of is that since Stable Diffusion is constantly being updated and improved, it is important to note the various versions used for the underlying packages. For Automatic 1111, this is reported at the bottom of the web-ui, and looks something like:

version: v1.3.2  •  python: 3.10.6  •  torch: 2.0.1+cu118  •  xformers: 0.0.17  •  gradio: 3.32.0  •  checkpoint: ad2a33c361

Differences in any of these components can result in performance changes.

Settings & Models

Stable Diffusion can be utilized for several uses (including text-to-image, image-to-image, inpainting, scaling, etc.), and have a host of settings you can adjust. From our testing, the most commonly changed settings (prompt, negative prompt, cfg scale, and seed) don’t impact performance meaningfully. Generating an image of a dog and or a mountain landscape should take the same amount of time. Even the model used tends to only make a minor difference in the generation time.

Stable Diffusion AutoMatic 1111 Sample image generation
All these images take almost the exact same amount of time to generate, even with different prompts and cfg scales.
Image
Open Full Resolution

Others, such as the steps, will change how long it takes to generate the image, but will give the same result in terms of iterations per second. While most people will use between 20 and 50 steps to generate an image, we recommend using a higher step count (such as 200) as that can help with run-to-run consistency.

The resolution will have the biggest impact on performance but also will change how much VRAM is required to generation the image. For benchmarking, we recommend using 512×512 to maximize compatibility across different GPU models.

Beyond the resolution, the sampling method (Euler, DPM, etc.) can make an impact, with specific methods taking roughly twice as long. “Euler” and “Euler a” are the most commonly used methods and tend to give among the best overall performance. Others, like DPM2, take roughly twice as long. From a GPU benchmarking perspective, we recommend sticking to a variation of Euler for consistency.

Recommended Prompts for Benchmarking

With the effectively infinite number of prompts and settings you can use, we wanted to provide a sample set of prompts to facilitate consistent benchmarking.

Text to Image

  • Prompt: “red sports car, (centered), driving, ((angled shot)), full car, wide angle, mountain road”
  • Negative prompt: N/A
  • Steps: 200
  • Cfg Scale: 7
  • Seed: 3936349264
  • Width: 512
  • Heigh: 512
  • Batch count: 1
  • Batch size: 1
  • Sampler: Euler

Image to Image

  • Source Image
  • Prompt: “photo of a minivan, cinematic, realistic, hyper detailed, maximum detail”
  • Negative prompt: N/A
  • Steps: 200
  • DeNoising Strength: 0.8
  • Cfg Scale: 7
  • Seed: 1417252111
  • Width: 512
  • Heigh: 512
  • Batch count: 1
  • Batch size: 1
  • Sampler: Euler

Suggested inpainting and scaling prompts coming soon!

Measuring Performance

Most implementations of Stable Diffusion report a performance metric of “iterations per second” or “it/s”. We recommend using this metric as it is the most commonly used measurement of performance for Stable Diffusion. If your testing method does not allow you to directly record the it/s, you can calculate it by dividing the number of iterations you are running by the number of seconds it took to complete the test.

For example, if you run our recommended text to image prompt for 200 iterations and it takes 15 seconds to generate the image, that is 200/15 or approximately 13.3 it/s.

Automation

Instructions and scripting for automation are coming soon!

Tower Computer Icon in Puget Systems Colors

Looking for a content creation workstation?

We build computers tailor-made for your workflow. 

Configure a System
Talking Head Icon in Puget Systems Colors

Don’t know where to start?
We can help!

Get in touch with one of our technical consultants today.

Talk to an Expert

Related Content

  • Blender 3.6: AMD Ray Tracing Performance
  • Cinebench 2024: Redefining Rendering Performance Evaluation
  • Stable Diffusion Benchmark Testing Methodology
  • Redshift adds AMD GPU support
View All Related Content

Latest Content

  • DaVinci Resolve Studio 18.6 Neural Engine Performance Optimizations
  • Blackmagic Desktop Video Version 12.4 Incompatibility with Windows 11
  • Blender 3.6: AMD Ray Tracing Performance
  • Cinebench 2024: Redefining Rendering Performance Evaluation
View All

Who is Puget Systems?

Puget Systems builds custom PCs tailored for your workflow

Extensive in-house testing
making you more productive and giving you more performance for your dollar

Reliable workstations
with fewer crashes and blue screens means more time working, less time waiting on your computer

Support that understands
your complex workflows and can get you back up and running ASAP

Proven track record
check out our customer testimonials and Reseller Ratings

Get Started

Browse Our Workstations

Fractal Design Define 7 Chassis with Puget Systems Logo

Select your workflow:

Content Creation
Engineering
Scientific Computing
More

Latest Articles

  • DaVinci Resolve Studio 18.6 Neural Engine Performance Optimizations
  • Blackmagic Desktop Video Version 12.4 Incompatibility with Windows 11
  • Blender 3.6: AMD Ray Tracing Performance
  • Cinebench 2024: Redefining Rendering Performance Evaluation
  • Intel Arc A770 and A750 Content Creation Review (Sept. 2023 Update)
View All

Post navigation

 Impact of Hardware-Accelerated GPU Scheduling on Content Creation PerformanceStable Diffusion Performance – NVIDIA GeForce VS AMD Radeon 
Puget Systems Logo
Build Your Own PC Site Map FAQ
facebook instagram linkedin rss twitter youtube

Optimized Solutions

  • Adobe Premiere
  • Adobe Photoshop
  • Solidworks
  • Autodesk AutoCAD
  • Machine Learning

Workstations

  • Content Creation
  • Engineering
  • Scientific PCs
  • More

Support

  • Online Guides
  • Request Support
  • Remote Help

Publications

  • All News
  • Puget Blog
  • HPC Blog
  • Hardware Articles
  • Case Studies

Policies

  • Warranty & Return
  • Terms and Conditions
  • Privacy Policy
  • Delivery Times
  • Accessibility

About Us

  • Testimonials
  • Careers
  • About Us
  • Contact Us

© Copyright 2023 - Puget Systems, All Rights Reserved.