Add performance tests to PR actions

Galen Elias (@galenelias)

It is important to maintain performance parity to std::span (at the very least when Safe Buffers are enabled). In order to track this, we should have performance tests between std::span and gsl::span for the latest standard library implementation across Clang, GCC, and MSVC.

Galen Elias (@galenelias) put together small benchmark comparing the two implementations on various workloads. They have been attached for convenience and provide a good starting point.

span_bench.zip

Carson Radtke (@carsonRadtke)

Hi Carson Radtke (@carsonRadtke)! I'm a student learning about open source and CI/CD. I would love to work on this issue and help integrate these performance tests into the PR actions. Could you assign it to me?

Santiago Ortiz (@SOrtiz1967)

Santiago Ortiz (@SOrtiz1967) Done! Let me know if you need any additional guidance.

Hi, should I keep the attached benchmark code as-is and use it as the baseline for the PR benchmark job? since the benchmark uses C++20 features, should the benchmark workflow target only the latest supported compiler/toolset versions?

Mohammad Abdul Gafoor (@Gafoor2005)

Mohammad Abdul Gafoor (@Gafoor2005) - If possible, we'd like the benchmarks to be compatible with all support compilers and compiler versions. Modifications to the attached benchmark code may be necessary.

FYI, Santiago Ortiz (@SOrtiz1967) claimed this bug a couple weeks ago, but I haven't seen any movement. I will unassign and open this up to anybody.

ci: Add span performance benchmarks (microsoft#1167)

Carson Radtke (@carsonRadtke)

hey Carson Radtke (@carsonRadtke) would you like the github actions bot commenting the benchmark results in the PR ?
like there will be 3 jobs in the workflow, so there will be 3 comments for each (Clang, GCC, MSVC).

below is a specific example of that

Benchmark Results - MSVC on windows-2025 (C++20)

Running C:\Users\Gafoor\Desktop\code\GSL\benchmark\Release\span_bench.exe
Run on (12 X 2496 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 1280 KiB (x6)
  L3 Unified 12288 KiB (x1)
------------------------------------------------------------------------
Benchmark                              Time             CPU   Iterations
------------------------------------------------------------------------
BM_IsSortedCustomStdSpan             479 ns          479 ns      1600000
BM_IsSortedCustomGslSpan            2317 ns         2344 ns       280000
BM_IsSortedStdSpan                   455 ns          446 ns      1120000
BM_IsSortedGslSpan                   431 ns          401 ns      1792000
BM_IsSortedRangesStdSpan             481 ns          471 ns      1659259
BM_IsSortedRangesGslSpan             613 ns          614 ns      1120000
BM_MinElementAlgorithmStdSpan        168 ns          160 ns      4977778
BM_MinElementAlgorithmGslSpan        152 ns          153 ns      4480000
BM_MinElementRangeForStdSpan        92.8 ns         90.7 ns      8960000
BM_MinElementRangeForGslSpan        70.8 ns         67.0 ns     11200000

these benchmark results will be saved in workflow artifacts too.

Mohammad Abdul Gafoor (@Gafoor2005)

Mohammad Abdul Gafoor (@Gafoor2005) - I think this just adds noise, I would only expect a comment or something if we spotted a regression. Also, I think we have more than 3 jobs, right? There should be three dimensions: one for each compiler, one for each compiler version, one for each C++ standard version. I think we would have ~40 jobs? Basically the same matrix as the compiler actions, but we only want to benchmark release builds.

Carson Radtke (@carsonRadtke)

Carson Radtke (@carsonRadtke) I want to clear something up. the benchmark currently outputs the (Time, CPU, Iterations) as seen in my previous comment. so this output might vary for each run, like different Time, CPU, Iterations.

we are not trying to calculate and catch a performance regression by comparing the numbers for each benchmark to it's previous runs. Instead we are here trying to see if our gsl span is as good as std span, and there is no performance difference between these two groups on each run.

The benchmark results on MSVC configuration has no major time difference between gsl span and std span, except one test (is sorted custom), which can be seen in my previous comment. And in the case of gcc configured run there is a visible time difference between the std and gsl.

So, the numbers change with compiler configurations and runs. even though we can use the gsl / std ratios to measure the parity.
using these ratios we can also keep track of previous run ratio and current run ratio. and can compare them to track any further regression on each configuration in a matrix.

as this benchmarks had to ran on each compiler, compiler version etc.. this std span at least needs C++20 or above

also if we do this, there might be a comment for each config of the matrix as shown below, not exactly some thing similar.

Carson Radtke (@carsonRadtke) i want to know your thoughts on this. any suggestion/advice?

we are not trying to calculate and catch a performance regression by comparing the numbers for each benchmark to it's previous runs. Instead we are here trying to see if our gsl span is as good as std span, and there is no performance difference between these two groups on each run.

Agreed.

The benchmark results on MSVC configuration has no major time difference between gsl span and std span, except one test (is sorted custom), which can be seen in my previous comment. And in the case of gcc configured run there is a visible time difference between the std and gsl.

When we find a large delta between std:: and gsl::, I'd like that to fail a pipeline. For this one existing performance difference, I'm happy to suppress it and open an issue so we can get these benchmarks in.

this std span at least needs C++20 or above

Right. So, then there wouldn't be ~40 tests, but I'd like to see all compilers with C++20, 23, and 26 (when supported). Also when moving to C++23, ranges tests may be interesting to include (but that can be a separate PR).

also if we do this, there might be a comment for each config of the matrix as shown below, not exactly some thing similar.

A little noisy, but I think it is fine as long as it is pointing out a real problem.

Carson Radtke (@carsonRadtke)

Hi Carson Radtke (@carsonRadtke) ,

I’ve opened a test PR in my fork to simulate the output of the workflow.
Please have a look and let me know if any changes are needed. If everything looks good, I’ll open a PR in the main repository.

Thank you!

github.com/Gafoor2005/GSL/pull/1

Cool! We tend to review code only as PRs into the microsoft/GSL repo. Please create a PR from your fork to this repo and I'll take a look.

Carson Radtke (@carsonRadtke)

Hi Carson Radtke (@carsonRadtke) ,
I'd like to help with this.Is the work in Gafoor2005's fork the base I should build on, or should I start a new branch from main?

I ran Galen's benchmark locally with 5 repetitions, with Ubuntu 24.04, g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, C++20, Release build, with my CPU around 1093.44 MHz, and got these results:

Workload	std::span (ns)	gsl::span (ns)	GSL / std
`std::is_sorted`	215	636	2.96x
`std::ranges::is_sorted`	215	622	2.89x
Custom `is_sorted`	215	636	2.96x
`std::min_element`	428	742	1.73x
Range-for min	155	427	2.75x

I'd like to help with this.Is the work in Gafoor2005's fork the base I should build on, or should I start a new branch from main?

The changes need to apply smoothly to main, so either start a new branch from main or rebase Gafoor2005's fork onto main. I would definitely incorporate Galen's benchmarks either way.

I ran Galen's benchmark locally with 5 repetitions, with Ubuntu 24.04, g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, C++20, Release build, with my CPU around 1093.44 MHz, and got these results:

It also may be worth understanding why we seem to have a perf regression in gsl::span compared to std::span. I don't think that was the case before and perhaps the culprit is one of my recent changes to the span implementation. Maybe #1263 or #1228?

Anyway, another evolution that we may want to benchmark as part of this is comparing gsl::dyn_array to std::vector. They don't have identical semantics, but if the vector never grows we should observe similar performance in dyn_array.

Carson Radtke (@carsonRadtke)

Hi Carson Radtke (@carsonRadtke) ,

I think I will start a new branch from main, and include Galen’s benchmarks. I'll take some reference from Gafoor's work(with credits of-course) since it does not seem fair to just copy his code.

It also may be worth understanding why we seem to have a perf regression in gsl::span compared to std::span. I don't think that was the case before and perhaps the culprit is one of my recent changes to the span implementation. Maybe #1263 or #1228?

I think I will see if the perf regression was just on my system or specific to my compiler. Nevertheless, I will definitely investigate that issue.

Anyway, another evolution that we may want to benchmark as part of this is comparing gsl::dyn_array to std::vector. They don't have identical semantics, but if the vector never grows we should observe similar performance in dyn_array.

Yes, I'll incorporate that benchmark as well.

Hi, Carson Radtke (@carsonRadtke) ,

I was trying to incorporate the gsl::dyn_array and std::vector benchmark in Galen's span_bench.cpp program given. But I have noticed that the version which the CMake pins over there(v.4.1.0) does not seem to have a gsl::dyn_array?

The include/gsl/ folder at that tag has no dyn_array header, while main does.

A test build on v.4.1.0 failed with error error: ‘dyn_array’ in namespace ‘gsl’ does not name a template type

Should I target main or another release version? The oldest release in which gsl::dyn_array appears is 5.0.0

The benchmarks need to target main and should run on every PR.

Carson Radtke (@carsonRadtke)

Hi Carson Radtke (@carsonRadtke) ,

I have modified the span_bench.cpp file to include benchmarks for std::span and gsl::dyn_array and I have obtained these results on my system with 5 repetitions, on Ubuntu 24.04, g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, C++20, Release build, with my CPU around 3332.78 MHz :

Span benchmarks (mean CPU times)

Workload	std::span (ns)	gsl::span (ns)	GSL / std
`std::is_sorted`	401	401	1.00
`std::ranges::is_sorted`	204	303	1.49
Custom `is_sorted`	203	203	1.00
`std::min_element`	400	587	1.47
Range-for min	146	146	1.00

Please note that for the std::min_element tests with gsl::span, I got something like, 600 ns ,599 ms, 598 ns, 595 ns, 543 ns. Here the 543 ns test seems like noise, so I think I should include the fact that the median was 598ns.

Also, in the previous test where my CPU was on around 1093.44 MHz, I had obtained quite different results.

dyn_array vs vector benchmarks (mean CPU time)

Workload	std::vector (ns)	gsl::dyn_array (ns)	dyn_array / vector
`std::is_sorted`	203	203	1.00
`std::ranges::is_sorted`	204	206	1.01
Custom `is_sorted`	203	400	1.97
`std::min_element`	400	203	0.51
Range-for min	191	188	0.98

I had a few clarifications I wanted to ask:

Where should I actually put the span_benchmark.cpp file?
I'd like to propose check that fails the job when the gsl/std ratio goes above a threshold. Do you have a threshold in mind? and should the known std::ranges::is_sorted and std::min_element gaps be suppressed with a tracking issue?

Where should I actually put the span_benchmark.cpp file?

A new directory benchmarks/ would probably be good.

I'd like to propose check that fails the job when the gsl/std ratio goes above a threshold. Do you have a threshold in mind? and should the known std::ranges::is_sorted and std::min_element gaps be suppressed with a tracking issue?

Let's start with 1.25x and a suppression w/ tracking issues sounds good for now.

The choice of compiler might also affect results. How does the comparison look when using MSVC?

Carson Radtke (@carsonRadtke)

Hi Carson Radtke (@carsonRadtke),
Thanks for the threshold and suppression guidance.

These are the MSVC results from Windows 11, using MSVC 19.51.36260 for x86 (32-bit), CPU at about 3793 MHz, Release, C++20, 5 repetitions.

MSVC span benchmarks (mean CPU time)

Workload	std::span (ns)	gsl::span (ns)	GSL / std
`std::is_sorted`	123	129	1.05
`std::ranges::is_sorted`	122	127	1.04
Custom `is_sorted`	439	661	1.51
`std::min_element`	98	95	0.96
Range-for min	71	71	1.01

MSVC dyn_array vs vector benchmarks (mean CPU time)

Workload	std::vector (ns)	gsl::dyn_array (ns)	dyn_array / vector
`std::is_sorted`	124	122	0.98
`std::ranges::is_sorted`	122	129	1.06
Custom `is_sorted`	455	1278	2.81
`std::min_element`	92	104	1.13
Range-for min	168	1369	8.15

On MSVC x86, the range-for min on gsl::dyn_array runs about 8 times slower than on std::vector

Navigation Menu

Repository navigation

Add performance tests to PR actions #1167

Benchmark Results - MSVC on windows-2025 (C++20)

Span benchmarks (mean CPU times)

dyn_array vs vector benchmarks (mean CPU time)

MSVC span benchmarks (mean CPU time)

MSVC dyn_array vs vector benchmarks (mean CPU time)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add performance tests to PR actions #1167

Description

Activity

SOrtiz1967 commented on May 3, 2026

carsonRadtke commented on May 4, 2026

Gafoor2005 commented on May 15, 2026

carsonRadtke commented on May 18, 2026

Gafoor2005 commented on May 19, 2026

Benchmark Results - MSVC on windows-2025 (C++20)

carsonRadtke commented on May 19, 2026

Gafoor2005 commented on May 22, 2026

carsonRadtke commented on May 26, 2026

Gafoor2005 commented on May 29, 2026

carsonRadtke commented on May 29, 2026

AtomOnTop commented on Oct 26, 2026

carsonRadtke commented on Oct 26, 2026

AtomOnTop commented on Oct 27, 2026

AtomOnTop commented on Oct 27, 2026

carsonRadtke commented on Oct 27, 2026

AtomOnTop commented on Oct 28, 2026

Span benchmarks (mean CPU times)

dyn_array vs vector benchmarks (mean CPU time)

carsonRadtke commented on Oct 29, 2026

AtomOnTop commented on Oct 30, 2026

MSVC span benchmarks (mean CPU time)

MSVC dyn_array vs vector benchmarks (mean CPU time)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions