From a20f9c89414a53402fdcd3a8fc61103cb466e02c Mon Sep 17 00:00:00 2001 From: eamihai Date: Tue, 23 Jun 2026 16:25:54 +0300 Subject: [PATCH 1/7] Setup README --- Benchx.svg | 4 ++++ README.md | 7 +++++++ 2 files changed, 11 insertions(+) create mode 100644 Benchx.svg create mode 100644 README.md diff --git a/Benchx.svg b/Benchx.svg new file mode 100644 index 0000000..ebeabcf --- /dev/null +++ b/Benchx.svg @@ -0,0 +1,4 @@ + + + +
Geo-distributed Evaluation Environment
Geo-distributed Evaluation Environment
 US West
 US West
Cross-regional
latency & jitter
Cross-regional...
config.yaml
config.yaml
Server (Partition 1)
Server (Partition 1)
Server (Partition 2)
Server (Partition 2)
Client
Client
     Workload Generator
     Workload Generator
 EU West 
 EU West 
Server (Partition 1)
Server (Partition 1)
Server (Partition 2)
Server (Partition 2)
Client
Client
     Workload Generator
     Workload Generator
Metrics
Metrics \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..681391a --- /dev/null +++ b/README.md @@ -0,0 +1,7 @@ +# Benchx + +Benchx is a configurable workload generator for benchmarking geo-distributed databases. + +![Benchx Design](Benchx.svg) + +## Running Benchx From 975a05953ddbcb9110572534f71754d85357bcb3 Mon Sep 17 00:00:00 2001 From: eamihai Date: Sat, 4 Jul 2026 22:45:19 +0100 Subject: [PATCH 2/7] Add new image --- Benchx.svg | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Benchx.svg b/Benchx.svg index ebeabcf..8647716 100644 --- a/Benchx.svg +++ b/Benchx.svg @@ -1,4 +1,4 @@ -
Geo-distributed Evaluation Environment
Geo-distributed Evaluation Environment
 US West
 US West
Cross-regional
latency & jitter
Cross-regional...
config.yaml
config.yaml
Server (Partition 1)
Server (Partition 1)
Server (Partition 2)
Server (Partition 2)
Client
Client
     Workload Generator
     Workload Generator
 EU West 
 EU West 
Server (Partition 1)
Server (Partition 1)
Server (Partition 2)
Server (Partition 2)
Client
Client
     Workload Generator
     Workload Generator
Metrics
Metrics \ No newline at end of file +
Geo-distributed Evaluation Environment
Geo-distributed Evaluation Environment
 US West
 US West
Cross-regional
latency & jitter
Cross-regional...
config.yaml
config.yaml
Server (Partition 1)
Server (Partition 1)
Server (Partition 2)
Server (Partition 2)
Client
Client
     Workload Generator
     Workload Generator
 EU West 
 EU West 
Server (Partition 1)
Server (Partition 1)
Server (Partition 2)
Server (Partition 2)
Client
Client
     Workload Generator
     Workload Generator
Metrics
Metrics \ No newline at end of file diff --git a/README.md b/README.md index 681391a..6bff010 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,6 @@ Benchx is a configurable workload generator for benchmarking geo-distributed databases. -![Benchx Design](Benchx.svg) +

## Running Benchx From 8ba2f7fde502709823c39296e45316fc348c514e Mon Sep 17 00:00:00 2001 From: eamihai Date: Sun, 5 Jul 2026 23:52:46 +0100 Subject: [PATCH 3/7] Add example --- Benchx.svg => BenchX.svg | 0 Example.svg | 1051 ++++++++++++++++++++++++++++++++++++++ README.md | 29 +- 3 files changed, 1076 insertions(+), 4 deletions(-) rename Benchx.svg => BenchX.svg (100%) create mode 100644 Example.svg diff --git a/Benchx.svg b/BenchX.svg similarity index 100% rename from Benchx.svg rename to BenchX.svg diff --git a/Example.svg b/Example.svg new file mode 100644 index 0000000..d49e672 --- /dev/null +++ b/Example.svg @@ -0,0 +1,1051 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + +10 +0 + + + + + +10 +1 + + + + + +10 +2 + + + + + +10 +3 + + + + + +10 +4 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Latency (ms) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Detock + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +SLOG + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Calvin + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Janus + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +no data +CRDB + + + + + +Short Read & Write + +Short Read-Only + +Large Read-Only + +Insert-Only + +Delete-Only + diff --git a/README.md b/README.md index 6bff010..6e0161b 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,28 @@ -# Benchx +# BenchX -Benchx is a configurable workload generator for benchmarking geo-distributed databases. +BenchX is a configurable OLTP workload generator for benchmarking geo-distributed databases. It is built on top of the TPC-C schema, and introduces a wide range of knobs to control the transaction mix, operation intensity, geo-distribution profile (i.e., percentage of multi-home, single-home, and foreign single-home), dependent transactions, data skew, and temporal dynamics. -

+Thanks to its configurability, BenchX covers a significant number of commonly used OLTP workload generators such as TPC-C, YCSB+T, PPS, SmallBank, MovR, DS Movie, and DS Hotels. Namely, with the right configuration, it replicates the effect of each of them. In addition, it reaches regions of the workload design space that none of them cover. -## Running Benchx +## Example of Analysis + +For example, we can use BenchX to analyze the effect of varying both the geo-distribution profile and the operation intensity on the throughput and latency of several databases, including Detock, SLOG, Calvin, Janus, and CockroachDB. + +

+ +## Running BenchX + +BenchX runs on top of the Gaia geo-distributed evaluation framework. BenchX is configured through a single YAML file that is picked up by the workload generator in each region. +```yaml +{ + transaction_mix: "50:25:10:10:5", + intensity: "10:5:10000:2:1", + mp_percent: "50:50:100:25:50", + fsh_percent: "33:33:0:20:50", + mh_percent: "33:0:20:0:50", + dependent_percent: "50:50:0:0:0", + skew: "0.5:0.0:0.9:0.0:0.999", + temporal_dynamics: "10:90,50:50,90:10" +} +``` +

\ No newline at end of file From 29c6ae40b684f0defb2128e540a647b05acc769d Mon Sep 17 00:00:00 2001 From: eamihai Date: Sun, 5 Jul 2026 23:59:34 +0100 Subject: [PATCH 4/7] Change example --- Example.svg | 1408 +++++++++++++++++++++++++-------------------------- 1 file changed, 704 insertions(+), 704 deletions(-) diff --git a/Example.svg b/Example.svg index d49e672..6e25833 100644 --- a/Example.svg +++ b/Example.svg @@ -1,1051 +1,1051 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - -0 + + - + - - -25 + + - + - - -50 + + - + - - -75 + + - + - - -100 -Geo-distribution (%) + + - + - - -10 -0 + + +0k - + - - -10 -1 + + +5k - + - - -10 -2 + + +10k +Throughput +(txns/s) - + - - -10 -3 - - - - -10 -4 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Latency (ms) + + - + - - + + + + + + +Throughput (Detock) + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - - - - - - -Detock - + + + + - + - - -0 - + - - -25 - + - - -50 - - - - -75 + + + + + + +Throughput (SLOG) + - + - - -100 -Geo-distribution (%) + + - + - - + + - + - - + + - + - - + + - + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - + + + - + + + - + - + - + - - + + + + + + +Throughput (Calvin) + - + + + - + + + - + + + - - - - - - -SLOG - + + + + - + - - -0 + + - + - - -25 + + - + - - -50 + + - + - - -75 + + - + - - -100 -Geo-distribution (%) - + - - - + - - - - - - + + + + + + +Throughput (Janus) + - + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - + + + - + + + - + + + - + + + - + + + - - + + + + + + + + +no data +Throughput (CRDB) + - + + + +0 - + + + +25 - + + + +50 - - - - - - -Calvin - + + + + +75 - + - - -0 + + +100 +Geo-distribution (%) - + - - -25 + + +10 +0 - + - - -50 + + +10 +2 - - - - -75 + + + + +10 +4 +Latency +(ms) - + - - -100 -Geo-distribution (%) - + - - - + - - - + - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - + - - + + + + + + +Latency (Detock) + - + + + +0 - + + + +25 - + + + +50 - + + + +75 - + + + +100 +Geo-distribution (%) - + + + - + + + - - - - - - -Janus - + + + + - + - - -0 - + - - -25 - + - - -50 - + - - -75 - + - - -100 -Geo-distribution (%) - + - - - + - - - - - - + + + + + + +Latency (SLOG) + - + - - + + +0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -no data -CRDB + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Latency (Calvin) + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Latency (Janus) + + + + + + +0 + + + + + +25 + + + + + +50 + + + + + +75 + + + + + +100 +Geo-distribution (%) + + + + + + + + + + + + + + + + + + + +no data +Latency (CRDB) - - - - -Short Read & Write - -Short Read-Only - -Large Read-Only - -Insert-Only - -Delete-Only + + + +Intensity: + +1 + +2 + +5 + +20 From 5296f00d74a8c82b551d2464485c1385aec01896 Mon Sep 17 00:00:00 2001 From: eamihai Date: Mon, 6 Jul 2026 00:02:01 +0100 Subject: [PATCH 5/7] Refactor --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 6e0161b..2bb1851 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,7 @@ BenchX is a configurable OLTP workload generator for benchmarking geo-distribute Thanks to its configurability, BenchX covers a significant number of commonly used OLTP workload generators such as TPC-C, YCSB+T, PPS, SmallBank, MovR, DS Movie, and DS Hotels. Namely, with the right configuration, it replicates the effect of each of them. In addition, it reaches regions of the workload design space that none of them cover. -## Example of Analysis - -For example, we can use BenchX to analyze the effect of varying both the geo-distribution profile and the operation intensity on the throughput and latency of several databases, including Detock, SLOG, Calvin, Janus, and CockroachDB. - -

- -## Running BenchX +## High-level Design BenchX runs on top of the Gaia geo-distributed evaluation framework. BenchX is configured through a single YAML file that is picked up by the workload generator in each region. ```yaml @@ -25,4 +19,10 @@ BenchX runs on top of the Gaia geo-distributed evaluation framework. BenchX is c temporal_dynamics: "10:90,50:50,90:10" } ``` -

\ No newline at end of file +

+ +## Example of Analysis + +For example, we can use BenchX to analyze the effect of varying both the geo-distribution profile and the operation intensity on the throughput and latency of several databases, including Detock, SLOG, Calvin, Janus, and CockroachDB. + +

\ No newline at end of file From 2840908c8d4e7da41ce0dbee36a0f89cee33684c Mon Sep 17 00:00:00 2001 From: eamihai Date: Mon, 6 Jul 2026 00:02:45 +0100 Subject: [PATCH 6/7] Refactor --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2bb1851..d37eeda 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Thanks to its configurability, BenchX covers a significant number of commonly us ## High-level Design -BenchX runs on top of the Gaia geo-distributed evaluation framework. BenchX is configured through a single YAML file that is picked up by the workload generator in each region. +BenchX is configured through a single YAML file that is picked up by the workload generator in each region. ```yaml { transaction_mix: "50:25:10:10:5", From 3b8851cf33a316b0fa39c2148751c289ce73a6b3 Mon Sep 17 00:00:00 2001 From: eamihai Date: Mon, 6 Jul 2026 00:17:58 +0100 Subject: [PATCH 7/7] Rephrase --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d37eeda..031b97a 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,12 @@ BenchX is a configurable OLTP workload generator for benchmarking geo-distributed databases. It is built on top of the TPC-C schema, and introduces a wide range of knobs to control the transaction mix, operation intensity, geo-distribution profile (i.e., percentage of multi-home, single-home, and foreign single-home), dependent transactions, data skew, and temporal dynamics. -Thanks to its configurability, BenchX covers a significant number of commonly used OLTP workload generators such as TPC-C, YCSB+T, PPS, SmallBank, MovR, DS Movie, and DS Hotels. Namely, with the right configuration, it replicates the effect of each of them. In addition, it reaches regions of the workload design space that none of them cover. +Thanks to its configurability, BenchX covers a significant number of commonly used OLTP workload generators including TPC-C, YCSB+T, PPS, SmallBank, MovR, DS Movie, and DS Hotels. Namely, with the right configuration, it replicates the effect of each of them. In addition, it reaches regions of the workload design space that none of them cover. ## High-level Design -BenchX is configured through a single YAML file that is picked up by the workload generator in each region. +The user provides a single YAML configuration file describing the desired workload. The workload generator in each region picks up this configuration and feeds the corresponding transactions into the database servers of its region. During the run, BenchX collects performance metrics and reports them back to the user. + ```yaml { transaction_mix: "50:25:10:10:5", @@ -23,6 +24,6 @@ BenchX is configured through a single YAML file that is picked up by the workloa ## Example of Analysis -For example, we can use BenchX to analyze the effect of varying both the geo-distribution profile and the operation intensity on the throughput and latency of several databases, including Detock, SLOG, Calvin, Janus, and CockroachDB. +Because every dimension of BenchX is independently tunable, we can sweep entire regions of the workload space in a single experiment. For example, varying both the geo-distribution profile and the operation intensity reveals how the throughput and latency of Detock, SLOG, Calvin, Janus, and CockroachDB evolve across these dimensions.

\ No newline at end of file