Add Manager Stress Testing framework

README.md

@@ -57,6 +57,9 @@ Tests are run using the following scripts in `bin/`:
   * `agitator` - Runs agitator
   * `gcs` - Runs garbage collection simulation
   * `monitor` - Runs availability monitor probe
+  * `manager-stress/managerstress.sh` - Stresses Manager and Compaction Coordinator operations
+  * `manager-stress/stresscompactor.sh` - Starts in-JVM Compactor simulators in a separate JVM
+  * `manager-stress/configure-mgr-stress-test.sh` - Prompts for settings and prints both commands
 
 Run the scripts without arguments to view usage.
 
@@ -176,6 +179,121 @@ verification test.
 
 See [gcs.md](docs/gcs.md).
 
+## Manager and Compaction Coordinator stress test
+
+The `managerstress` command starts local client JVMs that issue weighted create-table, delete-table,
+split, merge, tablet-availability, compaction, and bulk-import tasks against an existing Accumulo
+instance. It creates the requested number of uniquely named tables before starting workers and
+replenishes the table pool after deletes. A selected create task creates a new table generation
+before deleting the old generation so the pool returns to its configured size. Weights apply to
+selected workload tasks; the follow-up table creations after deletes are unweighted pool maintenance.
+
+All created table generations start with `TabletAvailability.UNHOSTED` and 10–999 evenly spaced
+initial splits, leaving at most 1,000 tablets. Split tasks add a random number of unique split rows
+without exceeding that tablet limit. Merge tasks sample a target from 1–100, clamp it to a valid
+2–100 tablet range, and use the table's actual split boundaries for the merge. Split rows and the
+bulk-import seed key use the same fixed-width hexadecimal row-key scheme.
+
+Availability tasks sample a random number of unique tablets from one through the table's full
+tablet count. Each selected tablet is assigned a random availability state different from its
+current state. Nonadjacent selections remain separate; adjacent selections with the same target
+state are grouped into one row-range request.
+
+Bulk import requires a shared HDFS directory accessible to all local workers. Each worker creates
+one one-entry RFile there at startup and stages a copy of it for every bulk import. Compaction tasks
+submit non-blocking major compactions. The coordinator removes the run-specific HDFS staging
+directory during shutdown, including any files left by workers that did not exit cleanly.
+
+`managerstress` temporarily configures the `mgrstress` compaction service in the system
+configuration. It uses `org.apache.accumulo.core.spi.compaction.RatioBasedCompactionPlanner`, with
+`compaction.service.mgrstress.planner.opts.groups` set to the group specified by
+the required `--compactor-resource-group` option. The stress namespace is created with
+`table.compaction.dispatcher.opts.service=mgrstress`. Previous ZooKeeper system-property overrides
+are restored when the test finishes; pre-existing values do not prevent startup.
+
+The required `--namespace` must name a namespace that does not already exist. `managerstress` creates
+it before creating tables, places all stress tables in it, and removes its tables and namespace when
+the run finishes. The configured Accumulo user needs permission to create and drop namespaces and
+tables, alter tables and namespaces, modify system configuration, compact, and bulk import. It also
+needs access to the configured HDFS directory.
+
+For example, the command below runs four client processes for 30 minutes, maintaining eight tables.
+Operation weights are relative; setting a weight to zero disables that operation.
+
+```bash
+./bin/manager-stress/managerstress.sh \
+  --namespace mgrstress_example \
+  --compactor-resource-group default \
+  --duration 30m \
+  --clients 4 \
+  --tables 8 \
+  --hdfs-dir hdfs://namenode:8020/tmp/accumulo-managerstress \
+  --create-weight 1 \
+  --delete-weight 1 \
+  --split-weight 2 \
+  --merge-weight 2 \
+  --availability-weight 1 \
+  --compact-weight 4 \
+  --bulk-import-weight 1
+```
+
+Use `--table-prefix` to choose a prefix for the generated tables; if omitted, a unique prefix is
+created for the run. Choose a namespace name that is unused on the Accumulo instance; the test fails
+if it already exists. The test removes all tables in its namespace and deletes the namespace after
+workers stop. Per-worker logs include operation attempts, completions, skips, races, errors, and
+average latency.
+
+When workers stop, `managerstress` logs totals aggregated from each worker's atomic metrics snapshot.
+For each operation, `submitted` counts workload operations that reached their Manager API call,
+`completed` counts successful synchronous operations, and `asyncAccepted` counts compaction and
+tablet-availability requests accepted by Accumulo. Availability and compaction requests do not wait
+for tablets to reach their target state. `failed` counts operation errors; `outcomeUnknown` is the
+subset with a transport failure or interruption after submission. Snapshot files are refreshed
+periodically in the run's temporary control directory and removed after the summary is logged.
+
+### Compactor simulator
+
+`stresscompactor` is designed to be used with the `managerstress` test framework. Run it alongside
+`managerstress` to provide Compactor simulators for external compaction requests generated by the
+stress tables. Each invocation starts the configured number of independently registered Compactor
+simulators as threads in that JVM. Start additional JVMs by invoking the command again (or on other
+hosts); set `JAVA_OPTS` separately for each process to tune its heap and garbage collector.
+
+Use the same group for `managerstress --compactor-resource-group` and
+`stresscompactor --resource-group`. Run `./bin/manager-stress/configure-mgr-stress-test.sh` to
+interactively enter settings for both processes and print the two commands. Press Enter to accept
+the displayed defaults. The client config prompt defaults to `../../conf/accumulo-client.properties`
+relative to the helper script; credentials should be stored in that file. The helper prints commands
+to stdout without starting the processes.
+
+The Manager must be able to reach the advertised host and callback ports. Configure the resource
+group to match the Compactor resource group used by the external compaction service for the stress
+tables. The configured Accumulo user must be permitted to perform system actions. When using SASL,
+the advertised host should be the host's canonical name.
+
+```bash
+JAVA_OPTS="-Xms2g -Xmx2g -XX:+UseG1GC" ./bin/manager-stress/stresscompactor.sh \
+  --compactors-per-jvm 16 \
+  --duration 30m \
+  --resource-group default \
+  --host compactor-client01.example.net \
+  --port-range 9600-9699 \
+  --success-weight 1 \
+  --failure-weight 1 \
+  --cancellation-weight 1
+```
+
+Each simulator applies `--resource-group` to its own `compactor.group` configuration property,
+registers its Compactor address, and serves the Manager's status, wake, and cancel RPCs. On receiving
+a job it immediately reports a result according to the configured relative weights. A successful
+result reports zero output entries, which commits as a zero-output compaction and removes the
+input-file references from the stress table. Simulators unregister and close their connections when
+the JVM's duration expires or it is stopped.
+
+Each simulator logs its job outcomes on shutdown, followed by a per-JVM aggregate. These counts are
+external compaction jobs, which may be multiple for one table-level compaction request; they are
+reported separately from `managerstress`'s accepted asynchronous request count.
+
 ## Agitator
 
 The agitator will periodically kill the Accumulo manager, tablet server, and Hadoop data node

bin/manager-stress/configure-mgr-stress-test.sh

@@ -0,0 +1,215 @@
+#! /usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+set -euo pipefail
+
+script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+default_client_config="${script_dir}/../../conf/accumulo-client.properties"
+default_namespace="mgrstress_$(date +%Y%m%d_%H%M%S)_${RANDOM}"
+default_host=$(hostname -f 2>/dev/null || hostname)
+
+prompt_default() {
+  local label=$1
+  local default=$2
+  local answer
+  printf '%s [%s]: ' "$label" "$default" >&2
+  IFS= read -r answer
+  REPLY=${answer:-$default}
+}
+
+prompt_optional() {
+  local label=$1
+  local default_description=$2
+  printf '%s [%s]: ' "$label" "$default_description" >&2
+  IFS= read -r REPLY
+}
+
+shell_quote() {
+  local value=$1
+  value=${value//\'/\'\\\'\'}
+  printf "'%s'" "$value"
+}
+
+printf '%s\n' \
+  'Enter the ManagerStress settings. Press Enter to accept the displayed defaults.' \
+  'Client passwords should be stored in the selected client properties file.' >&2
+
+prompt_default 'ManagerStress duration' '5m'
+manager_duration=$REPLY
+prompt_default 'ManagerStress client JVMs' '2'
+manager_clients=$REPLY
+prompt_default 'ManagerStress table count' '4'
+manager_tables=$REPLY
+prompt_default 'Namespace (generated names must not already exist)' "$default_namespace"
+manager_namespace=$REPLY
+prompt_default 'Compactor resource group (shared with StressCompactor)' 'default'
+compactor_resource_group=$REPLY
+prompt_optional 'Table prefix (blank generates one)' 'generated automatically'
+manager_table_prefix=$REPLY
+prompt_default 'Shared HDFS directory for bulk imports' 'hdfs:///tmp/accumulo-managerstress'
+manager_hdfs_dir=$REPLY
+prompt_default 'Create-table weight' '1'
+manager_create_weight=$REPLY
+prompt_default 'Delete-table weight' '1'
+manager_delete_weight=$REPLY
+prompt_default 'Split weight' '1'
+manager_split_weight=$REPLY
+prompt_default 'Merge weight' '1'
+manager_merge_weight=$REPLY
+prompt_default 'Tablet-availability weight' '1'
+manager_availability_weight=$REPLY
+prompt_default 'Compaction weight' '1'
+manager_compact_weight=$REPLY
+prompt_default 'Bulk-import weight' '1'
+manager_bulk_import_weight=$REPLY
+prompt_optional 'ManagerStress random seed (blank uses a random seed)' 'random per run'
+manager_seed=$REPLY
+prompt_optional 'ManagerStress JAVA_OPTS (blank uses JVM defaults)' 'empty'
+manager_java_opts=$REPLY
+
+printf '\n%s\n' 'Enter the StressCompactor settings.' >&2
+prompt_default 'StressCompactor JVM duration' '5m'
+compactor_duration=$REPLY
+prompt_default 'Compactors per JVM' '1'
+compactors_per_jvm=$REPLY
+prompt_default 'Compactor callback host' "$default_host"
+compactor_host=$REPLY
+prompt_default 'Compactor callback port range' '0'
+compactor_port_range=$REPLY
+prompt_default 'Compaction success weight' '1'
+compactor_success_weight=$REPLY
+prompt_default 'Compaction failure weight' '1'
+compactor_failure_weight=$REPLY
+prompt_default 'Compaction cancellation weight' '1'
+compactor_cancellation_weight=$REPLY
+prompt_optional 'StressCompactor random seed (blank uses a random seed)' 'random per run'
+compactor_seed=$REPLY
+prompt_optional 'StressCompactor JAVA_OPTS (blank uses JVM defaults)' 'empty'
+compactor_java_opts=$REPLY
+
+printf '\n%s\n' 'Enter shared Accumulo client settings.' >&2
+prompt_default 'Accumulo client config file' "$default_client_config"
+client_config=$REPLY
+prompt_optional 'Accumulo user override (blank uses client config)' 'from client config'
+client_user=$REPLY
+prompt_optional 'Authorizations (blank uses none)' 'empty'
+client_auths=$REPLY
+prompt_default 'Enable distributed tracing? (yes/no)' 'no'
+trace_choice=${REPLY,,}
+case "$trace_choice" in
+  yes | y) enable_trace=true ;;
+  no | n) enable_trace=false ;;
+  *)
+    printf 'Please answer yes or no.\n' >&2
+    exit 2
+    ;;
+esac
+
+client_args=(--config-file "$client_config")
+if [[ -n "$client_user" ]]; then
+  client_args+=(--user "$client_user")
+fi
+if [[ -n "$client_auths" ]]; then
+  client_args+=(--auths "$client_auths")
+fi
+if [[ "$enable_trace" == true ]]; then
+  client_args+=(--trace)
+fi
+while true; do
+  prompt_optional 'Additional client property override (key=value; blank to finish)' 'none'
+  if [[ -z "$REPLY" ]]; then
+    break
+  fi
+  client_args+=(-o "$REPLY")
+done
+
+manager_args=(
+  --namespace "$manager_namespace"
+  --compactor-resource-group "$compactor_resource_group"
+  --duration "$manager_duration"
+  --clients "$manager_clients"
+  --tables "$manager_tables"
+  --hdfs-dir "$manager_hdfs_dir"
+  --create-weight "$manager_create_weight"
+  --delete-weight "$manager_delete_weight"
+  --split-weight "$manager_split_weight"
+  --merge-weight "$manager_merge_weight"
+  --availability-weight "$manager_availability_weight"
+  --compact-weight "$manager_compact_weight"
+  --bulk-import-weight "$manager_bulk_import_weight"
+)
+if [[ -n "$manager_table_prefix" ]]; then
+  manager_args+=(--table-prefix "$manager_table_prefix")
+fi
+if [[ -n "$manager_seed" ]]; then
+  manager_args+=(--seed "$manager_seed")
+fi
+manager_args+=("${client_args[@]}")
+
+compactor_args=(
+  --resource-group "$compactor_resource_group"
+  --duration "$compactor_duration"
+  --compactors-per-jvm "$compactors_per_jvm"
+  --host "$compactor_host"
+  --port-range "$compactor_port_range"
+  --success-weight "$compactor_success_weight"
+  --failure-weight "$compactor_failure_weight"
+  --cancellation-weight "$compactor_cancellation_weight"
+)
+if [[ -n "$compactor_seed" ]]; then
+  compactor_args+=(--seed "$compactor_seed")
+fi
+compactor_args+=("${client_args[@]}")
+
+print_command() {
+  local label=$1
+  local java_opts=$2
+  local script=$3
+  local argument
+  local index=0
+  shift 3
+  local -a arguments=("$@")
+
+  printf '# %s\n' "$label"
+  if [[ -n "$java_opts" ]]; then
+    printf 'JAVA_OPTS=%s ' "$(shell_quote "$java_opts")"
+  fi
+  printf '%q' "$script"
+  while ((index < ${#arguments[@]})); do
+    argument=${arguments[index]}
+    printf ' \\\n  %q' "$argument"
+    if [[ "$argument" == --trace ]]; then
+      index=$((index + 1))
+      continue
+    fi
+    if ((index + 1 >= ${#arguments[@]})); then
+      printf 'Option %s is missing its value.\n' "$argument" >&2
+      return 2
+    fi
+    printf ' %q' "${arguments[index + 1]}"
+    index=$((index + 2))
+  done
+  printf '\n\n'
+}
+
+print_command 'ManagerStress' "$manager_java_opts" "${script_dir}/managerstress.sh" \
+  "${manager_args[@]}"
+print_command 'StressCompactor' "$compactor_java_opts" "${script_dir}/stresscompactor.sh" \
+  "${compactor_args[@]}"

bin/manager-stress/managerstress.sh

@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+bin_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+source "${bin_dir}/../build"
+
+export CLASSPATH="$TEST_JAR_PATH:$HADOOP_API_JAR:$HADOOP_RUNTIME_JAR:$CLASSPATH"
+
+client_config_set=false
+for arg in "$@"; do
+  case "$arg" in
+    -c | --config-file | --config-file=*)
+      client_config_set=true
+      break
+      ;;
+  esac
+done
+if [[ "$client_config_set" == false ]]; then
+  set -- "$@" -c "$ACCUMULO_CLIENT_PROPS"
+fi
+
+java $JAVA_OPTS -Dlog4j.configurationFile="file:$TEST_LOG4J" \
+  org.apache.accumulo.testing.manager.stress.ManagerStress "$@"