HBASE-30087: Create a nightly Jenkins test for the Read-Replica feature

[kgeisz]

HBASE-30087: Create a nightly Jenkins test for the Read-Replica feature

Note: Some code in the pull request was written using Claude Opus 4.6.

Introduction

• Apache HBase Read Replica Cluster Feature
• Read Replica Test Plan Document
• HBase Website Documentation for Read Replica Cluster

HBase's Read-Replica feature was merged into master in PR #8364. This pull request introduces integration tests for Read-Replica via Jenkins and Docker containers. Jenkins and Docker containers have been useful for testing this feature because it works around the META_TABLE_NAME issue mentioned in HBASE-29691 and PR #7730.

In a Read-Replica setup, clusters share the same storage location and need to have different META_TABLE_NAMEs. The table names are distinguished using the hbase.meta.table.suffix configuration property. However, this still leads to problems when using MiniHBaseCluster to test a multi-cluster Read-Replica setup because these clusters run in the same JVM and end up sharing the same static META_TABLE_NAME variable.

How It Works

• A Read-Replica stage has been added to the dev-support/Jenkinsfile.
• The dev-support/read-replica directory contains various files for building a Docker image, running containers with HBase procecces, and running integration test scripts written in Python. It also contains files and directories used by HBase, such as conf directories.
• The hbase_docker_client.py file is the most important Python file. Every test script uses this file to communicate with each hbase-docker container. It does so by using docker exec to run commands in a container's HBase shell.

The tests are triggered within a stage in HBase's nighly Jenkins run. The stage starts by cloning the current HBase repository to dev-support/read-replica. From here, the repo gets used to build a Docker image designed to run HBase in a Read-Replica setup. After the image has been built, two Docker containers are started, where one starts as the active cluster (read-write mode), and the other container starts as a replica cluster (read-only mode). These clusters share a data-store directory containing the hbase.rootdir. It is a mounted volume between each container and the local filesystem, and it is created by the Python scripts before container startup. The directory is also given 777 permissions in order to avoid HBase startup failures. There are other volumes as well for easy access, such as each container's conf directory, a utils directory for bulkloading data, and a logs directory for preserving HBase logs from each container.

Once the containers are up and running, a series of Python scripts are run as integration tests. They test expected behavior for a Read-Replica cluster setup, such as verifying valid/invalid startup, blocking writes on replica clusters, being able to refresh meta and HFiles on replica clusters to make them consistent with the active cluster, read-only flag flipping (changing hbase.global.readonly.enabled via dynamic configuration), verifying bug fixes, etc. If the Jenkins stage finishes successfully or a failure occurs at any point, then the HBase logs are saved to the Jenkins output directory, the containers are stopped, and the Docker image is deleted.

Other Information

• docker-compose.yml defines and configures two hbase-docker containers capable of running in a read-replica setup.
• build_images.sh uses Dockerfile to build an hbase-docker image. This script assumes there is an hbase repo in the same directory.
• .env defines environment variables used by docker-compose.yml, build_images.sh, and Python scripts.
• proto_compiler.py copies ActiveClusterSuffix.proto from the hbase repo and compiles it. The generated output is used for verifying the active.cluster.suffix.id file.
• utils is a directory containing scripts for bulkloading data into HBase.
• The cluster1 and cluster2 directories contain cluster-specific conf files and HBase log output directories. They each contain their own hbase-site.xml, log4j2.properties, and zoo.cfg files. The directories are defined as mounted volumes in docker-compose.yml. This is especially useful for changing the value of hbase.global.readonly.enabled in order to change a cluster's read-only mode. The mounted logs directory is useful because it helps preserve the HBase logs so they can be copied to the Jenkins output directory if a test failure occurs.

[Apache9]

Better add this to nightly or integration tests. Typically we do not require integration test when merging a PR.

[anmolnar]

Could it be easier to use docker python module to execute commands inside the container instead of subprocessing the docker command?

exec_run(cmd, stdout=True, stderr=True, stdin=False, tty=False, privileged=False, user='', detach=False, stream=False, socket=False, environment=None, workdir=None, demux=False)

Run a command inside this container. Similar to docker exec.

https://docker-py.readthedocs.io/en/stable/containers.html

[kgeisz]

I updated run_docker_exec_command() method in hbase_docker_client.py do use Docker's Python library for running docker exec commands. Unfortunately, this library does not support docker compose, so I am still using Python's subprocess to run docker compose commands.

[anmolnar]

lgtm.

[kgeisz]

Hi @Apache9, thanks for the feedback! I updated the PR by adding a read-replica stage to dev-support/Jenkinsfile. I removed the GitHub Action workflow file as well. Unfortunately, I am not an HBase committer, so I cannot test this file in an upstream Jenkins sandbox. I did, however, test the read-replica stage using a local Jenkins Docker container. The Jenkinsfile I used was a tuncated version of the upstream Jenkinsfile that only has code relevent to the read-replica test stage (I left out Yetus stages, etc.).

Is there any way this updated Jenkinsfile can be tested with upstream Jenkins before merging the PR?
cc. @anmolnar @taklwu @ndimiduk @apurtell