diff --git a/.github/actions/setup-chart-testing/action.yml b/.github/actions/setup-chart-testing/action.yml new file mode 100644 index 0000000..c0a0b39 --- /dev/null +++ b/.github/actions/setup-chart-testing/action.yml @@ -0,0 +1,72 @@ +name: Set up chart-testing +description: >- + Install helm chart-testing (ct) plus yamllint and yamale, reproducing + helm/chart-testing-action but wrapping the cosign-verified download in a retry + so a transient sigstore/Rekor HTTP/2 stream reset does not fail the job. + Requires actions/setup-python to have run first (yamllint/yamale go into a + dedicated venv built from that Python). +runs: + using: composite + steps: + - name: Install cosign + uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 + + # The upstream helm/chart-testing-action runs a bare `cosign verify-blob` + # whose Rekor transparency-log query is the one un-retried network call; a + # transient HTTP/2 stream reset there fails the whole Lint Chart job. Wrap + # the download + verify + install in a retry so the blip is recoverable. + - name: Download, verify and install chart-testing + uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0 + with: + shell: bash + timeout_minutes: 5 + max_attempts: 3 + retry_wait_seconds: 10 + command: | + set -euo pipefail + + CT_VERSION="v3.14.0" + YAMLLINT_VERSION="1.38.0" + YAMALE_VERSION="6.1.0" + + if [[ "$(uname -m)" == "aarch64" ]]; then arch=arm64; else arch=amd64; fi + ver="${CT_VERSION#v}" + dir="${RUNNER_TEMP}/ct" + base="https://github.com/helm/chart-testing/releases/download/${CT_VERSION}" + tarball="chart-testing_${ver}_linux_${arch}.tar.gz" + mkdir -p "${dir}" + + curl --retry 5 --retry-delay 1 -sSLo "${dir}/ct.tar.gz" "${base}/${tarball}" + curl --retry 5 --retry-delay 1 -sSLo "${dir}/ct.tar.gz.pem" "${base}/${tarball}.pem" + curl --retry 5 --retry-delay 1 -sSLo "${dir}/ct.tar.gz.sig" "${base}/${tarball}.sig" + + # Keyless verification against the chart-testing release workflow's + # identity. The identity/issuer are stable across ct versions (they + # reference the workflow ref, not the version), so only CT_VERSION + # changes on a bump. + cosign verify-blob \ + --certificate "${dir}/ct.tar.gz.pem" \ + --signature "${dir}/ct.tar.gz.sig" \ + --certificate-identity "https://github.com/helm/chart-testing/.github/workflows/release.yaml@refs/heads/main" \ + --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ + "${dir}/ct.tar.gz" + + tar -xzf "${dir}/ct.tar.gz" -C "${dir}" + rm -f "${dir}/ct.tar.gz" "${dir}/ct.tar.gz.pem" "${dir}/ct.tar.gz.sig" + + # Install yamllint + yamale into a dedicated venv and put its bin dir + # on PATH explicitly, mirroring upstream ct.sh. ct lint shells out to + # both tools by name, so relying on the ambient pip console-script dir + # being on PATH would be fragile — the venv makes resolution explicit. + python3 -m venv "${dir}/venv" + "${dir}/venv/bin/pip" install "yamllint==${YAMLLINT_VERSION}" "yamale==${YAMALE_VERSION}" + + # $GITHUB_PATH / $GITHUB_ENV are job-scoped files, so these appends + # persist to subsequent job steps even though they run inside the + # retry wrapper. CT_CONFIG_DIR points at the tarball's etc/ dir + # (chart_schema.yaml + lintconf.yaml), matching upstream ct.sh. + echo "${dir}" >> "${GITHUB_PATH}" + echo "${dir}/venv/bin" >> "${GITHUB_PATH}" + echo "CT_CONFIG_DIR=${dir}/etc" >> "${GITHUB_ENV}" + + "${dir}/ct" version diff --git a/.github/workflows/pr-privileged.yaml b/.github/workflows/pr-privileged.yaml new file mode 100644 index 0000000..94ce9ee --- /dev/null +++ b/.github/workflows/pr-privileged.yaml @@ -0,0 +1,201 @@ +name: PR Privileged Actions + +# workflow_run always runs the default-branch copy of this file, so a fork's +# PR cannot change what runs with these permissions. +on: + workflow_run: + workflows: ["PR Checks and Build"] + types: [completed] + +permissions: + pull-requests: write + security-events: write + actions: read + +jobs: + upload-sarif: + name: Upload Security Results + runs-on: ubuntu-latest + timeout-minutes: 10 + if: github.event.workflow_run.event == 'pull_request' + steps: + - name: Download PR metadata + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: pr-metadata + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + + - name: Read and validate PR number + id: pr + run: | + PR_NUMBER=$(cat pr-number.txt) + if ! [[ "$PR_NUMBER" =~ ^[0-9]+$ ]]; then + echo "ERROR: Invalid PR number: $PR_NUMBER" + exit 1 + fi + echo "number=$PR_NUMBER" >> "$GITHUB_OUTPUT" + + # pr-number.txt comes out of the pull_request run, which a fork's own + # copy of pr.yaml produces, so it can name any PR. Act on that PR only + # when it is the one this run built: same head commit, same head repo. + - name: Check the PR is the one this run built + id: match + env: + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ steps.pr.outputs.number }} + RUN_HEAD: ${{ github.event.workflow_run.head_sha }} ${{ github.event.workflow_run.head_repository.full_name }} + run: | + set -euo pipefail + pr_head="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq '"\(.head.sha) \(.head.repo.full_name)"')" + if [[ "${pr_head}" == "${RUN_HEAD}" ]]; then + echo "current=true" >> "$GITHUB_OUTPUT" + else + echo "::notice::PR #${PR_NUMBER} is at '${pr_head}', this run built '${RUN_HEAD}'; skipping" + echo "current=false" >> "$GITHUB_OUTPUT" + fi + + - name: Download SARIF artifact + id: download-sarif + if: steps.match.outputs.current == 'true' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: trivy-sarif + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + continue-on-error: true + + - name: Upload SARIF to GitHub Security + if: steps.match.outputs.current == 'true' && steps.download-sarif.outcome == 'success' + uses: github/codeql-action/upload-sarif@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4 + with: + sarif_file: trivy-fs-results.sarif + category: trivy-filesystem + ref: refs/pull/${{ steps.pr.outputs.number }}/head + sha: ${{ github.event.workflow_run.head_sha }} + + notify: + name: Notify PR + runs-on: ubuntu-latest + timeout-minutes: 10 + if: >- + github.event.workflow_run.event == 'pull_request' && + github.event.workflow_run.conclusion == 'success' + steps: + - name: Download PR metadata + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: pr-metadata + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + + - name: Read and validate PR number + id: pr + run: | + PR_NUMBER=$(cat pr-number.txt) + if ! [[ "$PR_NUMBER" =~ ^[0-9]+$ ]]; then + echo "ERROR: Invalid PR number: $PR_NUMBER" + exit 1 + fi + echo "number=$PR_NUMBER" >> "$GITHUB_OUTPUT" + + - name: Check the PR is the one this run built + id: match + env: + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ steps.pr.outputs.number }} + RUN_HEAD: ${{ github.event.workflow_run.head_sha }} ${{ github.event.workflow_run.head_repository.full_name }} + run: | + set -euo pipefail + pr_head="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq '"\(.head.sha) \(.head.repo.full_name)"')" + if [[ "${pr_head}" == "${RUN_HEAD}" ]]; then + echo "current=true" >> "$GITHUB_OUTPUT" + else + echo "::notice::PR #${PR_NUMBER} is at '${pr_head}', this run built '${RUN_HEAD}'; skipping" + echo "current=false" >> "$GITHUB_OUTPUT" + fi + + - name: Download image reference + if: steps.match.outputs.current == 'true' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: image-ref + path: image-ref + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + + # The reference comes out of a pull_request run and goes into a comment + # this token writes, so only the exact shape merge-manifests exports is + # accepted. + - name: Read and validate image reference + id: ref + if: steps.match.outputs.current == 'true' + run: | + set -euo pipefail + value="$(tr -d '[:space:]' < image-ref/robotlb.ref)" + if ! [[ "${value}" =~ ^ttl\.sh/treetscom-robotlb@sha256:[0-9a-f]{64}$ ]]; then + echo "ERROR: robotlb.ref is not a digest-pinned reference to ttl.sh/treetscom-robotlb" + exit 1 + fi + echo "image=${value}" >> "$GITHUB_OUTPUT" + + # One comment per PR, rewritten on every successful run: a comment + # naming an older build's digest would send readers to the wrong image. + - name: Comment on PR + if: steps.match.outputs.current == 'true' + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + env: + PR_NUMBER: ${{ steps.pr.outputs.number }} + IMAGE_REF: ${{ steps.ref.outputs.image }} + HEAD_SHA: ${{ github.event.workflow_run.head_sha }} + RUN_URL: ${{ github.event.workflow_run.html_url }} + with: + script: | + const marker = ''; + const prNumber = Number(process.env.PR_NUMBER); + const fence = '```'; + + const body = [ + marker, + '## Container image available', + '', + `Built from ${process.env.HEAD_SHA} in [this run](${process.env.RUN_URL}). Pull by digest: the \`pr-${prNumber}-1d\` tag on ttl.sh is anonymous, and anyone can overwrite it.`, + '', + `${fence}bash`, + `podman pull ${process.env.IMAGE_REF}`, + fence, + '', + '## Helm chart available', + '', + `${fence}bash`, + `helm install robotlb oci://ttl.sh/treetscom/robotlb --version 0.0.0-pr.${prNumber}-1d`, + fence, + '', + `The chart points at the \`pr-${prNumber}-1d\` image tag. ttl.sh expires both on its own schedule; re-run this PR's CI to republish.`, + ].join('\n'); + + const comments = await github.paginate(github.rest.issues.listComments, { + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + per_page: 100, + }); + const existing = comments.find( + (c) => c.user?.login === 'github-actions[bot]' && c.body?.includes(marker), + ); + + if (existing) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: existing.id, + body, + }); + } else { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + body, + }); + } diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml new file mode 100644 index 0000000..ad1df80 --- /dev/null +++ b/.github/workflows/pr.yaml @@ -0,0 +1,425 @@ +name: PR Checks and Build + +on: + pull_request: + types: [opened, synchronize, reopened] + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + +permissions: + contents: read + +env: + RUST_TOOLCHAIN: '1.98.0' + CI_IMAGE: ttl.sh/treetscom-robotlb + CHART_DIR: helm + CHART_NAME: robotlb + +jobs: + # pr-privileged.yaml reads this to know which PR the run belongs to. + save-pr-metadata: + name: Save PR Metadata + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Save PR number + run: echo "${{ github.event.pull_request.number }}" > pr-number.txt + + - name: Upload PR metadata + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: pr-metadata + path: pr-number.txt + retention-days: 1 + + security: + name: Security Scan + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Run Trivy vulnerability scanner (filesystem) + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 + with: + scan-type: 'fs' + scan-ref: '.' + format: 'sarif' + output: 'trivy-fs-results.sarif' + severity: 'CRITICAL,HIGH,MEDIUM' + + # A pull_request token cannot write security events; pr-privileged.yaml + # uploads this file from the base repository's context. + - name: Upload Trivy SARIF as artifact + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: trivy-sarif + path: trivy-fs-results.sarif + retention-days: 1 + + lint: + name: Lint + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Setup Rust + uses: actions-rust-lang/setup-rust-toolchain@ecabd13d1c56bd1345c230e542e9144811ad706f # v2.0.0 + with: + toolchain: ${{ env.RUST_TOOLCHAIN }} + components: rustfmt, clippy + + - name: cargo fmt + run: cargo fmt --all -- --check + + - name: cargo clippy + run: cargo clippy --all-targets --locked -- --deny warnings + + - name: markdownlint + uses: DavidAnson/markdownlint-cli2-action@21c1be1b93ad9ed58fa840aacc3f279cde2a72ff # v24 + with: + globs: '**/*.md' + + # actionlint also runs shellcheck on inline `run:` blocks. shellcheck comes + # from a digest-pinned image into /usr/local/bin, ahead of the runner's + # /usr/bin, so actionlint picks up the pinned one. + lint-workflows: + name: Lint Workflows + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Setup Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 + with: + go-version: '1.27.1' + cache: false + + - name: Install shellcheck and actionlint + run: | + set -euo pipefail + + ACTIONLINT_VERSION="v1.7.12" + # actionlint is Go, so the checksum database attests it. + go install "github.com/rhysd/actionlint/cmd/actionlint@${ACTIONLINT_VERSION}" + echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH" + + # The shellcheck releases carry no checksum or signature file, so + # the binary comes out of the upstream image, pinned by digest. + SHELLCHECK_IMAGE="koalaman/shellcheck:v0.11.0@sha256:61862eba1fcf09a484ebcc6feea46f1782532571a34ed51fedf90dd25f925a8d" + docker create --name shellcheck-tmp "${SHELLCHECK_IMAGE}" + sudo docker cp shellcheck-tmp:/bin/shellcheck /usr/local/bin/shellcheck + docker rm shellcheck-tmp + + shellcheck --version + "$(go env GOPATH)/bin/actionlint" --version + + - name: Run shellcheck (hack scripts) + run: shellcheck hack/*.sh + + - name: Run actionlint (workflows) + run: actionlint -color + + - name: Every workflow job carries a timeout + run: | + set -euo pipefail + missing=0 + for f in .github/workflows/*.y*ml; do + jobs="$(yq '.jobs | to_entries[] | select(.value.uses == null and .value["timeout-minutes"] == null) | .key' "${f}")" + for job in ${jobs}; do + echo "::error file=${f}::job '${job}' has no timeout-minutes" + missing=1 + done + done + exit "${missing}" + + test: + name: Test + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Setup Rust + uses: actions-rust-lang/setup-rust-toolchain@ecabd13d1c56bd1345c230e542e9144811ad706f # v2.0.0 + with: + toolchain: ${{ env.RUST_TOOLCHAIN }} + + - name: Run tests + run: cargo test --all --locked + + lint-chart: + name: Lint Chart + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + fetch-depth: 0 + + - name: Set up Helm + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5 + with: + version: 'v4.3.0' + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 + with: + python-version: '3.14.7' + + - name: Set up chart-testing + uses: ./.github/actions/setup-chart-testing + + - name: Run Helm Lint + run: helm lint "${CHART_DIR}" + + # The release workflow sets the chart version from the release tag, so + # a chart change carries no version bump of its own. The chart lists no + # maintainers. + - name: Run chart-testing (lint) + run: | + ct lint --charts "${CHART_DIR}" \ + --check-version-increment=false \ + --validate-maintainers=false + + - name: Template Chart with defaults + run: helm template test-release "${CHART_DIR}" + + build-and-push: + name: Build and Push (${{ matrix.arch }}) + needs: [security, lint, lint-workflows, test] + runs-on: ${{ matrix.runner }} + # ttl.sh sometimes stalls a push for tens of minutes; without a bound a + # stalled leg holds the runner for the six-hour default. + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - platform: linux/amd64 + arch: amd64 + runner: ubuntu-24.04 + - platform: linux/arm64 + arch: arm64 + runner: ubuntu-24.04-arm + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 + + # ttl.sh intermittently drops blob writes after a clean build. The retry + # runs only when the first attempt failed, and the gha layer cache keeps + # it cheap. + - name: Build and Push to ttl.sh by digest + id: build + continue-on-error: true + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7 + with: + context: . + file: ./Dockerfile + platforms: ${{ matrix.platform }} + outputs: type=image,name=${{ env.CI_IMAGE }},push-by-digest=true,name-canonical=true,push=true + cache-from: type=gha,scope=build-${{ matrix.arch }} + cache-to: type=gha,scope=build-${{ matrix.arch }},mode=max + + - name: Build and Push to ttl.sh by digest (retry) + id: build-retry + if: steps.build.outcome == 'failure' + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7 + with: + context: . + file: ./Dockerfile + platforms: ${{ matrix.platform }} + outputs: type=image,name=${{ env.CI_IMAGE }},push-by-digest=true,name-canonical=true,push=true + cache-from: type=gha,scope=build-${{ matrix.arch }} + cache-to: type=gha,scope=build-${{ matrix.arch }},mode=max + + - name: Export digest + run: | + mkdir -p /tmp/digests + if [[ "${{ steps.build.outcome }}" == "success" ]]; then + digest="${{ steps.build.outputs.digest }}" + else + digest="${{ steps.build-retry.outputs.digest }}" + fi + touch "/tmp/digests/${digest#sha256:}" + + - name: Upload digest + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: digests-${{ matrix.arch }} + path: /tmp/digests/* + if-no-files-found: error + retention-days: 1 + + merge-manifests: + name: Merge Manifests + runs-on: ubuntu-latest + needs: build-and-push + timeout-minutes: 20 + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + sparse-checkout: hack + + - name: Download digests + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + pattern: digests-* + path: /tmp/digests + merge-multiple: true + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 + + - name: Create manifest list and push + working-directory: /tmp/digests + run: | + set -euo pipefail + + PR_NUMBER="${{ github.event.pull_request.number }}" + TAG="pr-${PR_NUMBER}-1d" + # pr-N-1d is a human alias that later runs overwrite. The run-scoped + # tag is narrower, but ttl.sh is unauthenticated and the run id is + # public, so what it resolves to is checked in the next step. + RUN_TAG="pr-${PR_NUMBER}-run-${{ github.run_id }}" + + images=() + for digest_file in *; do + [[ -f "$digest_file" ]] || continue + images+=("${CI_IMAGE}@sha256:${digest_file}") + done + + ok=0 + for attempt in 1 2 3; do + if docker buildx imagetools create \ + --tag "${CI_IMAGE}:${TAG}" \ + --tag "${CI_IMAGE}:${RUN_TAG}" \ + "${images[@]}"; then + ok=1 + break + fi + if [[ "${attempt}" -lt 3 ]]; then + echo "imagetools create failed (attempt ${attempt}); retrying in 5s..." + sleep 5 + fi + done + [[ "${ok}" -eq 1 ]] + + # Anyone can overwrite a ttl.sh tag, so the manifest list is published by + # digest, after checking it lists exactly the manifests this run pushed. + - name: Export image reference + run: | + set -euo pipefail + + PR_NUMBER="${{ github.event.pull_request.number }}" + mkdir -p /tmp/refs + digest="$(docker buildx imagetools inspect \ + "${CI_IMAGE}:pr-${PR_NUMBER}-run-${{ github.run_id }}" \ + --format '{{.Manifest.Digest}}')" + ref="${CI_IMAGE}@${digest}" + + docker buildx imagetools inspect "${ref}" --raw > /tmp/index.json + hack/verify-manifest-children.sh "${CI_IMAGE}" /tmp/index.json /tmp/digests + + echo "${ref}" > /tmp/refs/robotlb.ref + docker buildx imagetools inspect "${ref}" + + - name: Upload image reference + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: image-ref + path: /tmp/refs/* + if-no-files-found: error + retention-days: 1 + + - name: Generate build summary + run: | + PR_NUMBER="${{ github.event.pull_request.number }}" + IMAGE_REF="$(cat /tmp/refs/robotlb.ref)" + + { + echo "## Multi-arch Container Image" + echo "" + echo "- \`${IMAGE_REF}\`" + echo "- \`${CI_IMAGE}:pr-${PR_NUMBER}-1d\` (mutable: anyone can overwrite a ttl.sh tag)" + echo "" + echo "Platforms: linux/amd64, linux/arm64" + echo "" + echo '```bash' + echo "podman pull ${IMAGE_REF}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + build-chart: + name: Build Chart + runs-on: ubuntu-latest + needs: [security, lint, lint-workflows, test, lint-chart] + # Same ttl.sh budget as the image legs: the chart push stalls the same way. + timeout-minutes: 60 + steps: + - name: Checkout PR code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Set up Helm + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5 + with: + version: 'v4.3.0' + + - name: Point the chart at the PR image + run: | + set -euo pipefail + + PR_NUMBER="${{ github.event.pull_request.number }}" + TAG="pr-${PR_NUMBER}-1d" + + yq --inplace ".version = \"0.0.0-pr.${PR_NUMBER}-1d\"" "${CHART_DIR}/Chart.yaml" + yq --inplace ".appVersion = \"${TAG}\"" "${CHART_DIR}/Chart.yaml" + yq --inplace ".image.repository = \"${CI_IMAGE}\"" "${CHART_DIR}/values.yaml" + yq --inplace ".image.tag = \"${TAG}\"" "${CHART_DIR}/values.yaml" + # The PR tag moves with every push, so a node must not keep a cached + # older build on a rolling restart. + yq --inplace ".image.pullPolicy = \"Always\"" "${CHART_DIR}/values.yaml" + + [[ "$(yq '.image.repository' "${CHART_DIR}/values.yaml")" == "${CI_IMAGE}" ]] + [[ "$(yq '.image.tag' "${CHART_DIR}/values.yaml")" == "${TAG}" ]] + + - name: Package and push chart to ttl.sh + run: | + set -euo pipefail + + CHART_VERSION="0.0.0-pr.${{ github.event.pull_request.number }}-1d" + helm package "${CHART_DIR}" + + ok=0 + for attempt in 1 2 3; do + if helm push "${CHART_NAME}-${CHART_VERSION}.tgz" oci://ttl.sh/treetscom; then + ok=1 + break + fi + if [[ "${attempt}" -lt 3 ]]; then + echo "helm push failed (attempt ${attempt}); retrying in 5s..." + sleep 5 + fi + done + [[ "${ok}" -eq 1 ]] + + { + echo "## Helm Chart" + echo "" + echo '```bash' + echo "helm install robotlb oci://ttl.sh/treetscom/${CHART_NAME} --version ${CHART_VERSION}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 7337937..9d39392 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -4,55 +4,228 @@ on: release: types: [released] +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + permissions: contents: read - packages: write +env: + IMAGE: ghcr.io/treetscom/robotlb + CHART_DIR: helm + CHART_NAME: robotlb + CHART_OCI_REPO: ghcr.io/treetscom/charts/robotlb jobs: - release_image: - runs-on: ubuntu-latest + build-and-push: + name: Build and Push (${{ matrix.arch }}) + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + permissions: + contents: read + packages: write + strategy: + fail-fast: false + matrix: + include: + - platform: linux/amd64 + arch: amd64 + runner: ubuntu-24.04 + - platform: linux/arm64 + arch: arm64 + runner: ubuntu-24.04-arm steps: - - name: Checkout - uses: actions/checkout@v4 + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + # The binary reports the version from Cargo.toml, which stays 0.0.0 in git. - name: Update version - # Update a version in Cargo.toml before building the app. run: sed -i '0,/^version = .*/{s/version = .*/version = "${{ github.ref_name }}"/}' Cargo.toml - - name: Set up Docker - uses: docker/setup-qemu-action@v3 + - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 + - name: Login to GitHub Container Registry - uses: docker/login-action@v2 + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: - registry: ghcr.io - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - name: Build and push - uses: docker/build-push-action@v2 + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Docker meta + id: meta + uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6 + with: + images: ${{ env.IMAGE }} + + - name: Build and Push by Digest + id: build + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7 with: context: . file: ./Dockerfile - platforms: linux/amd64 # ,linux/arm64,linux/arm/v7 (uncomment once the fix for long builds is found). - push: true - tags: ghcr.io/treetscom/robotlb:latest,ghcr.io/treetscom/robotlb:${{ github.ref_name }} + platforms: ${{ matrix.platform }} + outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true + cache-from: type=gha,scope=build-${{ matrix.arch }} + cache-to: type=gha,scope=build-${{ matrix.arch }},mode=max + labels: ${{ steps.meta.outputs.labels }} - upload_helm: + - name: Export digest + run: | + mkdir -p /tmp/digests + digest="${{ steps.build.outputs.digest }}" + touch "/tmp/digests/${digest#sha256:}" + + - name: Upload digest + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: digests-${{ matrix.arch }} + path: /tmp/digests/* + if-no-files-found: error + retention-days: 1 + + merge-manifests: + name: Merge Manifests runs-on: ubuntu-latest - needs: [release_image] + needs: build-and-push + timeout-minutes: 20 + permissions: + contents: read + packages: write + id-token: write steps: - - uses: actions/checkout@v4 - - uses: actions-rust-lang/setup-rust-toolchain@v1 + - name: Download digests + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + pattern: digests-* + path: /tmp/digests + merge-multiple: true + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 + + - name: Login to GitHub Container Registry + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: - toolchain: stable - - uses: azure/setup-helm@v4.2.0 + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Docker meta + id: meta + uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6 with: - version: latest - - name: Build Helm chart + images: ${{ env.IMAGE }} + tags: | + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + type=raw,value=latest + + - name: Create manifest list and push + working-directory: /tmp/digests + run: | + set -euo pipefail + mapfile -t tag_args < <(jq -r '.tags[] | "-t", .' <<< "$DOCKER_METADATA_OUTPUT_JSON") + images=() + for digest_file in *; do + [[ -f "$digest_file" ]] || continue + images+=("${IMAGE}@sha256:${digest_file}") + done + docker buildx imagetools create "${tag_args[@]}" "${images[@]}" + + # Signing the digest covers every tag, so each tag is checked to point at it. + - name: Resolve manifest digest + id: digest run: | - helm package --app-version "${{ github.ref_name }}" --dependency-update ./helm - helm show chart *.tgz - helm registry login -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }} ghcr.io - helm push *.tgz oci://ghcr.io/treetscom/charts + set -euo pipefail + digest="$(docker buildx imagetools inspect "${IMAGE}:${{ steps.meta.outputs.version }}" --format '{{.Manifest.Digest}}')" + [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] \ + || { echo "::error::cannot resolve a manifest digest"; exit 1; } + while read -r tag; do + [[ -n "${tag}" ]] || continue + tag_digest="$(docker buildx imagetools inspect "${tag}" --format '{{.Manifest.Digest}}')" + if [[ "${tag_digest}" != "${digest}" ]]; then + echo "::error::${tag} resolves to ${tag_digest}, expected ${digest}" + exit 1 + fi + done <<< "$DOCKER_METADATA_OUTPUT_TAGS" + echo "ref=${IMAGE}@${digest}" >> "$GITHUB_OUTPUT" + + - name: Install cosign + uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 + + - name: Sign container image + env: + IMAGE_REF: ${{ steps.digest.outputs.ref }} + run: cosign sign --yes "${IMAGE_REF}" + + - name: Generate container summary + env: + IMAGE_REF: ${{ steps.digest.outputs.ref }} + run: | + { + echo "## Container Image" + echo "" + echo '```text' + echo "$DOCKER_METADATA_OUTPUT_TAGS" + echo '```' + echo "" + echo '```bash' + echo "cosign verify ${IMAGE_REF} --certificate-identity-regexp=https://github.com/${{ github.repository }} --certificate-oidc-issuer=https://token.actions.githubusercontent.com" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + publish-chart: + name: Publish Helm Chart + runs-on: ubuntu-latest + needs: merge-manifests + timeout-minutes: 20 + permissions: + contents: write + packages: write + id-token: write + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Install Helm + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5 + with: + version: 'v4.3.0' + + - name: Install cosign + uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 + + - name: Package chart + run: | + helm package "${CHART_DIR}" \ + --version "${{ github.ref_name }}" \ + --app-version "${{ github.ref_name }}" + + - name: Push and sign chart + run: | + set -euo pipefail + echo "${{ secrets.GITHUB_TOKEN }}" \ + | helm registry login ghcr.io --username "${{ github.actor }}" --password-stdin + + output="$(helm push "${CHART_NAME}-${{ github.ref_name }}.tgz" oci://ghcr.io/treetscom/charts 2>&1)" + echo "${output}" + digest="$(sed -n 's/^Digest: \(sha256:[0-9a-f]\{64\}\)$/\1/p' <<< "${output}")" + [[ -n "${digest}" ]] || { echo "::error::helm push reported no chart digest"; exit 1; } + + cosign sign --yes "${CHART_OCI_REPO}@${digest}" + + { + echo "## Helm Chart" + echo "" + echo '```bash' + echo "helm install robotlb oci://${CHART_OCI_REPO} --version ${{ github.ref_name }}" + echo "cosign verify ${CHART_OCI_REPO}@${digest} --certificate-identity-regexp=https://github.com/${{ github.repository }} --certificate-oidc-issuer=https://token.actions.githubusercontent.com" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + - name: Attach chart to the release env: - HELM_EXPERIMENTAL_OCI: 1 + GH_TOKEN: ${{ github.token }} + run: gh release upload "${{ github.ref_name }}" "${CHART_NAME}-${{ github.ref_name }}.tgz" diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml deleted file mode 100644 index af8b2a1..0000000 --- a/.github/workflows/tests.yaml +++ /dev/null @@ -1,16 +0,0 @@ -name: "Testing package" - -on: pull_request - -jobs: - pre-commit: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v3 - - uses: actions-rs/toolchain@v1 - with: - toolchain: stable - override: true - components: rustfmt, clippy - - uses: pre-commit/action@v3.0.0 diff --git a/.markdownlint-cli2.yaml b/.markdownlint-cli2.yaml new file mode 100644 index 0000000..69d5ae9 --- /dev/null +++ b/.markdownlint-cli2.yaml @@ -0,0 +1,23 @@ +# markdownlint-cli2 configuration + +ignores: + - target/** + +# Rule configuration +config: + # Disable line length rule - technical documentation often needs longer lines + MD013: false + # Allow both duplicate headings - common in docs with multiple sections + MD024: false + # Allow ordered lists with any prefix style (1. 2. 3. or all 1.) + MD029: false + # Allow any spacing after list markers - MkDocs compatibility + MD030: false + # Allow inline HTML for badges, special formatting + MD033: false + # Allow bare URLs - common in technical docs + MD034: false + # Allow both fenced and indented code blocks + MD046: false + # Disable table column style rule - compact style is fine + MD060: false diff --git a/README.md b/README.md index 47f69c2..7c5f1d6 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,6 @@ Every port of the service needs an allocated `nodePort`. A Hetzner load balancer > Earlier releases treated every service as if it had the `Local` policy. Services that leave `externalTrafficPolicy` unset therefore get the full node list on upgrade, which changes the targets of their existing balancers. - ## Configuration This project has two places for configuration: environment variables and service annotations. @@ -63,7 +62,7 @@ Environment variables are mainly used to override default arguments and provide Here’s a complete list of parameters for the operator's binary: -``` +```text Usage: robotlb [OPTIONS] --hcloud-token Options: @@ -97,10 +96,8 @@ Options: Print help ``` - ### Service annotations - ```yaml apiVersion: v1 kind: Service diff --git a/hack/verify-manifest-children.sh b/hack/verify-manifest-children.sh new file mode 100755 index 0000000..8ac4681 --- /dev/null +++ b/hack/verify-manifest-children.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# verify-manifest-children.sh — assert a published manifest index was assembled +# out of the images this build job pushed. +# +# The merge job reads its manifest-list digest back from a tag, and ttl.sh has +# no authentication while the run id in that tag is public for as long as the +# run is. In the seconds between the manifest push and the read-back, anyone can +# repoint that tag. The digest read back would then be theirs, and every check +# downstream would accept it, because it really is a well-formed content address +# to a real image. It is just not this build. +# +# What `/tmp/digests` names is not what the published index lists. Provenance is +# on, so each per-arch build pushes an OCI index of its own -- a platform +# manifest plus an attestation manifest -- and `imagetools create` flattens +# index sources, so the published index carries those children and never the +# source digests. Each pushed digest is therefore resolved to its own index, and +# it is that index's children that are compared. +# +# The comparison is equality, not containment. Nothing here constrains the +# `platform` field, and a runtime pulls the image by platform, so +# an index that keeps all of this job's manifests, relabels the amd64 one and +# puts a foreign manifest in the linux/amd64 slot would satisfy a subset test +# and still hand over a substituted image. Requiring the published set to hold +# nothing else is what closes that. `imagetools create` adds nothing of its own +# to a flattened index -- a real run publishes exactly the union of its sources' +# children -- so equality is what the merge job actually produces. +# +# Usage: verify-manifest-children.sh + +set -euo pipefail + +die() { echo "ERROR: $*" >&2; exit 1; } + +[[ $# -eq 3 ]] || die "usage: $0 " + +image="$1" +published="$2" +digests_dir="$3" + +[[ -f "${published}" ]] || die "manifest index ${published} does not exist" +[[ -d "${digests_dir}" ]] || die "digests directory ${digests_dir} does not exist" + +expected="" +for digest_file in "${digests_dir}"/*; do + [[ -f "${digest_file}" ]] || continue + source_digest="sha256:${digest_file##*/}" + + raw="$(docker buildx imagetools inspect "${image}@${source_digest}" --raw)" \ + || die "cannot read ${source_digest}, which this job pushed" + + # With provenance off a build pushes a plain manifest instead of an index. It + # has no children, and then the source digest is itself what must appear. + children="$(jq --raw-output '.manifests[]?.digest' <<< "${raw}")" + [[ -n "${children}" ]] || children="${source_digest}" + expected+="${children}"$'\n' +done + +# Without this an empty directory would leave both sets empty and equal, and the +# check would wave through any index at all. +[[ -n "${expected//[[:space:]]/}" ]] \ + || die "no pushed digests to check in ${digests_dir}" + +expected="$(grep --invert-match '^$' <<< "${expected}" | LC_ALL=C sort --unique)" +found="$(jq --raw-output '.manifests[]?.digest' "${published}" | LC_ALL=C sort --unique)" + +missing="$(grep --fixed-strings --line-regexp --invert-match \ + --file=<(printf '%s\n' "${found}") <<< "${expected}" || true)" +[[ -z "${missing}" ]] \ + || die "the published index does not list $(tr '\n' ' ' <<< "${missing}")-- pushed by this job" + +extra="$(grep --fixed-strings --line-regexp --invert-match \ + --file=<(printf '%s\n' "${expected}") <<< "${found}" || true)" +[[ -z "${extra}" ]] \ + || die "the published index also lists $(tr '\n' ' ' <<< "${extra}")-- not pushed by this job" + +echo "published index matches the $(wc -l <<< "${expected}" | tr -d ' ') manifests this job pushed" diff --git a/src/config.rs b/src/config.rs index 6ce213b..bd0c500 100644 --- a/src/config.rs +++ b/src/config.rs @@ -30,13 +30,13 @@ pub struct OperatorConfig { pub default_lb_interval: i32, /// Default location of a load balancer. - /// https://docs.hetzner.com/cloud/general/locations/ + /// #[arg(long, env = "ROBOTLB_DEFAULT_LB_LOCATION", default_value = "hel1")] pub default_lb_location: String, /// Type of a load balancer. It differs in price, number of connections, /// target servers, etc. The default value is the smallest balancer. - /// https://docs.hetzner.com/cloud/load-balancers/overview#pricing + /// #[arg(long, env = "ROBOTLB_DEFAULT_LB_TYPE", default_value = "lb11")] pub default_balancer_type: String, @@ -44,7 +44,8 @@ pub struct OperatorConfig { /// Possible values: /// * `least-connections` /// * `round-robin` - /// https://docs.hetzner.com/cloud/load-balancers/overview#load-balancers + /// + /// #[arg( long, env = "ROBOTLB_DEFAULT_LB_ALGORITHM", @@ -54,7 +55,7 @@ pub struct OperatorConfig { /// Default load balancer proxy mode. If enabled, the load balancer will /// act as a proxy for the target servers. The default value is `false`. - /// https://docs.hetzner.com/cloud/load-balancers/faq/#what-does-proxy-protocol-mean-and-should-i-enable-it + /// #[arg( long, env = "ROBOTLB_DEFAULT_LB_PROXY_MODE_ENABLED", diff --git a/src/finalizers.rs b/src/finalizers.rs index 2f65e5b..a2fdc74 100644 --- a/src/finalizers.rs +++ b/src/finalizers.rs @@ -35,9 +35,7 @@ pub fn check(service: &Service) -> bool { .metadata .finalizers .as_ref() - .map_or(false, |finalizers| { - finalizers.contains(&consts::FINALIZER_NAME.to_string()) - }) + .is_some_and(|finalizers| finalizers.contains(&consts::FINALIZER_NAME.to_string())) } /// Remove finalizer from the service. diff --git a/src/lb.rs b/src/lb.rs index f3cf34e..3ce5b90 100644 --- a/src/lb.rs +++ b/src/lb.rs @@ -135,7 +135,7 @@ impl LoadBalancer { .annotations() .get(consts::LB_NAME_LABEL_NAME) .cloned() - .unwrap_or(svc.name_any()); + .unwrap_or_else(|| svc.name_any()); let private_ip = svc .annotations() @@ -424,7 +424,7 @@ impl LoadBalancer { &self, hcloud_balancer: &hcloud::models::LoadBalancer, ) -> RobotLBResult<()> { - if *hcloud_balancer.algorithm == self.algorithm.clone().into() { + if *hcloud_balancer.algorithm == self.algorithm.clone() { return Ok(()); } tracing::info!( @@ -437,7 +437,7 @@ impl LoadBalancer { &self.hcloud_config, ChangeAlgorithmParams { id: hcloud_balancer.id, - body: Some(self.algorithm.clone().into()), + body: Some(self.algorithm.clone()), }, ) }) @@ -613,7 +613,7 @@ impl LoadBalancer { let hcloud_balancers = hcloud::apis::load_balancers_api::list_load_balancers( &self.hcloud_config, ListLoadBalancersParams { - name: Some(self.name.to_string()), + name: Some(self.name.clone()), ..Default::default() }, ) @@ -688,15 +688,13 @@ impl LoadBalancer { network_name ); return Err(RobotLBError::HCloudError(format!( - "Found more than one network with name {}", - network_name, + "Found more than one network with name {network_name}", ))); } if response.networks.is_empty() { tracing::warn!("Network with name {} not found", network_name); return Err(RobotLBError::HCloudError(format!( - "Network with name {} not found", - network_name, + "Network with name {network_name} not found", ))); } diff --git a/src/main.rs b/src/main.rs index feb4553..d6b2dc8 100644 --- a/src/main.rs +++ b/src/main.rs @@ -322,8 +322,7 @@ async fn sync_service(svc: Arc, context: Arc) -> RobotL .spec .as_ref() .and_then(|s| s.type_.as_ref()) - .map(String::as_str) - .unwrap_or("ClusterIP"); + .map_or("ClusterIP", String::as_str); if svc_type != "LoadBalancer" { tracing::debug!("Service type is not LoadBalancer. Skipping..."); return Err(RobotLBError::SkipService); @@ -333,8 +332,7 @@ async fn sync_service(svc: Arc, context: Arc) -> RobotL .spec .as_ref() .and_then(|s| s.load_balancer_class.as_ref()) - .map(String::as_str) - .unwrap_or(consts::ROBOTLB_LB_CLASS); + .map_or(consts::ROBOTLB_LB_CLASS, String::as_str); if lb_type != consts::ROBOTLB_LB_CLASS { tracing::debug!("Load balancer class is not robotlb. Skipping..."); return Err(RobotLBError::SkipService); @@ -544,10 +542,11 @@ pub async fn reconcile_load_balancer( svc: Arc, context: Arc, ) -> RobotLBResult { - let mut node_ip_type = "InternalIP"; - if lb.network_name.is_none() { - node_ip_type = "ExternalIP"; - } + let node_ip_type = if lb.network_name.is_none() { + "ExternalIP" + } else { + "InternalIP" + }; let nodes = match node_source(&svc, context.config.dynamic_node_selector) { NodeSource::Annotation => get_nodes_by_selector(&svc, &context).await?, @@ -603,7 +602,7 @@ pub async fn reconcile_load_balancer( "ip": ipv4, "dns": dns_ipv4, "ip_mode": "VIP" - })) + })); } if context.config.ipv6_ingress { if let Some(ipv6) = &ipv6 { @@ -611,7 +610,7 @@ pub async fn reconcile_load_balancer( "ip": ipv6, "dns": dns_ipv6, "ip_mode": "VIP" - })) + })); } } diff --git a/tutorial.md b/tutorial.md index 37974f5..36d6ced 100644 --- a/tutorial.md +++ b/tutorial.md @@ -6,6 +6,7 @@ On each server, I will be using the latest Ubuntu LTS release which is `24.04`. We start with deploying control-plane nodes. Those nodes won't be serving any workloads, but only responsible for managing internal Kubernetes jobs. +## Cloud side ### Cloud servers @@ -16,7 +17,6 @@ You can buy 5 CAX11s which should be sufficient for small and medium-sized clust ### Setup users - As an additional layer of security, you can create a user called k3s on all servers which will be parts of the cluster. ```bash @@ -39,8 +39,7 @@ After the installation is complete, remove the k3s from sudoers. sed -i '/^k3s ALL.*/d' /etc/sudoers ``` - -### Network. +### Network Once servers have been purchased, let's create a private network. To do so, go to a network tab and create a new network in the same zone as your servers with the subnet `10.10.0.0/16`. This subnet can be anything you want, but if you want to go with any other subnet, please make sure that it doesn't overlap with `--cluster-cidr` (used to give IPs to pods) or `--service-cidr` (used to provide IPs for services). For k3s these values can be found here: https://docs.k3s.io/cli/server#networking. @@ -88,7 +87,6 @@ $ ip addr As you can see, the address `10.10.1.2/32` is assigned to the interface `enp7s0`. Now let's create a config that will suit our needs. First, we need public and private keys for the server. Run it on the server. - ```bash wg genkey | tee privatekey | wg pubkey > publickey ``` @@ -237,7 +235,6 @@ Important things to note here: * TLS san should include load-balancer IP, otherwise your cert will be rejected. * Make sure to use the correct flannel interface for inter-node communication. - ```bash #!/bin/bash # SSH user to use during the installation. You can go with root if you skipped creating a user. @@ -303,7 +300,6 @@ join_server "10.10.1.2" "10.10.1.3" After running this script you should see that control planes became healthy on our load-balancer at `http://10.10.1.4`. Also, in your generated kubeconfig file change the address of the server to the load-balancer's address. So it will become: - ```yaml ... server: https://10.10.1.4:6443 @@ -325,17 +321,15 @@ To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. You should see that all IPs are pointing to a load-balancer. -# Robot side +## Robot side Now let's create agent nodes on the Hetzner robot. I will buy 2 servers in the same region as our cloud servers. After they are ready, create a `vSwitch` and connect robot servers to the switch. But before setting up all the virtual interfaces, create a `vSwitch` type subnet in our cloud network. - ### vSwitch backbone - To do so, go to a cloud `network console` > `subnets` > `add subnet` and select a subnet range that you think will be sufficient for all your robot servers. Keep in mind that you can add only one `vSwitch` to a cloud network. I would go with `10.10.192.0/18`, because I want to also host dedicated servers for other purposes, like databases and other things. After this is ready, you need to expose routes to the `vSwitch` by clicking on three dots after the `vSwitch` subnet and choosing the appropriate option in the dropdown.