Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/helm-install-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ jobs:
--set externalRedis.passwordKey=redis-password \
--set admin.enabled=true \
--set admin.secret=test-admin-secret-for-ci \
--set frontproxy.enabled=true \
--set replicaCount=3 \
--set resources.limits.cpu=100m \
--set resources.requests.cpu=50m \
Expand All @@ -173,6 +174,35 @@ jobs:
fi
echo "✓ All 3 s3proxy pods running"

- name: Verify front proxy
run: |
set -euo pipefail
echo "=== Front proxy pods ready ==="
kubectl wait --for=condition=ready pod \
-l app.kubernetes.io/component=frontproxy -n s3proxy --timeout=120s
kubectl get pods -n s3proxy -l app.kubernetes.io/component=frontproxy

echo "=== S3 round-trip THROUGH the front proxy ==="
kubectl run frontproxy-test -n s3proxy --rm -i --restart=Never \
--image=amazon/aws-cli:latest \
--env="AWS_ACCESS_KEY_ID=minioadmin" \
--env="AWS_SECRET_ACCESS_KEY=minioadmin" \
--env="AWS_DEFAULT_REGION=us-east-1" \
--command -- /bin/sh -c '
set -e
ENDPOINT="http://s3proxy-python-frontproxy:80"
echo "via front proxy - $(date)" > /tmp/fp.txt
ORIG=$(md5sum /tmp/fp.txt | cut -c1-32)
aws --endpoint-url $ENDPOINT s3 mb s3://frontproxy-bucket
aws --endpoint-url $ENDPOINT s3 cp /tmp/fp.txt s3://frontproxy-bucket/fp.txt
aws --endpoint-url $ENDPOINT s3 cp s3://frontproxy-bucket/fp.txt /tmp/down.txt
DOWN=$(md5sum /tmp/down.txt | cut -c1-32)
[ "$ORIG" = "$DOWN" ] || { echo "✗ round-trip via front proxy failed"; exit 1; }
aws --endpoint-url $ENDPOINT s3 rm s3://frontproxy-bucket/fp.txt
aws --endpoint-url $ENDPOINT s3 rb s3://frontproxy-bucket
echo "✓ S3 round-trip through front proxy succeeded"
'

- name: Check health endpoint
run: |
kubectl port-forward svc/s3proxy-python 4433:4433 -n s3proxy &
Expand Down Expand Up @@ -305,6 +335,9 @@ jobs:
echo "=== S3Proxy Logs ==="
kubectl logs -l app.kubernetes.io/name=s3proxy-python -n s3proxy --tail=100
echo ""
echo "=== Front Proxy Logs ==="
kubectl logs -l app.kubernetes.io/component=frontproxy -n s3proxy --tail=100
echo ""
echo "=== MinIO Logs ==="
kubectl logs -l app=minio -n s3proxy --tail=50
echo ""
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ aws s3 --endpoint-url http://s3proxy-python:4433 cp file.txt s3://bucket/

Use the **same credentials** you configured above. That's it.

> **Endpoints** — In-cluster: `http://s3proxy-python.<ns>:4433` · Gateway: `http://s3-gateway.<ns>` · Ingress: `https://s3proxy.example.com`
> **Endpoints** — In-cluster: `http://s3proxy-python.<ns>:4433` · Front proxy (even load distribution, `frontproxy.enabled=true`): `http://s3proxy-python-frontproxy.<ns>` · Outside the cluster: enable `ingress` in front of the front proxy
>
> **Health** — `GET /healthz` · `GET /readyz` · **Metrics** — `GET /metrics`

Expand Down Expand Up @@ -131,8 +131,8 @@ A request signed by an access key with no configured KEK is rejected. Via Helm:
| `secrets.existingSecrets.enabled` | `false` | Use existing K8s secret |
| `admin.secret` | `change-me` | Secret signing admin session cookies (when admin UI on) |
| `redis-ha.enabled` | `true` | Deploy embedded Redis HA |
| `gateway.enabled` | `false` | Create gateway service |
| `ingress.enabled` | `false` | Enable ingress |
| `frontproxy.enabled` | `false` | Bundled HAProxy for even load distribution (no external dependency) |
| `ingress.enabled` | `false` | Expose S3 outside the cluster via Ingress (requires `frontproxy.enabled`) |
| `performance.memoryLimitMb` | `64` | Memory budget for streaming concurrency |

See [chart/README.md](chart/README.md) for all options.
Expand Down
62 changes: 54 additions & 8 deletions chart/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ helm install s3proxy oci://ghcr.io/serversidehannes/s3proxy-python/charts/s3prox
| `admin.existingSecret.passwordKey` | `S3PROXY_ADMIN_PASSWORD` | Password key in the existing secret |
| `admin.existingSecret.secretKey` | `S3PROXY_ADMIN_SECRET` | Session-secret key in the existing secret |
| `admin.ingress.enabled` | `false` | Dedicated Ingress for the dashboard (keep off unless intentionally exposing it) |
| `admin.ingress.className` | `nginx` | Ingress class for the admin Ingress |
| `admin.ingress.className` | `""` | Ingress class for the admin Ingress (set to your cluster's controller) |
| `admin.ingress.host` | `""` | Hostname for the dashboard (required when enabled) |
| `admin.ingress.annotations` | `{}` | Annotations (e.g. IP allowlist) for the admin Ingress |
| `admin.ingress.tls` | `[]` | TLS config for the admin Ingress |
Expand All @@ -53,14 +53,22 @@ helm install s3proxy oci://ghcr.io/serversidehannes/s3proxy-python/charts/s3prox
| `externalRedis.passwordKey` | `redis-password` | Key name in Redis secret |
| `service.type` | `ClusterIP` | Service type |
| `service.port` | `4433` | Service port |
| `ingress.enabled` | `false` | Enable ingress |
| `ingress.className` | `nginx` | Ingress class |
| `ingress.annotations` | nginx streaming defaults | Ingress annotations |
| `ingress.hosts` | `[]` | Ingress hostnames |
| `frontproxy.enabled` | `false` | Deploy a bundled HAProxy front proxy for even load distribution (see below) |
| `frontproxy.replicaCount` | `2` | Front proxy replicas |
| `frontproxy.image.repository` | `haproxy` | Front proxy image |
| `frontproxy.image.tag` | `3.0-alpine` | Front proxy image tag |
| `frontproxy.service.type` | `ClusterIP` | Front proxy service type |
| `frontproxy.service.port` | `80` | Front proxy service port (clients connect here) |
| `frontproxy.timeouts.client` | `1h` | Client-side timeout (tolerate large transfers) |
| `frontproxy.timeouts.server` | `1h` | Backend timeout (tolerate large transfers) |
| `frontproxy.timeouts.connect` | `10s` | Backend connect timeout |
| `frontproxy.podDisruptionBudget.enabled` | `true` | Enable front proxy PDB |
| `frontproxy.podDisruptionBudget.minAvailable` | `1` | Min available front proxy pods |
| `ingress.enabled` | `false` | Expose S3 outside the cluster via Ingress (requires `frontproxy.enabled`) |
| `ingress.className` | `""` | Ingress class (set to your cluster's controller) |
| `ingress.annotations` | `{}` | Ingress annotations (controller-specific tuning) |
| `ingress.hosts` | `[]` | Ingress host/path rules |
| `ingress.tls` | `[]` | Ingress TLS config |
| `gateway.enabled` | `false` | ExternalName gateway service |
| `gateway.serviceName` | `s3-gateway` | Gateway service name |
| `gateway.ingressService` | `ingress-nginx-controller...` | Target ingress service |
| `resources.requests.cpu` | `100m` | CPU request |
| `resources.requests.memory` | `512Mi` | Memory request |
| `resources.limits.cpu` | `500m` | CPU limit |
Expand All @@ -71,3 +79,41 @@ helm install s3proxy oci://ghcr.io/serversidehannes/s3proxy-python/charts/s3prox
| `topologySpreadConstraints` | `[]` | Topology spread |
| `podDisruptionBudget.enabled` | `true` | Enable PDB |
| `podDisruptionBudget.minAvailable` | `1` | Min available pods |

## Even load distribution (front proxy)

A plain `ClusterIP` Service balances per **connection** (L4, via kube-proxy). S3
clients (boto3, aws-cli) hold long-lived keep-alive connections, so each client
tends to pin to a single pod — leaving load uneven across replicas.

Setting `frontproxy.enabled=true` deploys a small bundled HAProxy in front of the
s3proxy pods that balances per **request** (L7), spreading even a single client's
stream across every pod. It is fully self-contained — it requires **no** ingress
controller, service mesh, or anything else in the cluster.

```bash
helm install s3proxy ... --set frontproxy.enabled=true
```

Clients then connect to the front proxy Service (`s3proxy-python-frontproxy`)
instead of the s3proxy Service. The front proxy runs `replicaCount: 2` with a PDB so
it is not a single point of failure.

### Exposing S3 outside the cluster

For S3 operations from **outside** the cluster, enable an Ingress in front of the
front proxy. It is controller-agnostic — set `ingress.className` to whatever ingress
controller you run and add any controller-specific tuning via `ingress.annotations`.

```bash
helm install s3proxy ... \
--set frontproxy.enabled=true \
--set ingress.enabled=true \
--set ingress.className=nginx \
--set 'ingress.hosts[0].host=s3.example.com' \
--set 'ingress.hosts[0].paths[0].path=/' \
--set 'ingress.hosts[0].paths[0].pathType=Prefix'
```

The Ingress routes to the front proxy, so external clients also get even per-request
distribution. `ingress.enabled` therefore requires `frontproxy.enabled=true`.
1 change: 1 addition & 0 deletions chart/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ spec:
metadata:
labels:
{{- include "s3proxy.labels" . | nindent 8 }}
app.kubernetes.io/component: server
spec:
containers:
- name: {{ .Chart.Name }}
Expand Down
39 changes: 39 additions & 0 deletions chart/templates/frontproxy-configmap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{{- if .Values.frontproxy.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Chart.Name }}-frontproxy
labels:
{{- include "s3proxy.labels" . | nindent 4 }}
app.kubernetes.io/component: frontproxy
data:
haproxy.cfg: |
global
log stdout format raw local0
maxconn 4096

defaults
mode http
log global
option httplog
option dontlognull
timeout connect {{ .Values.frontproxy.timeouts.connect }}
timeout client {{ .Values.frontproxy.timeouts.client }}
timeout server {{ .Values.frontproxy.timeouts.server }}

frontend http_in
bind *:{{ .Values.frontproxy.service.port }}
default_backend s3proxy_pods

backend s3proxy_pods
balance leastconn
# Per-request balancing: don't pin a client to one server, and re-resolve the
# headless Service via DNS so new/restarted pods are picked up automatically.
option redispatch
server-template pod 1-{{ int .Values.replicaCount }} {{ .Chart.Name }}-headless:{{ .Values.service.port }} check resolvers k8s init-addr none

resolvers k8s
parse-resolv-conf
accepted_payload_size 8192
hold valid 5s
{{- end }}
68 changes: 68 additions & 0 deletions chart/templates/frontproxy-deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
{{- if .Values.frontproxy.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Chart.Name }}-frontproxy
labels:
{{- include "s3proxy.labels" . | nindent 4 }}
app.kubernetes.io/component: frontproxy
spec:
replicas: {{ .Values.frontproxy.replicaCount }}
selector:
matchLabels:
{{- include "s3proxy.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: frontproxy
template:
metadata:
labels:
{{- include "s3proxy.labels" . | nindent 8 }}
app.kubernetes.io/component: frontproxy
annotations:
checksum/config: {{ include (print $.Template.BasePath "/frontproxy-configmap.yaml") . | sha256sum }}
spec:
containers:
- name: haproxy
image: "{{ .Values.frontproxy.image.repository }}:{{ .Values.frontproxy.image.tag }}"
imagePullPolicy: {{ .Values.frontproxy.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.frontproxy.service.port }}
protocol: TCP
volumeMounts:
- name: config
mountPath: /usr/local/etc/haproxy/haproxy.cfg
subPath: haproxy.cfg
readOnly: true
livenessProbe:
tcpSocket:
port: http
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
tcpSocket:
port: http
initialDelaySeconds: 2
periodSeconds: 5
resources:
{{- toYaml .Values.frontproxy.resources | nindent 12 }}
securityContext:
runAsNonRoot: true
runAsUser: 99
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumes:
- name: config
configMap:
name: {{ .Chart.Name }}-frontproxy
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
{{- include "s3proxy.selectorLabels" . | nindent 14 }}
app.kubernetes.io/component: frontproxy
{{- end }}
15 changes: 15 additions & 0 deletions chart/templates/frontproxy-pdb.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{{- if and .Values.frontproxy.enabled .Values.frontproxy.podDisruptionBudget.enabled }}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ .Chart.Name }}-frontproxy
labels:
{{- include "s3proxy.labels" . | nindent 4 }}
app.kubernetes.io/component: frontproxy
spec:
minAvailable: {{ .Values.frontproxy.podDisruptionBudget.minAvailable }}
selector:
matchLabels:
{{- include "s3proxy.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: frontproxy
{{- end }}
19 changes: 19 additions & 0 deletions chart/templates/frontproxy-service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{{- if .Values.frontproxy.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ .Chart.Name }}-frontproxy
labels:
{{- include "s3proxy.labels" . | nindent 4 }}
app.kubernetes.io/component: frontproxy
spec:
type: {{ .Values.frontproxy.service.type }}
selector:
{{- include "s3proxy.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: frontproxy
ports:
- name: http
port: {{ .Values.frontproxy.service.port }}
targetPort: http
protocol: TCP
{{- end }}
13 changes: 0 additions & 13 deletions chart/templates/gateway-service.yaml

This file was deleted.

32 changes: 13 additions & 19 deletions chart/templates/ingress.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
{{- if .Values.ingress.enabled -}}
{{- if not .Values.frontproxy.enabled }}
{{- fail "ingress.enabled requires frontproxy.enabled=true — the ingress routes to the front proxy" }}
{{- end }}
{{- if not .Values.ingress.hosts }}
{{- fail "ingress.enabled is true but ingress.hosts is empty — set at least one host" }}
{{- end }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
Expand All @@ -10,10 +16,12 @@ metadata:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
ingressClassName: {{ .Values.ingress.className }}
{{- if .Values.ingress.tls }}
{{- with .Values.ingress.className }}
ingressClassName: {{ . }}
{{- end }}
{{- with .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
{{- range . }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
Expand All @@ -22,19 +30,6 @@ spec:
{{- end }}
{{- end }}
rules:
{{- if and .Values.gateway.enabled (not .Values.ingress.hosts) }}
# Auto-generate host from gateway service name + namespace
- host: {{ printf "%s.%s" .Values.gateway.serviceName .Release.Namespace | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ .Chart.Name }}
port:
number: {{ .Values.service.port }}
{{- else }}
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
Expand All @@ -44,10 +39,9 @@ spec:
pathType: {{ .pathType }}
backend:
service:
name: {{ $.Chart.Name }}
name: {{ $.Chart.Name }}-frontproxy
port:
number: {{ $.Values.service.port }}
number: {{ $.Values.frontproxy.service.port }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
Loading