Skip to content

LCORE-2035: Add logic for tunnel and interception proxies on konflux #1758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
radofuchs merged 6 commits into from
May 21, 2026
Merged

LCORE-2035: Add logic for tunnel and interception proxies on konflux #1758

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8a5fce6
add tunnel proxy and interception proxy servers to konflux tests
May 18, 2026
3609d25
reenable tests
May 18, 2026
176e358
fix
May 18, 2026
0757895
fix tests and add flaky tags
May 20, 2026
1478b0f
use constants instead of magic numbers for pport numbers
May 21, 2026
1992e24
fix
May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions tests/e2e-prow/rhoai/manifests/lightspeed/e2e-interception-proxy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# In-cluster TLS-intercepting proxy for proxy.feature (Konflux / Prow).
# Llama Stack run.yaml points at http://e2e-interception-proxy.<ns>.svc.cluster.local:8889
apiVersion: v1
kind: Pod
metadata:
name: e2e-interception-proxy
labels:
app: e2e-interception-proxy
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: e2e-interception-proxy
image: python:3.12-slim
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
workingDir: /app
env:
- name: HOME
value: /tmp
- name: PYTHONPATH
value: /app:/tmp/pydeps
command:
- /bin/sh
- -c
- |
set -e
pip install --quiet --no-cache-dir --target /tmp/pydeps 'trustme>=1.2.1'
exec python /app/interception_proxy.py
Comment on lines +35 to +37
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify runtime dependency fetch is in startup command and identify where trustme is used.
rg -n "pip install|trustme" tests/e2e-prow/rhoai/manifests/lightspeed/e2e-interception-proxy.yaml tests/e2e/proxy/interception_proxy.py

Repository: lightspeed-core/lightspeed-stack

Length of output: 892

Avoid runtime package installation in pod startup path.

The pip install trustme command in the pod startup (line 36) introduces an external network dependency during pod initialization, which can cause flaky or slow e2e runs. Since trustme is required by the interception proxy logic (tests/e2e/proxy/interception_proxy.py imports and uses it), prebuild the container image with this dependency included so pod startup is deterministic and does not depend on external package indices.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e-prow/rhoai/manifests/lightspeed/e2e-interception-proxy.yaml` around
lines 35 - 37, Remove the runtime pip install from the pod startup and instead
bake the trustme dependency into the container image used by the interception
proxy; specifically, eliminate the "pip install --quiet --no-cache-dir --target
/tmp/pydeps 'trustme>=1.2.1'" line in the startup snippet that ultimately execs
python /app/interception_proxy.py and update the Dockerfile or image build
pipeline that produces the image used by tests/e2e-proxy to pip-install trustme
(or add it to requirements) so tests/e2e/proxy/interception_proxy.py can import
trustme at runtime without network access.

ports:
- containerPort: 8889
name: proxy
- containerPort: 8886
name: stats
volumeMounts:
- name: proxy-scripts
mountPath: /app
readOnly: true
readinessProbe:
httpGet:
path: /stats
port: stats
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /stats
port: stats
initialDelaySeconds: 10
periodSeconds: 15
volumes:
- name: proxy-scripts
configMap:
name: e2e-interception-proxy-script
---
apiVersion: v1
kind: Service
metadata:
name: e2e-interception-proxy
spec:
selector:
app: e2e-interception-proxy
ports:
- name: proxy
port: 8889
targetPort: proxy
- name: stats
port: 8886
targetPort: stats
69 changes: 69 additions & 0 deletions tests/e2e-prow/rhoai/manifests/lightspeed/e2e-tunnel-proxy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# In-cluster HTTP CONNECT tunnel proxy for proxy.feature (Konflux / Prow).
# Llama Stack run.yaml points at http://e2e-tunnel-proxy.<ns>.svc.cluster.local:8888
apiVersion: v1
kind: Pod
metadata:
name: e2e-tunnel-proxy
labels:
app: e2e-tunnel-proxy
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: e2e-tunnel-proxy
image: python:3.12-slim
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
workingDir: /app
env:
- name: PYTHONPATH
value: /app
command: ["python", "/app/tunnel_proxy.py"]
ports:
- containerPort: 8888
name: proxy
- containerPort: 8887
name: stats
volumeMounts:
- name: proxy-scripts
mountPath: /app
readOnly: true
readinessProbe:
httpGet:
path: /stats
port: stats
initialDelaySeconds: 2
periodSeconds: 5
livenessProbe:
httpGet:
path: /stats
port: stats
initialDelaySeconds: 5
periodSeconds: 15
volumes:
- name: proxy-scripts
configMap:
name: e2e-tunnel-proxy-script
---
apiVersion: v1
kind: Service
metadata:
name: e2e-tunnel-proxy
spec:
selector:
app: e2e-tunnel-proxy
ports:
- name: proxy
port: 8888
targetPort: proxy
- name: stats
port: 8887
targetPort: stats
9 changes: 9 additions & 0 deletions tests/e2e-prow/rhoai/manifests/lightspeed/llama-stack-openai.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,11 @@ spec:
- name: rag-data
mountPath: /opt/app-root/rag-data-cm
readOnly: true
# proxy.feature (interception): PEM from Secret e2e-interception-proxy-ca (optional).
- name: interception-proxy-ca
mountPath: /tmp/interception-proxy-ca.pem
subPath: ca.pem
readOnly: true
Comment thread
radofuchs marked this conversation as resolved.
volumes:
- name: app-root
emptyDir: {}
Expand All @@ -213,3 +218,7 @@ spec:
- name: rag-data
configMap:
name: rag-data
- name: interception-proxy-ca
secret:
secretName: e2e-interception-proxy-ca
optional: true
3 changes: 3 additions & 0 deletions tests/e2e-prow/rhoai/pipeline-konflux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,9 @@ oc wait pod/mock-jwks pod/mock-mcp \
}
log "✅ Mock servers deployed"

# e2e-tunnel-proxy and e2e-interception-proxy are deployed from proxy.feature steps
# (see tests/e2e/features/steps/proxy.py + e2e-ops deploy-e2e-*-proxy).

#========================================
# 5. DEPLOY LIGHTSPEED STACK AND LLAMA STACK
#========================================
Expand Down
161 changes: 159 additions & 2 deletions tests/e2e-prow/rhoai/scripts/e2e-ops.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
# update-configmap <name> <file> - Update ConfigMap from file
# get-configmap-content <name> - Get ConfigMap content (outputs to stdout)
# disrupt-llama-stack - Delete llama-stack pod to disrupt connection
# deploy-e2e-tunnel-proxy - Deploy in-cluster tunnel proxy (proxy.feature step)
# deploy-e2e-interception-proxy - Deploy in-cluster interception proxy (proxy.feature step)

set -e

Expand Down Expand Up @@ -340,16 +342,30 @@ cmd_restart_llama_stack() {

echo "Applying pod manifest..."
if [[ "${E2E_KONFLUX_E2E:-0}" == "1" ]]; then
# Interception-proxy e2e: refresh Secret before pod recreate so the volume mount is populated.
if [[ "${E2E_COPY_INTERCEPTION_CA_TO_LLAMA:-0}" == "1" ]]; then
echo "[e2e-ops] Syncing e2e-interception-proxy-ca secret before llama-stack apply..."
if ! cmd_sync_interception_proxy_ca_secret; then
echo "===== Llama-stack restore FAILED (interception CA secret sync) ====="
exit 1
fi
fi
_LLAMA_SVC_FQDN="llama-stack-service-svc.${NAMESPACE}.svc.cluster.local"
oc create secret generic llama-stack-ip-secret \
--from-literal=key="$_LLAMA_SVC_FQDN" \
-n "$NAMESPACE" \
--dry-run=client -o yaml | oc apply -f -
oc apply -n "$NAMESPACE" -f "$MANIFEST_DIR/llama-stack-openai.yaml"
wait_for_pod "llama-stack-service" 60
wait_for_pod "llama-stack-service" 90
echo "Labeling pod for service..."
oc label pod llama-stack-service pod=llama-stack-service -n "$NAMESPACE" --overwrite
if ! wait_for_llama_stack_http_health 35; then
if [[ "${E2E_COPY_INTERCEPTION_CA_TO_LLAMA:-0}" == "1" ]]; then
if ! _verify_interception_ca_mounted_in_llama; then
echo "===== Llama-stack restore FAILED (interception CA not mounted) ====="
exit 1
fi
fi
if ! wait_for_llama_stack_http_health 50; then
echo "===== Llama-stack restore FAILED (HTTP not healthy) ====="
exit 1
fi
Expand Down Expand Up @@ -612,6 +628,123 @@ cmd_get_configmap_content() {
-o "go-template={{index .data \"$configmap_key\"}}"
}

cmd_tunnel_proxy_stats() {
local pod_name
pod_name=$(oc get pod -n "$NAMESPACE" -l app=e2e-tunnel-proxy \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || pod_name=""

if [[ -z "$pod_name" ]]; then
echo "ERROR: no e2e-tunnel-proxy pod in namespace $NAMESPACE" >&2
return 1
fi

oc exec -n "$NAMESPACE" "$pod_name" -- \
python3 -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8887/stats', timeout=5).read().decode())"
}

cmd_interception_proxy_stats() {
local pod_name
pod_name=$(oc get pod -n "$NAMESPACE" -l app=e2e-interception-proxy \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || pod_name=""

if [[ -z "$pod_name" ]]; then
echo "ERROR: no e2e-interception-proxy pod in namespace $NAMESPACE" >&2
return 1
fi

oc exec -n "$NAMESPACE" "$pod_name" -- \
python3 -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8886/stats', timeout=5).read().decode())"
}

cmd_sync_interception_proxy_ca_secret() {
local proxy_pod_name tmp
proxy_pod_name=$(oc get pod -n "$NAMESPACE" -l app=e2e-interception-proxy \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || proxy_pod_name=""

if [[ -z "$proxy_pod_name" ]]; then
echo "ERROR: no e2e-interception-proxy pod in namespace $NAMESPACE" >&2
return 1
fi

tmp=$(mktemp)
if ! oc exec -n "$NAMESPACE" "$proxy_pod_name" -- \
cat /tmp/interception-proxy-ca.pem >"$tmp"; then
rm -f "$tmp"
echo "ERROR: failed to read CA from e2e-interception-proxy pod" >&2
return 1
fi
if [[ ! -s "$tmp" ]]; then
rm -f "$tmp"
echo "ERROR: interception-proxy CA PEM is empty" >&2
return 1
fi

if ! oc create secret generic e2e-interception-proxy-ca \
--from-file=ca.pem="$tmp" \
-n "$NAMESPACE" \
--dry-run=client -o yaml | oc apply -n "$NAMESPACE" -f -; then
rm -f "$tmp"
echo "ERROR: failed to apply e2e-interception-proxy-ca secret" >&2
return 1
fi
rm -f "$tmp"
echo "✓ Secret e2e-interception-proxy-ca updated (ca.pem)"
}

_verify_interception_ca_mounted_in_llama() {
local llama_pod_name="llama-stack-service"
if oc exec -n "$NAMESPACE" "$llama_pod_name" -c llama-stack-container -- \
test -s /tmp/interception-proxy-ca.pem; then
echo "✓ interception-proxy CA present at /tmp/interception-proxy-ca.pem in llama-stack"
return 0
fi
echo "ERROR: /tmp/interception-proxy-ca.pem missing or empty in llama-stack pod" >&2
oc exec -n "$NAMESPACE" "$llama_pod_name" -c llama-stack-container -- \
ls -la /tmp/interception-proxy-ca.pem 2>&1 || true
return 1
}

cmd_copy_interception_proxy_ca_to_llama() {
# Legacy name: publish CA via Secret (mounted by llama-stack-openai.yaml).
cmd_sync_interception_proxy_ca_secret
}

_e2e_repo_root() {
cd "$SCRIPT_DIR/../../../.." && pwd
}

cmd_deploy_e2e_tunnel_proxy() {
local repo_root
repo_root="$(_e2e_repo_root)"
echo "Deploying e2e-tunnel-proxy in namespace $NAMESPACE..."
oc create configmap e2e-tunnel-proxy-script -n "$NAMESPACE" \
--from-file=tunnel_proxy.py="$repo_root/tests/e2e/proxy/tunnel_proxy.py" \
--dry-run=client -o yaml | oc apply -f -
oc apply -n "$NAMESPACE" -f "$MANIFEST_DIR/e2e-tunnel-proxy.yaml"
if ! oc wait pod/e2e-tunnel-proxy -n "$NAMESPACE" --for=condition=Ready --timeout=120s; then
echo "ERROR: e2e-tunnel-proxy failed to become ready" >&2
oc describe pod e2e-tunnel-proxy -n "$NAMESPACE" 2>/dev/null | tail -25 || true
return 1
fi
echo "✓ e2e-tunnel-proxy ready at http://e2e-tunnel-proxy.${NAMESPACE}.svc.cluster.local:8888"
}

cmd_deploy_e2e_interception_proxy() {
local repo_root
repo_root="$(_e2e_repo_root)"
echo "Deploying e2e-interception-proxy in namespace $NAMESPACE..."
oc create configmap e2e-interception-proxy-script -n "$NAMESPACE" \
--from-file=interception_proxy.py="$repo_root/tests/e2e/proxy/interception_proxy.py" \
--dry-run=client -o yaml | oc apply -f -
oc apply -n "$NAMESPACE" -f "$MANIFEST_DIR/e2e-interception-proxy.yaml"
if ! oc wait pod/e2e-interception-proxy -n "$NAMESPACE" --for=condition=Ready --timeout=180s; then
echo "ERROR: e2e-interception-proxy failed to become ready" >&2
oc describe pod e2e-interception-proxy -n "$NAMESPACE" 2>/dev/null | tail -25 || true
return 1
fi
echo "✓ e2e-interception-proxy ready at http://e2e-interception-proxy.${NAMESPACE}.svc.cluster.local:8889"
}

cmd_disrupt_llama_stack() {
local pod_name="llama-stack-service"

Expand Down Expand Up @@ -664,6 +797,24 @@ case "$COMMAND" in
disrupt-llama-stack)
cmd_disrupt_llama_stack
;;
tunnel-proxy-stats)
cmd_tunnel_proxy_stats
;;
interception-proxy-stats)
cmd_interception_proxy_stats
;;
copy-interception-proxy-ca-to-llama)
cmd_copy_interception_proxy_ca_to_llama
;;
sync-interception-proxy-ca-secret)
cmd_sync_interception_proxy_ca_secret
;;
deploy-e2e-tunnel-proxy)
cmd_deploy_e2e_tunnel_proxy
;;
deploy-e2e-interception-proxy)
cmd_deploy_e2e_interception_proxy
;;
*)
echo "Usage: $0 <command> [args...]"
echo ""
Expand All @@ -676,6 +827,12 @@ case "$COMMAND" in
echo " update-configmap <name> <file> - Update ConfigMap from file"
echo " get-configmap-content <name> - Get ConfigMap content (outputs to stdout)"
echo " disrupt-llama-stack - Delete llama-stack pod to disrupt connection"
echo " tunnel-proxy-stats - JSON stats from in-cluster e2e-tunnel-proxy"
echo " interception-proxy-stats - JSON stats from in-cluster e2e-interception-proxy"
echo " copy-interception-proxy-ca-to-llama - Alias for sync-interception-proxy-ca-secret"
echo " sync-interception-proxy-ca-secret - Publish trustme CA to Secret for llama mount"
echo " deploy-e2e-tunnel-proxy - Deploy in-cluster tunnel proxy pod"
echo " deploy-e2e-interception-proxy - Deploy in-cluster interception proxy pod"
exit 1
;;
esac
Loading
Loading