AWS eks documentation change
Summary
Updated vLLM deployment parameters, Grafana access instructions, Open WebUI configuration, and port-forwarding steps
Security assessment
Added explicit Grafana password retrieval via Kubernetes secret improves security documentation by showing proper secret access. No evidence of vulnerability fixes; changes appear operational (tuning max-num-seqs) or instructional.
Diff
diff --git a/eks/latest/userguide/ml-inference-load-serve-model.md b/eks/latest/userguide/ml-inference-load-serve-model.md index 2362d8539..1e4d7f54e 100644 --- a//eks/latest/userguide/ml-inference-load-serve-model.md +++ b//eks/latest/userguide/ml-inference-load-serve-model.md @@ -174,4 +174 @@ This section uses [AWS Deep Learning Containers](https://github.com/aws/deep-lea -This deployment uses the following AWS DLC for [vLLM 0.21.0](https://gallery.ecr.aws/deep-learning-containers/vllm) with SOCI support: - - - public.ecr.aws/deep-learning-containers/vllm:0.21.0-gpu-py312-cu130-ubuntu22.04-ec2-v1.0-soci +This deployment uses the following AWS DLC for [vLLM 0.21.0](https://gallery.ecr.aws/deep-learning-containers/vllm) with SOCI support: `public.ecr.aws/deep-learning-containers/vllm:0.21.0-gpu-py312-cu130-ubuntu22.04-ec2-v1.0-soci`. @@ -223 +220 @@ Apply the manifest: - - "--max-num-seqs=1" + - "--max-num-seqs=128" @@ -403 +400,6 @@ To access Grafana, start a port-forward to the Grafana service: -Open [http://localhost:3000](http://localhost:3000) in your browser and navigate to **Dashboards > GPU Monitoring > vLLM Metrics**. +Open [http://localhost:3000](http://localhost:3000) in your browser and log in with username `admin` and the password from the following command: + + + kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo + +Navigate to **Dashboards > GPU Monitoring > vLLM Metrics**. @@ -460,0 +463,2 @@ To deploy the Open WebUI application, apply the following manifest: + - name: RAG_EMBEDDING_ENGINE + value: "" @@ -495 +499 @@ Expected output: -To access the application, set up port forwarding and open the application in your browser: +To access the application, set up port forwarding: @@ -498,3 +502 @@ To access the application, set up port forwarding and open the application in yo - kubectl port-forward svc/open-webui 8080:80 & - sleep 5 - echo "Open WebUI: http://localhost:8080" + kubectl port-forward svc/open-webui 8080:80 @@ -506 +508 @@ The chat interface appears where you can interact with the Ministral model. -When you finish testing, stop the backgrounded port-forward processes by running `kill %1 %2` (or run `jobs` to list them and `kill %<jobspec>` for each). +When you finish testing, stop the port-forward with kbd:[Ctrl+C]. @@ -532 +534 @@ Inference -Cluster configuration +Autoscaling