Kfserving Deployment
Environment
-
Istio v1.9.3 -
Knative v0.20.0 -
Cert Manager v1.3.0 -
kfserving v0.5.1 -
install istioctl v1
wget https://github.com/istio/istio/releases/download/1.9.3/istio-1.9.3-linux-amd64.tar.gz
tar xf istio-1.9.3-linux-amd64.tar.gz
cd istio-1.9.3/
cp bin/istioctl /usr/local/bin
Serving Component
1、Install the Custom Resource Definitions(aka CRDs):
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.20.0/serving-crds.yaml
2、Install the core components of Serving (see below for optional extensions)
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.20.0/serving-core.yaml
3、Pick a networking layer:Istio
cat << EOF > ./istio-minimal-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
proxy:
autoInject: disabled
useMCP: false
jwtPolicy: first-party-jwt
addonComponents:
pilot:
enabled: true
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
EOF
istioctl install -f istio-minimal-operator.yaml
kubectl label namespace knative-serving istio-injection=enabled
- Set
PeerAuthentication to PERMISSIVE on knative-serving system namespace.
cat <<EOF | kubectl apply -f -
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
name: "default"
namespace: "knative-serving"
spec:
mtls:
mode: PERMISSIVE
EOF
-
Updating the config-istio configmap to use a non-default local gateway
- If you create a custom service and deployment for local gateway with a name other than
cluster-local-gateway, you need to update gateway configmap config-istio under the knative-serving namespace.Edit the config-istio configmap.
kubectl edit configmap config-istio -n knative-serving
- Replace the
local-gateway.knative-serving.cluster-local-gateway field with the custom service. As an example, if you name both the service and deployment custom-local-gateway under the namespace istio-system, it should be updated to.
custom-local-gateway.istio-system.svc.cluster.local
- if both the custom service and deployment are labeled with
custom: custom-local-gateway, not the default istio: cluster-local-gateway, you must update gateway instance cluster-local-gateway in the knative-serving namespace.
kubectl edit gateway cluster-local-gateway -n knative-serving
Replace the label selector with the label of your service: istio: cluster-local-gateway
For the service above, it should be updated to: custom: custom-local-gateway
If there is a change in service ports (compared to that of cluster-local-gateway), update the port info in the gateway accordingly. -
configuring DNS
kubectl get svc -nistio-system
- This external IP can be used with your DNS provider with a wildcard
A record. However, for a basic non-production set up, this external IP address can be used with xip.io in the config-domain ConfigMap in knative-serving.You can edit this by using the following command
kubectl edit cm config-domain --namespace knative-serving
- Given the external IP above, change the content to
apiVersion: v1
kind: ConfigMap
metadata:
name: config-domain
namespace: knative-serving
data:
34.83.80.117.xip.io: ""
4、Configure DNS
option 1: Real DNS
To configure DNS for Knative, take the External IP or CNAME from setting up networking, and configure it with your DNS provider as follows:
- If the networking layer produced an External IP address, then configure a wildcard
A record for the domain:
# Here knative.example.com is the domain suffix for your cluster
*.knative.example.com == A 35.233.41.212
-
If the networking layer produced a CNAME, then configure a CNAME record for the domain: # Here knative.example.com is the domain suffix for your cluster
*.knative.example.com == CNAME a317a278525d111e89f272a164fd35fb-1510370581.eu-central-1.elb.amazonaws.com
Once your DNS provider has been configured, direct Knative to use that domain:
kubectl patch configmap/config-domain \
--namespace knative-serving \
--type merge \
--patch '{"data":{"knative.example.com":""}}'
option 2: Magic DNS
We ship a simple Kubernetes Job called “default domain” that will (see caveats) configure Knative Serving to use xip.ioas the default DNS suffix.
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.20.0/serving-default-domain.yaml
EventingComponent
5、Install the Custom Resource Definitions(aka CRDs):
kubectl apply --filename https://github.com/knative/eventing/releases/download/v0.20.0/eventing-crds.yaml
6、Install the core components of Eventing (see below for optional extensions)
kubectl apply --filename https://github.com/knative/eventing/releases/download/v0.20.0/eventing-core.yaml
7、Install a default Channel (messaging) layer (alphabetical).
Apache Kafka Channel
-
Install Apache Kafka for Kubernetes -
Create a namespace for your Apache Kafka installation, like kafka: kubectl create namespace kafka
- Install the Strimzi operator, like:
```bash
curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.2/strimzi-cluster-operator-0.16.2.yaml" \
| sed 's/namespace: .*/namespace: kafka/' \
| kubectl -n kafka apply -f -
-
Describe the size of your Apache Kafka installation in kafka.yaml, like: apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.4.0
replicas: 1
listeners:
plain: {}
tls: {}
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.4"
storage:
type: ephemeral
zookeeper:
replicas: 3
storage:
type: ephemeral
entityOperator:
topicOperator: {}
userOperator: {}
- Deploy the Apache Kafka cluster
```bash
kubectl apply -n kafka -f kafka.yaml
- install the Apache Kafka Channel:
curl -L "https://github.com/knative-sandbox/eventing-kafka/releases/download/v0.20.0/channel-consolidated.yaml" \
| sed 's/REPLACE_WITH_CLUSTER_URL/my-cluster-kafka-bootstrap.kafka:9092/' \
| kubectl apply --filename -
8、Install a Broker (eventing) layer
Apache Kafka Broker
- Install the Kafka controlle
kubectl apply --filename https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/v0.20.0/eventing-kafka-controller.yaml
- Install the Kafka Broker data plane
kubectl apply --filename https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/v0.20.0/eventing-kafka-broker.yaml
9、install cert-manager
- Install the
CustomResourceDefinitions and cert-manager itself
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.0/cert-manager.yaml
Note: When running on GKE (Google Kubernetes Engine), you may encounter a ‘permission denied’ error when creating some of these resources. This is a nuance of the way GKE handles RBAC and IAM permissions, and as such you should ‘elevate’ your own privileges to that of a ‘cluster-admin’ before running the above command. If you have already run the above command, you should run them again after elevating your permissions:
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--user=$(gcloud config get-value core/account)
10、Install KFServing CRD and Controller
kubectl apply -f https://github.com/kubeflow/kfserving/releases/download/0.5.1/kfserving.yaml
11、Setup Ingress Gateway
If the default ingress gateway setup does not fit your need, you can choose to setup a custom ingress gateway
-
Configure Custom Ingress Gateway
- Create Gateway Service and Deployment Instance，You’ll need to create the gateway service and deployment instance to handle traffic first. Let’s say you customized the default
istio-ingressgateway to custom-ingressgateway as follows.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
proxy:
autoInject: disabled
useMCP: false
jwtPolicy: first-party-jwt
addonComponents:
pilot:
enabled: true
prometheus:
enabled: false
components:
ingressGateways:
- name: custom-ingressgateway
enabled: true
namespace: custom-ns
label:
istio: custom-gateway
- name: cluster-local-gateway
enabled: true
label:
istio: cluster-local-gateway
app: cluster-local-gateway
k8s:
service:
type: ClusterIP
ports:
- port: 15020
name: status-port
- port: 80
name: http2
- port: 443
name: https
- Update Knative Gateway，Update gateway instance
knative-ingress-gateway under knative-serving namespace:
kubectl edit gateway knative-ingress-gateway -n knative-serving
Replace the label selector with the label of your service: istio: ingressgateway
For the service above, it should be updated to: istio: custom-gateway
If there is a change in service ports (compared with that of istio-ingressgateway), update the port info in the gateway accordingly
Update gateway configmap config-istio under knative-serving namespace: kubectl edit configmap config-istio -n knative-serving
Replace the istio-ingressgateway.istio-system.svc.cluster.local field with the fully qualified url of your service. gateway.knative-serving.knative-ingress-gateway: "istio-ingressgateway.istio-system.svc.cluster.local"
For the service above, it should be updated to: gateway.knative-serving.knative-ingress-gateway: custom-ingressgateway.custom-ns.svc.cluster.local
- In addition you need to update KFServing configmap to use the custom ingress gateway.
apiVersion: v1
kind: ConfigMap
metadata:
name: inferenceservice-config
namespace: kfserving-system
data:
predictors: |-
{
"tensorflow": {
"image": "tensorflow/serving",
"defaultImageVersion": "1.14.0",
"defaultGpuImageVersion": "1.14.0-gpu",
"defaultTimeout": "60",
"supportedFrameworks": [
"tensorflow"
],
"multiModelServer": false
},
"onnx": {
"image": "mcr.microsoft.com/onnxruntime/server",
"defaultImageVersion": "v1.0.0",
"supportedFrameworks": [
"onnx"
],
"multiModelServer": false
},
"sklearn": {
"v1": {
"image": "gcr.io/kfserving/sklearnserver",
"defaultImageVersion": "latest",
"supportedFrameworks": [
"sklearn"
],
"multiModelServer": true
},
"v2": {
"image": "docker.io/seldonio/mlserver",
"defaultImageVersion": "0.2.1",
"supportedFrameworks": [
"sklearn"
],
"multiModelServer": false
}
},
"xgboost": {
"v1": {
"image": "gcr.io/kfserving/xgbserver",
"defaultImageVersion": "latest",
"supportedFrameworks": [
"xgboost"
],
"multiModelServer": true
},
"v2": {
"image": "docker.io/seldonio/mlserver",
"defaultImageVersion": "0.2.1",
"supportedFrameworks": [
"xgboost"
],
"multiModelServer": false
}
},
"pytorch": {
"v1" : {
"image": "gcr.io/kfserving/pytorchserver",
"defaultImageVersion": "latest",
"defaultGpuImageVersion": "latest-gpu",
"supportedFrameworks": [
"pytorch"
],
"multiModelServer": false
},
"v2" : {
"image": "pytorch/torchserve-kfs",
"defaultImageVersion": "0.4.0",
"defaultGpuImageVersion": "0.4.0-gpu",
"supportedFrameworks": [
"pytorch"
],
"multiModelServer": false
}
},
"triton": {
"image": "nvcr.io/nvidia/tritonserver",
"defaultImageVersion": "20.08-py3",
"supportedFrameworks": [
"tensorrt",
"tensorflow",
"onnx",
"pytorch",
"caffe2"
],
"multiModelServer": true
},
"pmml": {
"image": "kfserving/pmmlserver",
"defaultImageVersion": "latest",
"supportedFrameworks": [
"pmml"
],
"multiModelServer": false
},
"lightgbm": {
"image": "kfserving/lgbserver",
"defaultImageVersion": "latest",
"supportedFrameworks": [
"lightgbm"
],
"multiModelServer": false
},
"paddle": {
"image": "kfserving/paddleserver",
"defaultImageVersion": "latest",
"supportedFrameworks": [
"paddle"
],
"multiModelServer": false
}
}
transformers: |-
{
}
explainers: |-
{
"alibi": {
"image" : "kfserving/alibi-explainer",
"defaultImageVersion": "latest"
},
"aix": {
"image" : "kfserving/aix-explainer",
"defaultImageVersion": "latest"
},
"art": {
"image" : "kfserving/art-explainer",
"defaultImageVersion": "latest"
}
}
storageInitializer: |-
{
"image" : "gcr.io/kfserving/storage-initializer:latest",
"memoryRequest": "100Mi",
"memoryLimit": "1Gi",
"cpuRequest": "100m",
"cpuLimit": "1"
}
credentials: |-
{
"gcs": {
"gcsCredentialFileName": "gcloud-application-credentials.json"
},
"s3": {
"s3AccessKeyIDName": "AWS_ACCESS_KEY_ID",
"s3SecretAccessKeyName": "AWS_SECRET_ACCESS_KEY"
}
}
ingress: |-
{
"ingressGateway" : "$(ingressGateway)",
"ingressService" : "istio-ingressgateway.istio-system.svc.cluster.local",
"localGateway" : "knative-serving/knative-local-gateway",
"localGatewayService" : "knative-local-gateway.istio-system.svc.cluster.local"
}
logger: |-
{
"image" : "kfserving/agent:latest",
"memoryRequest": "100Mi",
"memoryLimit": "1Gi",
"cpuRequest": "100m",
"cpuLimit": "1",
"defaultUrl": "http://default-broker"
}
batcher: |-
{
"image" : "kfserving/agent:latest",
"memoryRequest": "1Gi",
"memoryLimit": "1Gi",
"cpuRequest": "1",
"cpuLimit": "1"
}
agent: |-
{
"image" : "kfserving/agent:latest",
"memoryRequest": "100Mi",
"memoryLimit": "1Gi",
"cpuRequest": "100m",
"cpuLimit": "1"
}
-
Configure Custom Domain
- Edit the domain configuration config-map to replace
example.com with your own domain, for example mydomain.com:
kubectl edit cm config-domain --namespace knative-serving
This command opens your default text editor and allows you to edit the config map. apiVersion: v1
data:
_example: |
################################
# #
# EXAMPLE CONFIGURATION #
# #
################################
# ...
example.com: |
kind: ConfigMap
- Edit the file to replace
example.com with the domain you’d like to use, remove the _example key and save your changes. In this example, we configure mydomain.com for all routes:
apiVersion: v1
data:
mydomain.com: ""
kind: ConfigMap
[...]
- Create a new file,
config-domain.yaml and paste the following text, replacing the example.org and example.com values with the new domain you want to use:
apiVersion: v1
kind: ConfigMap
metadata:
name: config-domain
namespace: knative-serving
data:
example.org: |
selector:
app: prod
example.com: ""
- Apply updated domain configuration to your cluster:
kubectl apply --filename config-domain.yaml
- You can map the domain to the IP address of your Knative gateway in your local machine with:
INGRESSGATEWAY=istio-ingressgateway
export GATEWAY_IP=`kubectl get svc $INGRESSGATEWAY --namespace istio-system --output jsonpath="{.status.loadBalancer.ingress[*]['ip']}"`
echo -e "$GATEWAY_IP\t$DOMAIN_NAME" | sudo tee -a /etc/hosts
Publish your Domain
Follow these steps to make your domain publicly accessible:
Set static IP for Knative Gateway
You might want to set a static IP for your Knative gateway, so that the gateway IP does not change each time your cluster is restarted.
To publish your domain, you need to update your DNS provider to point to the IP address for your service ingress.
-
Create a wildcard record for the namespace and custom domain to the ingress IP Address, which would enable hostnames for multiple services in the same namespace to work without creating additional DNS entries. *.default.mydomain.com 59 IN A 35.237.28.44
-
Create an A record to point from the fully qualified domain name to the IP address of your Knative gateway. This step needs to be done for each Knative Service or Route created. helloworld-go.default.mydomain.com 59 IN A 35.237.28.44
If you are using Google Cloud DNS, you can find step-by-step instructions in the Cloud DNS quickstart.
Once the domain update has propagated, you can access your app using the fully qualified domain name of the deployed route, for example http://helloworld-go.default.mydomain.com
- Configure HTTPS Connection
Determine the ingress IP and ports
Execute the following command to determine if your kubernetes cluster is running in an environment that supports external load balancers
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 172.21.109.129 130.211.10.121 ... 17h
If the EXTERNAL-IP value is set, your environment has an external load balancer that you can use for the ingress gateway.
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
If the EXTERNAL-IP value is none (or perpetually pending), your environment does not provide an external load balancer for the ingress gateway. In this case, you can access the gateway using the service’s node port.
# GKE
export INGRESS_HOST=worker-node-address
# Minikube
export INGRESS_HOST=$(minikube ip)
# Other environment(On Prem)
export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
Alternatively you can do Port Forward for testing purpose
INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')
kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80
# start another terminal
export INGRESS_HOST=localhost
export INGRESS_PORT=8080
|