机器学习平台kubeflow搭建

文章目录

机器学习平台kubeflow搭建
前言
一、搭建流程
二、问题
三、总结

前言

首先来一段官网的介绍：Kubeflow项目致力于使Kubernetes上机器学习(ML)工作流的部署变得简单、可移植和可扩展。我们的目标不是重新创建其他服务，而是提供一种直接的方式，为ML部署最佳的开源系统到不同的基础设施。无论你在哪里运行Kubernetes，你都应该能够运行Kubeflow。(谷歌翻译)

kubeflow可以运行在多种云平台基础设施上（如：GKE）。本文的kubeflow安装在自建的k8s平台上。
基础环境如下：

k0s: v1.21.5+k0s.1
k0sctl: v0.11.3
k8s: 1.21.5 (k0s构建单机版)
kubeflow: 1.4.0

`官方文档前提条件：

kubernetes 版本1.19 (我用的1.21，碰到了问题，文章后面有解决方案)
kustomize 版本3.2.0不能用4.X (我用的4.4.1通过，反而3.X的不能通过，估计和k8s版本有关)

注意：kubeflow从1.3版本开始，安装方式出现了变化，具体变化请看manifests项目的 readme

一、搭建流程

1. k0s 构建k8s平台

具体搭建方法请百度。

2. 准备pv

kubeflow的组件需要存储，所以需要提前准备好pv，本次实验存储采用的本地磁盘存储的方式。流程如下:

mkdir -p /data/istio-authservice /data/katib-mysql /data/minio /data/mysql-pv-claim

编写kubeflow-storage.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: authservice
  namespace: istio-system
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/istio-authservice"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  namespace: kubeflow
  name: katib-mysql
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/katib-mysql"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio
  namespace: kubeflow
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/minio"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-claim
  namespace: kubeflow
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/mysql-pv-claim"

创建storageclass 和pv

kubectl apply -f kubeflow-storage.yaml

3. 安装kubeflow

拉取manifest项目

git clone https://github.com/kubeflow/manifests.git

修改yaml，下面每个文件里面添加 storageClassName: local-storage
- apps/katib/upstream/components/mysql/pvc.yaml
- apps/pipeline/upstream/third-party/minio/base/minio-pvc.yaml
- apps/pipeline/upstream/third-party/mysql/base/mysql-pv-claim.yaml
- common/oidc-authservice/base/pvc.yaml
安装kubeflow

cd /path/to/manifests

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

二、问题

Q: authservice-0 pod 启动失败：Error opening bolt store: open /var/lib/authservice/data.db: permission denied
A: 通过翻看manifests项目的authservice StatefulSet定义文件 common/oidc-authservice/overlays/ibm-storage-config/statefulset.yaml：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: authservice
spec:
  template:
    spec:
      initContainers:
      - name: fix-permission
        image: busybox
        command: ['sh', '-c']
        args: ['chmod -R 777 /var/lib/authservice;']
        volumeMounts:
        - mountPath: /var/lib/authservice
          name: data