科研项目进展
希望能拿到国家级项目
1. 主机+树莓派， 部署kubeedge
前提：需要了解docker， k8s
Cloud core部署在 虚拟机 Linux
Edge core部署在 树莓派
云边协同联邦学习
我看了个例子，云端下放算法程序，摄像头本地处理后反馈云端，云端更新参数后下放迭代后程序，让我们编应用我们是不是可以朝这种方向去搞
思路1：直接使用kubeedge的 sedna （难度较低，有例子）
https://gitee.com/mirrors_kubeedge/sedna/blob/main/examples/federated_learning/yolov5_coco128_mistnet/README.md
上述例子实现了合作训练yolo5。例子容易跑，但自己程序可能相对难写。
借用自带数据库，可以直接处理不同节点的数据参数？分散处理不同容器中工作的数据，容器中运行程序，可能是一个容器对应一块存储空间，信息互不干扰？？实现联邦机器学习？
思路2：在kubeedge 平台部署fate等联邦学习框架，实现联邦学习。
? 可能更好编程
在平台上编写自主联邦学习应用
不用kubeege 直接用Kubefate 快速部署联邦学习
Object Detection Experiment
Assume that there are two edge nodes and a cloud node. Data on the edge nodes cannot be migrated to the cloud due to privacy issues.
Base on this scenario, we will demonstrate the mnist example.
Prepare Nodes
CLOUD_NODE="cloud-node-name"//定义云节点名称
EDGE1_NODE="edge1-node-name"//定义边缘计算节点，提供数据的节点，协同机器学习的节点，注意理解如何分散处理数据的
EDGE2_NODE="edge2-node-name"
Install Sedna
Follow the Sedna installation document to install Sedna.
安装Sedna，是不是类似K8S那样 的平台中安装这样一个数据库，用来存边缘节点处理后的数据
云节点中的容器运行的是不是迭代学习之类的程序？
边缘节点进行数据训练，云端节点形成全局认知，云计算平台实现第二轮操作的处理，以及后面迭代的处理
这种资源调度便是通过借助K8S平台处理，节点间的沟通命令传输等等
Prepare Dataset
Download dataset //下载数据集
Create data interface for EDGE1_NODE.
mkdir -p /data/1
cd /data/1
wget https://github.com/ultralytics/yolov5/releases/download/v1.0/coco128.zip
unzip coco128.zip -d COCO //还好学了Linux知道是个啥，从网站下载数据集便于训练，然后解压
Create data interface for EDGE2_NODE.
mkdir -p /data/2
cd /data/2
wget https://github.com/ultralytics/yolov5/releases/download/v1.0/coco128.zip
unzip coco128.zip -d COCO//边缘节点的两个不同的数据集
Prepare Images //准备图片干嘛？？
https://www.cnblogs.com/xueqiuqiu/articles/13716528.html
分布式学习的资料
https://zhuanlan.zhihu.com/p/449027699
知乎关于联邦学习算法的介绍
https://toutiao.io/posts/6csvlfj/preview
Sedna应用场景 云边协同
边云协同增量学习提升模型精度
下面为具体搭载例子分析
本周任务–跑起来例子
项目例子–需要提升重拾英文了
https://toutiao.io/posts/6csvlfj/preview
This example uses these images:
- aggregation worker:
kubeedge/sedna-example-federated-learning-mistnet-yolo-aggregator:v0.4.0// - train worker:
kubeedge/sedna-example-federated-learning-mistnet-yolo-client:v0.4.0
These images are generated by the script build_images.sh.
Create Federated Learning Job
Create Dataset
create dataset for $EDGE1_NODE and $EDGE2_NODE
kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Dataset
metadata:
name: "coco-dataset-1"
spec:
url: "/data/1/COCO"
format: "dir"
nodeName: $EDGE1_NODE
EOF
kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Dataset
metadata:
name: "coco-dataset-2"
spec:
url: "/data/2/COCO"
format: "dir"
nodeName: $EDGE2_NODE
EOF
Create Model
create the directory /model and /pretrained in $EDGE1_NODE and $EDGE2_NODE.
mkdir -p /model
mkdir -p /pretrained
create the directory /model and /pretrained in the host of $CLOUD_NODE (download links here)
# on the cloud side
mkdir -p /model
mkdir -p /pretrained
cd /pretrained
wget https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/yolov5_coco128_mistnet/yolov5.pth
create model
kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Model
metadata:
name: "yolo-v5-model"
spec:
url: "/model/yolov5.pth"
format: "pth"
EOF
kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Model
metadata:
name: "yolo-v5-pretrained-model"
spec:
url: "/pretrained/yolov5.pth"
format: "pth"
EOF
Create a secret with your S3 user credential. (Optional)
kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: mysecret
annotations:
s3-endpoint: s3.amazonaws.com
s3-usehttps: "1"
stringData:
ACCESS_KEY_ID: XXXX
SECRET_ACCESS_KEY: XXXXXXXX
EOF
Start Federated Learning Job
kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: FederatedLearningJob
metadata:
name: yolo-v5
spec:
pretrainedModel: # option
name: "yolo-v5-pretrained-model"
transmitter: # option
ws: { } # option, by default
s3: # optional, but at least one
aggDataPath: "s3://sedna/fl/aggregation_data"
credentialName: mysecret
aggregationWorker:
model:
name: "yolo-v5-model"
template:
spec:
nodeName: $CLOUD_NODE
containers:
- image: kubeedge/sedna-example-federated-learning-mistnet-yolo-aggregator:v0.4.0
name: agg-worker
imagePullPolicy: IfNotPresent
env: # user defined environments
- name: "cut_layer"
value: "4"
- name: "epsilon"
value: "100"
- name: "aggregation_algorithm"
value: "mistnet"
- name: "batch_size"
value: "32"
- name: "epochs"
value: "100"
resources: # user defined resources
limits:
memory: 8Gi
trainingWorkers:
- dataset:
name: "coco-dataset-1"
template:
spec:
nodeName: $EDGE1_NODE
containers:
- image: kubeedge/sedna-example-federated-learning-mistnet-yolo-client:v0.4.0
name: train-worker
imagePullPolicy: IfNotPresent
args: [ "-i", "1" ]
env: # user defined environments
- name: "cut_layer"
value: "4"
- name: "epsilon"
value: "100"
- name: "aggregation_algorithm"
value: "mistnet"
- name: "batch_size"
value: "32"
- name: "learning_rate"
value: "0.001"
- name: "epochs"
value: "1"
resources: # user defined resources
limits:
memory: 2Gi
- dataset:
name: "coco-dataset-2"
template:
spec:
nodeName: $EDGE2_NODE
containers:
- image: kubeedge/sedna-example-federated-learning-mistnet-yolo-client:v0.4.0
name: train-worker
imagePullPolicy: IfNotPresent
args: [ "-i", "2" ]
env: # user defined environments
- name: "cut_layer"
value: "4"
- name: "epsilon"
value: "100"
- name: "aggregation_algorithm"
value: "mistnet"
- name: "batch_size"
value: "32"
- name: "learning_rate"
value: "0.001"
- name: "epochs"
value: "1"
resources: # user defined resources
limits:
memory: 2Gi
EOF
下面为两个例子
https://github.com/kubeedge/sedna/blob/main/examples/federated_learning/surface_defect_detection/README.md
https://gitee.com/mirrors_kubeedge/sedna/blob/main/examples/federated_learning/yolov5_coco128_mistnet/README.md
