141 lines
3.9 KiB
Markdown
141 lines
3.9 KiB
Markdown
---
|
||
created: "2026-03-09"
|
||
type: resource
|
||
tags: [kubernetes, infrastructure, devops, drone-ci, argocd, docker-registry]
|
||
source: "openbb-invest-api 项目部署实践"
|
||
---
|
||
|
||
# OpenBB Invest API - K8s 基础设施
|
||
|
||
## 概述
|
||
|
||
OpenBB Invest API 的完整 Kubernetes 部署架构，包含集群信息、CI/CD 流水线和 GitOps 配置。
|
||
|
||
## 集群
|
||
|
||
| 节点 | 角色 | IP |
|
||
|------|------|----|
|
||
| k8s-cp1 | control-plane | 192.168.68.11 |
|
||
| k8s-w1 | worker | 192.168.68.21 |
|
||
| k8s-w2 | worker | 192.168.68.22 |
|
||
|
||
- Kubernetes v1.35.0
|
||
- 容器运行时: containerd 1.7.28
|
||
- 负载均衡: MetalLB
|
||
- Ingress: ingress-nginx
|
||
- 存储: Proxmox CSI
|
||
- 证书管理: cert-manager
|
||
|
||
## kubeconfig
|
||
|
||
- 位置: `C:\Users\yaoji\.kube\config`
|
||
- 认证: 证书认证 (admin 用户)
|
||
- API server: `https://192.168.68.11:6443`
|
||
|
||
## Docker Registry
|
||
|
||
- 命名空间: `registry`
|
||
- 镜像: `registry:2`
|
||
- 服务: NodePort 30500
|
||
- 持久卷: 10Gi
|
||
- 访问地址: `http://192.168.68.11:30500`
|
||
- 基础设施仓库: `C:\Users\yaoji\git\ColaCoder\k8s-infra\registry\`
|
||
|
||
### Worker 节点 containerd 配置
|
||
|
||
两个 worker 节点的 `/etc/containerd/config.toml`:
|
||
|
||
```toml
|
||
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.68.11:30500"]
|
||
endpoint = ["http://192.168.68.11:30500"]
|
||
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.registry.svc.cluster.local:5000"]
|
||
endpoint = ["http://registry.registry.svc.cluster.local:5000"]
|
||
```
|
||
|
||
Control plane 不需要配置（有 taint，pod 不会调度到上面）。
|
||
|
||
## Drone CI
|
||
|
||
- 命名空间: `drone`
|
||
- Server: 容器端口 80，Service 端口 8080
|
||
- Runner: Kubernetes runner（容量 4）
|
||
- Ingress: `drone.k8s.home`
|
||
|
||
### Runner ConfigMap
|
||
|
||
| 键 | 值 |
|
||
|-----|-------|
|
||
| DRONE_RPC_HOST | drone.drone.svc.cluster.local:8080 |
|
||
| DRONE_RPC_PROTO | http |
|
||
| DRONE_NAMESPACE_DEFAULT | drone |
|
||
|
||
### 流水线 (.drone.yml)
|
||
|
||
使用 kaniko 构建（k8s runner 不支持 privileged 模式）:
|
||
|
||
```yaml
|
||
kind: pipeline
|
||
type: kubernetes
|
||
name: build-and-push
|
||
trigger:
|
||
branch: [main, develop]
|
||
event: [push, custom]
|
||
steps:
|
||
- name: build-and-push
|
||
image: gcr.io/kaniko-project/executor:debug
|
||
commands:
|
||
- /kaniko/executor
|
||
--context=/drone/src
|
||
--dockerfile=Dockerfile
|
||
--destination=192.168.68.11:30500/invest-api:${DRONE_COMMIT_SHA:0:8}
|
||
--destination=192.168.68.11:30500/invest-api:latest
|
||
--insecure --skip-tls-verify
|
||
```
|
||
|
||
## ArgoCD
|
||
|
||
- 命名空间: `argocd`
|
||
- Application: `invest-api`
|
||
- 源仓库: `https://git.colacoder.com/kai/openbb-invest-api.git`，路径 `k8s/base`
|
||
- 目标分支: `main`，命名空间 `invest-api`
|
||
- 同步策略: 自动（prune + selfHeal + CreateNamespace）
|
||
|
||
## Gitea
|
||
|
||
- URL: `https://git.colacoder.com`
|
||
- 仓库: `kai/openbb-invest-api`
|
||
- SSH: `ssh://git@git.colacoder.com:2200/kai/openbb-invest-api.git`
|
||
|
||
## 部署流程
|
||
|
||
1. `git push` 到 Gitea（main/develop 分支）
|
||
2. Gitea webhook 触发 Drone CI
|
||
3. Drone/kaniko 构建 Docker 镜像并推送到 `192.168.68.11:30500`
|
||
4. ArgoCD 检测 `k8s/base/` 中的 manifest 变化并自动同步
|
||
5. k8s 从 registry 拉取镜像并部署
|
||
|
||
## invest-api K8s Manifests (k8s/base/)
|
||
|
||
- 命名空间: `invest-api`
|
||
- Deployment: 镜像 `192.168.68.11:30500/invest-api:latest`，100m-500m CPU，256Mi-512Mi 内存，健康检查 `/health:8000`
|
||
- Service: ClusterIP 端口 8000
|
||
- Secret: `invest-api-secrets`（可选，用于 API 密钥）
|
||
|
||
## 本地 Docker Desktop
|
||
|
||
`C:\Users\yaoji\.docker\daemon.json`:
|
||
|
||
```json
|
||
{"insecure-registries": ["192.168.68.11:30500"]}
|
||
```
|
||
|
||
## 踩坑记录
|
||
|
||
- **Drone Runner RPC 连接超时**: Runner 默认连接端口 80，但 Service 暴露的是 8080。通过 patch configmap 添加 `:8080` 修复。
|
||
- **Drone 手动触发无反应**: UI 手动触发发送的 event 是 `custom`，需要在 `.drone.yml` trigger 中添加 `custom` event。
|
||
- **kubeconfig 传输损坏**: 通过聊天传输 RSA 私钥会被截断/损坏，需要通过 SSH 直接传输文件。
|
||
|
||
## Related
|
||
|
||
- [[OpenBB Invest API]]
|