335 lines
9.0 KiB
Markdown
335 lines
9.0 KiB
Markdown
---
|
||
created: "2026-03-10"
|
||
type: resource
|
||
tags: [infrastructure, homelab, kubernetes, ci-cd, gitops]
|
||
source: "HomeLab 部署实践"
|
||
---
|
||
|
||
# HomeLab 基础设施文档
|
||
|
||
## 网络拓扑概览
|
||
|
||
```
|
||
[NAS (192.168.68.x)] ── Gitea (Docker), AdGuard Home (DNS)
|
||
|
|
||
[192.168.68.x 内网]
|
||
|
|
||
[K8s Cluster]
|
||
├── k8s-cp1 (Control Plane) ── 192.168.68.11
|
||
├── k8s-w1 (Worker) ── 192.168.68.21
|
||
└── k8s-w2 (Worker) ── 192.168.68.22
|
||
├── Docker Registry ── NodePort 30500
|
||
├── Drone CI ── NodePort 30344 (webhook), Ingress drone.k8s.home
|
||
├── ArgoCD ── Ingress argocd.k8s.home
|
||
├── ingress-nginx ── LoadBalancer 192.168.68.240
|
||
└── 应用 (invest-api 等)
|
||
```
|
||
|
||
## DNS (AdGuard Home)
|
||
|
||
DNS 服务器: `192.168.68.63`
|
||
|
||
### DNS 重写规则
|
||
|
||
| 域名 | IP | 说明 |
|
||
|------|-----|------|
|
||
| `invest-api.k8s.home` | 192.168.68.240 | OpenBB 投资分析 API |
|
||
| `drone.k8s.home` | 192.168.68.240 | Drone CI |
|
||
| `argocd.k8s.home` | 192.168.68.240 | ArgoCD |
|
||
|
||
注意: 所有 `*.k8s.home` 域名都应指向 ingress-nginx 的 LoadBalancer IP `192.168.68.240`（由 MetalLB 分配），而不是节点 IP。
|
||
|
||
### K8s 节点网络配置
|
||
|
||
K8s 节点 DNS 使用 `8.8.8.8`（绕过 OpenClash fake-ip），否则会导致 ImagePullBackOff。
|
||
网关统一指向 `192.168.68.63`（iStoreOS 主网关）。
|
||
OpenClash 配置了 `SRC-IP-CIDR` 规则让三个节点流量直连。
|
||
|
||
配置文件: `/etc/netplan/50-cloud-init.yaml`（每个节点）
|
||
|
||
## 已部署应用
|
||
|
||
| 应用 | URL | 命名空间 | 说明 |
|
||
|------|-----|----------|------|
|
||
| OpenBB Invest API | `https://invest-api.k8s.home` | invest-api | 投资分析 REST API（50 端点）|
|
||
| Drone CI | `https://drone.k8s.home` | drone | CI/CD |
|
||
| ArgoCD | `https://argocd.k8s.home` | argocd | GitOps 部署 |
|
||
| Swagger UI | `https://invest-api.k8s.home/docs` | invest-api | API 文档 |
|
||
|
||
## Kubernetes 集群
|
||
|
||
### 节点信息
|
||
|
||
| 节点 | 角色 | IP | 说明 |
|
||
|------|------|----|------|
|
||
| k8s-cp1 | Control Plane | 192.168.68.11 | API Server, etcd, scheduler |
|
||
| k8s-w1 | Worker | 192.168.68.21 | 应用负载 |
|
||
| k8s-w2 | Worker | 192.168.68.22 | 应用负载 |
|
||
|
||
### 集群组件
|
||
|
||
| 组件 | 版本/说明 |
|
||
|------|----------|
|
||
| Kubernetes | v1.35.0 |
|
||
| 容器运行时 | containerd 1.7.28 |
|
||
| CNI | 默认 |
|
||
| 负载均衡 | MetalLB (分配 IP: 192.168.68.240) |
|
||
| Ingress | ingress-nginx (External IP: 192.168.68.240) |
|
||
| 存储 | Proxmox CSI |
|
||
| 证书管理 | cert-manager |
|
||
|
||
### kubeconfig
|
||
|
||
- API Server: `https://192.168.68.11:6443`
|
||
- 认证方式: 证书认证 (admin 用户)
|
||
- 本地配置: `C:\Users\yaoji\.kube\config`
|
||
- 获取方式: 从 Control Plane `/etc/kubernetes/admin.conf` 拷贝
|
||
|
||
### 已部署命名空间
|
||
|
||
| 命名空间 | 用途 |
|
||
|----------|------|
|
||
| argocd | ArgoCD GitOps |
|
||
| drone | Drone CI/CD |
|
||
| registry | Docker Registry |
|
||
| ingress-nginx | Ingress 控制器 |
|
||
| cert-manager | TLS 证书管理 |
|
||
| metallb-system | 负载均衡 |
|
||
| csi-proxmox / proxmox-csi | 存储 |
|
||
| invest-api | OpenBB 投资分析 API |
|
||
|
||
---
|
||
|
||
## Git 服务 (Gitea)
|
||
|
||
| 项目 | 值 |
|
||
|------|-----|
|
||
| 部署位置 | NAS Docker |
|
||
| URL | `https://git.colacoder.com` |
|
||
| SSH | `ssh://git@git.colacoder.com:2200/` |
|
||
| 管理员用户 | kai |
|
||
| 邮箱 | wangyaojia@gmail.com |
|
||
|
||
### Gitea 配置注意事项
|
||
|
||
`app.ini` 中需要的配置:
|
||
|
||
```ini
|
||
[webhook]
|
||
ALLOWED_HOST_LIST = private
|
||
SKIP_TLS_VERIFY = true
|
||
```
|
||
|
||
- `ALLOWED_HOST_LIST = private` — 允许 webhook 发送到内网地址
|
||
- `SKIP_TLS_VERIFY = true` — 允许连接自签名证书的服务
|
||
- 修改后需要重启 Gitea 容器
|
||
|
||
### 仓库列表
|
||
|
||
| 仓库 | 用途 |
|
||
|------|------|
|
||
| kai/openbb-invest-api | OpenBB 投资分析 API |
|
||
|
||
---
|
||
|
||
## CI/CD (Drone CI)
|
||
|
||
### 部署信息
|
||
|
||
| 项目 | 值 |
|
||
|------|-----|
|
||
| 命名空间 | drone |
|
||
| 版本 | 2.12.1 |
|
||
| Ingress | `https://drone.k8s.home` |
|
||
| Runner 类型 | Kubernetes Runner |
|
||
| Runner 容量 | 4 并发 |
|
||
|
||
### 访问方式
|
||
|
||
| 方式 | 地址 |
|
||
|------|------|
|
||
| Web UI | `https://drone.k8s.home` |
|
||
| Webhook (给 Gitea 用) | `http://192.168.68.21:30344/hook` |
|
||
| API | `http://192.168.68.21:30344/api/` |
|
||
| API Token | `c7hDypuu5p41r5k6svR0x6QomInqrE6f` |
|
||
|
||
### Drone Server 配置 (ConfigMap: drone)
|
||
|
||
| 键 | 值 |
|
||
|-----|-------|
|
||
| DRONE_GITEA_SERVER | `https://git.colacoder.com/` |
|
||
| DRONE_GITEA_CLIENT_ID | `c95249a5-9cad-4813-89b4-3f4f9f7d3cee` |
|
||
| DRONE_SERVER_HOST | drone.k8s.home |
|
||
| DRONE_SERVER_PROTO | https |
|
||
| DRONE_USER_CREATE | username:kai,admin:true |
|
||
|
||
### Drone Runner 配置 (ConfigMap: drone-runner-drone-runner-kube)
|
||
|
||
| 键 | 值 |
|
||
|-----|-------|
|
||
| DRONE_RPC_HOST | drone.drone.svc.cluster.local:8080 |
|
||
| DRONE_RPC_PROTO | http |
|
||
| DRONE_NAMESPACE_DEFAULT | drone |
|
||
| DRONE_RUNNER_CAPACITY | 4 |
|
||
|
||
### RBAC
|
||
|
||
Runner ServiceAccount (`drone:drone-runner-drone-runner-kube`) 需要在 `drone` namespace 有以下权限:
|
||
- secrets: create, delete
|
||
- pods, pods/log: get, create, delete, list, watch, update
|
||
|
||
Helm 安装时 RBAC 错误地创建在 `default` namespace，需要手动在 `drone` namespace 创建 Role 和 RoleBinding。
|
||
|
||
### Gitea Webhook 配置
|
||
|
||
| 项目 | 值 |
|
||
|------|-----|
|
||
| Target URL | `http://192.168.68.21:30344/hook` |
|
||
| Content Type | application/json |
|
||
| Secret | `KE0HQksXhJ53ojiLwgAIp0JC4QCl1NsE` |
|
||
| Events | Push |
|
||
|
||
注意: Drone 无法自动在 Gitea 创建 webhook（OAuth 权限问题），需要手动创建。
|
||
|
||
### Pipeline 模板 (.drone.yml)
|
||
|
||
使用 kaniko 构建（Kubernetes Runner 不支持 privileged 模式的 `plugins/docker`）:
|
||
|
||
```yaml
|
||
kind: pipeline
|
||
type: kubernetes
|
||
name: build-and-push
|
||
|
||
trigger:
|
||
branch: [main, develop]
|
||
event: [push, custom]
|
||
|
||
steps:
|
||
- name: build-and-push
|
||
image: gcr.io/kaniko-project/executor:debug
|
||
commands:
|
||
- >
|
||
/kaniko/executor
|
||
--context=/drone/src
|
||
--dockerfile=Dockerfile
|
||
--destination=192.168.68.11:30500/IMAGE_NAME:${DRONE_COMMIT_SHA:0:8}
|
||
--destination=192.168.68.11:30500/IMAGE_NAME:latest
|
||
--insecure
|
||
--skip-tls-verify
|
||
```
|
||
|
||
---
|
||
|
||
## Docker Registry
|
||
|
||
| 项目 | 值 |
|
||
|------|-----|
|
||
| 命名空间 | registry |
|
||
| 镜像 | registry:2 |
|
||
| Service | NodePort 30500 |
|
||
| 存储 | PVC 10Gi |
|
||
| 访问地址 | `http://192.168.68.11:30500` |
|
||
| 基础设施仓库 | `C:\Users\yaoji\git\ColaCoder\k8s-infra\registry\` |
|
||
|
||
### Registry API
|
||
|
||
```bash
|
||
# 查看所有镜像
|
||
curl http://192.168.68.11:30500/v2/_catalog
|
||
|
||
# 查看镜像 tags
|
||
curl http://192.168.68.11:30500/v2/IMAGE_NAME/tags/list
|
||
```
|
||
|
||
### Worker 节点 containerd 配置
|
||
|
||
两个 Worker 节点 `/etc/containerd/config.toml` 中添加:
|
||
|
||
```toml
|
||
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.68.11:30500"]
|
||
endpoint = ["http://192.168.68.11:30500"]
|
||
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.registry.svc.cluster.local:5000"]
|
||
endpoint = ["http://registry.registry.svc.cluster.local:5000"]
|
||
```
|
||
|
||
修改后需要 `sudo systemctl restart containerd`。Control Plane 不需要配置。
|
||
|
||
### 本地 Docker Desktop 配置
|
||
|
||
`C:\Users\yaoji\.docker\daemon.json`:
|
||
|
||
```json
|
||
{
|
||
"insecure-registries": ["192.168.68.11:30500"]
|
||
}
|
||
```
|
||
|
||
修改后需要重启 Docker Desktop。
|
||
|
||
---
|
||
|
||
## ArgoCD
|
||
|
||
| 项目 | 值 |
|
||
|------|-----|
|
||
| 命名空间 | argocd |
|
||
| 同步策略 | 自动 (prune + selfHeal) |
|
||
| CreateNamespace | true |
|
||
|
||
### 已注册应用
|
||
|
||
| 应用 | 源仓库 | 路径 | 分支 | 目标命名空间 |
|
||
|------|--------|------|------|-------------|
|
||
| invest-api | `https://git.colacoder.com/kai/openbb-invest-api.git` | k8s/base | main | invest-api |
|
||
|
||
---
|
||
|
||
## 完整部署流程 (GitOps)
|
||
|
||
```
|
||
开发者 git push
|
||
↓
|
||
Gitea 接收代码
|
||
↓ (webhook)
|
||
Drone CI 触发构建
|
||
↓ (kaniko)
|
||
Docker 镜像推送到 Registry (192.168.68.11:30500)
|
||
↓
|
||
ArgoCD 检测 k8s manifest 变化
|
||
↓ (自动同步)
|
||
K8s 拉取镜像并部署
|
||
```
|
||
|
||
---
|
||
|
||
## 踩坑记录
|
||
|
||
### Gitea Webhook
|
||
|
||
1. **webhook 被拒**: Gitea 默认不允许发送 webhook 到内网地址，需要 `ALLOWED_HOST_LIST = private`
|
||
2. **TLS 验证失败**: 内网自签名证书需要 `SKIP_TLS_VERIFY = true`
|
||
3. **403 Forbidden**: Drone OAuth2 应用丢失需要重新创建；webhook 手动创建时需要填写正确的 secret
|
||
|
||
### Drone CI
|
||
|
||
1. **Runner 无法连接 Server**: Service 端口 8080 但 Runner 连的默认 80，需要在 `DRONE_RPC_HOST` 加 `:8080`
|
||
2. **手动触发无反应**: `.drone.yml` trigger event 需要包含 `custom`
|
||
3. **RBAC 权限不足**: Helm 把 Role 创建在 `default` namespace，需要手动在 `drone` namespace 创建
|
||
4. **不能用 plugins/docker**: Kubernetes Runner 不支持 privileged 模式，改用 kaniko
|
||
|
||
### Docker 镜像
|
||
|
||
1. **OpenBB 需要 home 目录**: `nobody` 用户没有 home，需要创建 `appuser` 并预建 `.openbb_platform` 目录
|
||
2. **OpenBB 需要写入 site-packages**: 启动时写 `.build.lock`，需要 `chown -R appuser:appuser /usr/local/lib/python3.12/site-packages/openbb`
|
||
|
||
### DNS / Ingress
|
||
|
||
1. **Ingress IP 不是节点 IP**: `*.k8s.home` 域名必须指向 MetalLB 分配的 LoadBalancer IP (`192.168.68.240`)，不是节点 IP (`192.168.68.22`)
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- [[OpenBB Invest API - K8s Infrastructure]]
|
||
- [[OpenBB Invest API]]
|