Loading README.md +30 −7 Original line number Diff line number Diff line Loading @@ -18,10 +18,19 @@ 如果您需要在真实的集群(1个 Master 节点 + 多个 Worker 节点)上部署该系统,我们提供了开箱即用的 Ansible Playbooks。由于我们有多个不同的部署环境(如 `p368`, `csu`, `zjlab`),我们为每个环境准备了专属的 Ansible Inventory 文件。 **💡 进阶:自定义 Worker 节点的并发度** 您可以在 Inventory 文件(例如 `inventory.p368.ini`)中,为每台 Worker 机器灵活配置变量: - `worker_concurrency`: 设置该台机器上 Worker 容器的 Slots 数量(例如 16 或 64)。 *(通过这种方式,您可以让高配物理机加大并发,低配物理机少跑,充分压榨集群性能!)* **💡 进阶:为每个 Worker 单独指定并发度** 您可以直接在 Inventory 文件(例如 `inventory.p368.ini`)的每个 Worker 节点条目上设置 `worker_concurrency`,从而让不同机器使用不同的 Celery 并发数: ```ini [worker] worker-a ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker-a worker_concurrency=16 worker-b ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker-b worker_concurrency=64 worker-c ansible_host=192.168.25.21 ansible_user=root deploy_dir=/opt/csst-airflow-worker-c worker_concurrency=8 ``` - `worker_concurrency`:该节点上 Airflow Celery Worker 的并发槽位数。 - 建议按机器 CPU / 内存资源分别设置,不要所有 Worker 都使用同一个值。 - 如无特殊需要,请在每个 Worker 上显式写出该值,避免依赖默认值。 #### 1. 准备工作 - **安装 Ansible**:如果您的执行机(通常是您的本地电脑或 Master 节点)尚未安装 Ansible,请先通过以下命令安装: Loading Loading @@ -56,8 +65,8 @@ node-master ansible_host=192.168.25.18 ansible_user=root deploy_dir=/opt/csst-airflow-master [worker] node-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 node-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 node-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 worker_concurrency=16 node-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 worker_concurrency=32 [all:vars] env_name=csu Loading @@ -65,6 +74,20 @@ env_name=csu deploy_dir=/opt/csst-airflow ``` 如果多个 Worker 机器规格相同,也可以在 `[worker:vars]` 中统一配置默认值,再在个别节点上单独覆盖: ```ini [worker:vars] worker_concurrency=16 [worker] node-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 node-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 node-worker3 ansible_host=192.168.25.21 ansible_user=root deploy_dir=/opt/csst-airflow-worker3 worker_concurrency=32 ``` 上述配置会在部署时写入目标节点的 `.env`,并在启动 Worker 容器时传递给 `AIRFLOW__CELERY__WORKER_CONCURRENCY`,因此只需要修改 Inventory 后重新执行 Ansible 部署或更新即可生效。 #### 3. 配置环境变量 为了让集群中的所有组件(特别是 Worker 节点)能够正确找到主节点和其他依赖,您需要为该环境配置专用的 `.env` 文件。 在 `envs/` 目录下,复制或编辑对应环境的 `.env` 文件(例如 `envs/csu.env`)。您**必须**修改 `MASTER_IP` 和 `HARBOR` 这两个关键变量: Loading docker-celery-3.0.5/ansible/deploy.yml +20 −0 Original line number Diff line number Diff line Loading @@ -59,6 +59,16 @@ _env_name: "{{ env_name | default('p368') }}" _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}" tasks: - name: Ensure envs directory exists file: path: "{{ _deploy_dir }}/envs" state: directory - name: Render .env file from template template: src: "{{ playbook_dir }}/roles/common/templates/env.j2" dest: "{{ _deploy_dir }}/envs/{{ _env_name }}.env" - name: Start Master services (postgres, redis, elasticsearch, airflow core, api, etc.) shell: | {{ docker_compose_cmd }} --env-file envs/{{ _env_name }}.env --profile master up -d Loading @@ -72,6 +82,16 @@ _env_name: "{{ env_name | default('p368') }}" _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}" tasks: - name: Ensure envs directory exists file: path: "{{ _deploy_dir }}/envs" state: directory - name: Render .env file from template template: src: "{{ playbook_dir }}/roles/common/templates/env.j2" dest: "{{ _deploy_dir }}/envs/{{ _env_name }}.env" - name: Start Worker services (celery-worker, filebeat, monitoring) shell: | export AIRFLOW__CELERY__WORKER_CONCURRENCY={{ worker_concurrency | default(16) }} Loading docker-celery-3.0.5/ansible/inventories/inventory.csu.ini +5 −4 Original line number Diff line number Diff line [master] # Replace with your actual master node IP # Replace with your actual master node IP or hostname csu-master ansible_host=192.168.25.18 ansible_user=root [worker] # Add all your worker node IPs here # e.g., csu-worker1 ansible_host=192.168.25.19 ansible_user=root # Add one worker node per line # e.g., csu-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 worker_concurrency=16 # e.g., csu-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 worker_concurrency=32 [all:vars] # The environment name to deploy (e.g., p368, csu, zjlab) Loading docker-celery-3.0.5/ansible/inventories/inventory.zjlab.ini +5 −4 Original line number Diff line number Diff line [master] # Replace with your actual master node IP # Replace with your actual master node IP or hostname zjlab-master ansible_host=10.200.60.244 ansible_user=root [worker] # Add all your worker node IPs here # e.g., zjlab-worker1 ansible_host=10.200.60.245 ansible_user=root # Add one worker node per line # e.g., zjlab-worker1 ansible_host=10.200.60.245 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 worker_concurrency=16 # e.g., zjlab-worker2 ansible_host=10.200.60.246 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 worker_concurrency=32 [all:vars] # The environment name to deploy (e.g., p368, csu, zjlab) Loading docker-celery-3.0.5/ansible/roles/common/templates/env.j2 +1 −0 Original line number Diff line number Diff line Loading @@ -13,3 +13,4 @@ HARBOR={{ harbor }} # --- Celery Worker Config --- # Defines the concurrency per worker node. Can be overridden in inventory. WORKER_CONCURRENCY={{ worker_concurrency | default(8) }} WORKER_LABEL={{ ansible_host }} Loading
README.md +30 −7 Original line number Diff line number Diff line Loading @@ -18,10 +18,19 @@ 如果您需要在真实的集群(1个 Master 节点 + 多个 Worker 节点)上部署该系统,我们提供了开箱即用的 Ansible Playbooks。由于我们有多个不同的部署环境(如 `p368`, `csu`, `zjlab`),我们为每个环境准备了专属的 Ansible Inventory 文件。 **💡 进阶:自定义 Worker 节点的并发度** 您可以在 Inventory 文件(例如 `inventory.p368.ini`)中,为每台 Worker 机器灵活配置变量: - `worker_concurrency`: 设置该台机器上 Worker 容器的 Slots 数量(例如 16 或 64)。 *(通过这种方式,您可以让高配物理机加大并发,低配物理机少跑,充分压榨集群性能!)* **💡 进阶:为每个 Worker 单独指定并发度** 您可以直接在 Inventory 文件(例如 `inventory.p368.ini`)的每个 Worker 节点条目上设置 `worker_concurrency`,从而让不同机器使用不同的 Celery 并发数: ```ini [worker] worker-a ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker-a worker_concurrency=16 worker-b ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker-b worker_concurrency=64 worker-c ansible_host=192.168.25.21 ansible_user=root deploy_dir=/opt/csst-airflow-worker-c worker_concurrency=8 ``` - `worker_concurrency`:该节点上 Airflow Celery Worker 的并发槽位数。 - 建议按机器 CPU / 内存资源分别设置,不要所有 Worker 都使用同一个值。 - 如无特殊需要,请在每个 Worker 上显式写出该值,避免依赖默认值。 #### 1. 准备工作 - **安装 Ansible**:如果您的执行机(通常是您的本地电脑或 Master 节点)尚未安装 Ansible,请先通过以下命令安装: Loading Loading @@ -56,8 +65,8 @@ node-master ansible_host=192.168.25.18 ansible_user=root deploy_dir=/opt/csst-airflow-master [worker] node-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 node-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 node-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 worker_concurrency=16 node-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 worker_concurrency=32 [all:vars] env_name=csu Loading @@ -65,6 +74,20 @@ env_name=csu deploy_dir=/opt/csst-airflow ``` 如果多个 Worker 机器规格相同,也可以在 `[worker:vars]` 中统一配置默认值,再在个别节点上单独覆盖: ```ini [worker:vars] worker_concurrency=16 [worker] node-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 node-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 node-worker3 ansible_host=192.168.25.21 ansible_user=root deploy_dir=/opt/csst-airflow-worker3 worker_concurrency=32 ``` 上述配置会在部署时写入目标节点的 `.env`,并在启动 Worker 容器时传递给 `AIRFLOW__CELERY__WORKER_CONCURRENCY`,因此只需要修改 Inventory 后重新执行 Ansible 部署或更新即可生效。 #### 3. 配置环境变量 为了让集群中的所有组件(特别是 Worker 节点)能够正确找到主节点和其他依赖,您需要为该环境配置专用的 `.env` 文件。 在 `envs/` 目录下,复制或编辑对应环境的 `.env` 文件(例如 `envs/csu.env`)。您**必须**修改 `MASTER_IP` 和 `HARBOR` 这两个关键变量: Loading
docker-celery-3.0.5/ansible/deploy.yml +20 −0 Original line number Diff line number Diff line Loading @@ -59,6 +59,16 @@ _env_name: "{{ env_name | default('p368') }}" _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}" tasks: - name: Ensure envs directory exists file: path: "{{ _deploy_dir }}/envs" state: directory - name: Render .env file from template template: src: "{{ playbook_dir }}/roles/common/templates/env.j2" dest: "{{ _deploy_dir }}/envs/{{ _env_name }}.env" - name: Start Master services (postgres, redis, elasticsearch, airflow core, api, etc.) shell: | {{ docker_compose_cmd }} --env-file envs/{{ _env_name }}.env --profile master up -d Loading @@ -72,6 +82,16 @@ _env_name: "{{ env_name | default('p368') }}" _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}" tasks: - name: Ensure envs directory exists file: path: "{{ _deploy_dir }}/envs" state: directory - name: Render .env file from template template: src: "{{ playbook_dir }}/roles/common/templates/env.j2" dest: "{{ _deploy_dir }}/envs/{{ _env_name }}.env" - name: Start Worker services (celery-worker, filebeat, monitoring) shell: | export AIRFLOW__CELERY__WORKER_CONCURRENCY={{ worker_concurrency | default(16) }} Loading
docker-celery-3.0.5/ansible/inventories/inventory.csu.ini +5 −4 Original line number Diff line number Diff line [master] # Replace with your actual master node IP # Replace with your actual master node IP or hostname csu-master ansible_host=192.168.25.18 ansible_user=root [worker] # Add all your worker node IPs here # e.g., csu-worker1 ansible_host=192.168.25.19 ansible_user=root # Add one worker node per line # e.g., csu-worker1 ansible_host=192.168.25.19 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 worker_concurrency=16 # e.g., csu-worker2 ansible_host=192.168.25.20 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 worker_concurrency=32 [all:vars] # The environment name to deploy (e.g., p368, csu, zjlab) Loading
docker-celery-3.0.5/ansible/inventories/inventory.zjlab.ini +5 −4 Original line number Diff line number Diff line [master] # Replace with your actual master node IP # Replace with your actual master node IP or hostname zjlab-master ansible_host=10.200.60.244 ansible_user=root [worker] # Add all your worker node IPs here # e.g., zjlab-worker1 ansible_host=10.200.60.245 ansible_user=root # Add one worker node per line # e.g., zjlab-worker1 ansible_host=10.200.60.245 ansible_user=root deploy_dir=/opt/csst-airflow-worker1 worker_concurrency=16 # e.g., zjlab-worker2 ansible_host=10.200.60.246 ansible_user=root deploy_dir=/opt/csst-airflow-worker2 worker_concurrency=32 [all:vars] # The environment name to deploy (e.g., p368, csu, zjlab) Loading
docker-celery-3.0.5/ansible/roles/common/templates/env.j2 +1 −0 Original line number Diff line number Diff line Loading @@ -13,3 +13,4 @@ HARBOR={{ harbor }} # --- Celery Worker Config --- # Defines the concurrency per worker node. Can be overridden in inventory. WORKER_CONCURRENCY={{ worker_concurrency | default(8) }} WORKER_LABEL={{ ansible_host }}