Commit 159008a7 authored by BO ZHANG's avatar BO ZHANG 🏀
Browse files

feat: 新增任务优先级支持并重构环境变量管理

parent 4aee03c7

README.md

+43 −3
Original line number Diff line number Diff line
@@ -88,6 +88,33 @@ make ansible-down 
ansible-playbook -i deploy_configs/<env>/inventory.ini ansible/<playbook>.yml -e env=<env>

```



### 私有 Harbor 镜像拉取（Docker login）



如果你在 DAG 运行期使用 DockerOperator 从私有 Harbor 拉取镜像，需要确保 **每个 master/worker 宿主机**已完成 `docker login`（因为 DockerOperator 实际由宿主机 `dockerd` 拉取镜像）。



本项目已在 `ansible/deploy.yml` 与 `ansible/update.yml` 中提供可选的自动登录步骤：当 `harbor_project`（用于镜像前缀）与 Harbor 凭证同时存在时，会在所有节点执行一次 `docker login`。



建议优先在 `deploy_configs/<env>/inventory.ini` 配置 `harbor_registry`（例如 `harbor.csst.nao:10443`）用于登录；避免仅为了登录而设置 `harbor_project` 导致 docker-compose 把第三方镜像也切到 Harbor 前缀。



推荐做法：不要把明文密码写进 `inventory.ini`。

- 方案 A（推荐）：在执行 Ansible 的控制端设置环境变量 `HARBOR_USERNAME`/`HARBOR_PASSWORD`。

- 方案 B：使用 Ansible Vault 加密保存 `harbor_username`/`harbor_password`。



方案 A 的使用示例：



```bash

export ENV=p368

export HARBOR_USERNAME=... 

export HARBOR_PASSWORD=...



cd docker-celery-3.0.5

ansible-playbook -i deploy_configs/${ENV}/inventory.ini ansible/update.yml -e env=${ENV}

```



可选开关（用于从源码构建本地镜像以验证前端/网关改动）：

- `build_task_portal`：是否在 master 上 `docker compose build task-portal`（默认 `true`）

- `build_api_gateway`：是否在 master 上 `docker compose build api-gateway`（默认 `false`，仅在需要验证网关源码时开启）



## 环境配置文件



所有环境配置位于 `docker-celery-3.0.5/deploy_configs/`：
@@ -96,14 +123,23 @@ ansible-playbook -i deploy_configs/<env>/inventory.ini ansible/<playbook>.yml -e 
- 存放 Airflow 容器基础静态配置（如 `AIRFLOW_UID`）。



2) `deploy_configs/<env>/inventory.ini`（必需）

- 定义部署拓扑（master/worker）、SSH 连接与关键变量（如 `deploy_dir`、`harbor_project`、`image_tag`、`jwt_secret`）。

- 定义部署拓扑（master/worker）、SSH 连接与关键变量（如 `deploy_dir`、`harbor_project`、`image_tag`、`jwt_secret`）。`worker` 组支持按主机单独设置 `worker_concurrency`，Ansible 在启动各 worker 时会把该值映射到容器内并发配置。



3) `deploy_configs/<env>/variables.toml`（可选）

- 业务级变量导入（DFS/CCDS/OSS 等）。

3) `deploy_configs/<env>/variables.toml`（可选，但推荐）

- `runtime`：唯一的共享运行时环境分组。部署时会同时导入 Airflow Variables（键名固定为 `runtime`）并渲染为 `deploy_configs/<env>/gateway.env`，分别供 DAG 任务执行阶段与 API Gateway 启动阶段使用。

- 其他分组可按业务需要保留；当前不再依赖 `deploy_configs/<env>/dag.env`。



4) `deploy_configs/<env>/connections.json`（可选）

- Airflow Connections 预置。



`variables.toml` 推荐结构示例：



```toml

[runtime]

CSST_DFS_GATEWAY = "10.73.0.27:28000"

CCDS_SERVER_URL = "http://10.73.0.27:29000"

```



## 访问入口与端口



主控节点（Master）常见端口：
@@ -127,6 +163,10 @@ Worker 节点通常不暴露核心业务端口，仅暴露监控相关端口供 


推荐业务方直接对接 API Gateway（避免 Airflow 侧鉴权变化带来的耦合，同时提供 JSONB 查询与批量触发能力）。接口与示例见 [api.md](file:///home/cham/PycharmProjects/csst-airflow/api.md)。



- 批量提交接口支持可选 `priority=high|normal|low`，默认 `normal`。

- Gateway 会将优先级映射到实际调度 DAG：`<dag_id>`、`<dag_id>__high`、`<dag_id>__low`。

- 当前主要工厂 DAG 已提供 `high/low` 变体，并为任务设置 `priority_weight` 与 `weight_rule=absolute`。



## 相关文档



- API Gateway：`docker-celery-3.0.5/api_gateway/README.md`

docker-celery-3.0.5/ansible/deploy.yml

+141 −17
Original line number Diff line number Diff line
@@ -7,8 +7,23 @@ 
    _env: "{{ env | default('p368') }}"

    _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}"

    local_project_dir: ".."

    _docker_registry: "{{ (harbor_registry | default('') | trim) if (harbor_registry | default('') | trim | length > 0) else (((harbor_project | default('')).split('/')[0]) if (harbor_project | default('') | length > 0) else '') }}"

    _docker_login_username: "{{ harbor_username | default(lookup('env', 'HARBOR_USERNAME'), true) | default('', true) }}"

    _docker_login_password: "{{ harbor_password | default(lookup('env', 'HARBOR_PASSWORD'), true) | default('', true) }}"



  tasks:

    - name: Docker login to registry (optional)

      shell: |

        set -e

        echo "{{ _docker_login_password }}" | docker login "{{ _docker_registry }}" -u "{{ _docker_login_username }}" --password-stdin

      args:

        executable: /bin/bash

      when:

        - _docker_registry | length > 0

        - _docker_login_username | length > 0

        - _docker_login_password | length > 0

      no_log: true



    - name: Check if docker compose plugin is available

      command: docker compose version

      register: docker_compose_check
@@ -58,6 +73,11 @@ 
  vars:

    _env: "{{ env | default('p368') }}"

    _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}"

    _build_task_portal: "{{ build_task_portal | default('true') }}"

    _build_api_gateway: "{{ build_api_gateway | default('false') }}"

    _extra_hosts_list: "{{ (extra_hosts | default('') ).split(',') | map('trim') | reject('equalto','') | list }}"

    _harbor_api_username: "{{ lookup('env', 'HARBOR_USERNAME') | default('', true) }}"

    _harbor_api_password: "{{ lookup('env', 'HARBOR_PASSWORD') | default('', true) }}"

  tasks:

    - name: Ensure deploy_configs directory exists

      file:
@@ -69,32 +89,98 @@ 
        src: "{{ playbook_dir }}/../deploy_configs/airflow.env"

        dest: "{{ _deploy_dir }}/deploy_configs/airflow.env"



    - name: Copy gateway.env to remote

      copy:

        src: "{{ playbook_dir }}/../deploy_configs/{{ _env }}/gateway.env"

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/gateway.env"

      ignore_errors: yes



    - name: Copy variables.toml to remote

      copy:

        src: "{{ playbook_dir }}/../deploy_configs/{{ _env }}/variables.toml"

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/variables.toml"

      ignore_errors: yes



    - name: Render gateway.env from variables.toml

      shell: |

        python3 scripts/variables_toml_tool.py gateway-env deploy_configs/{{ _env }}/variables.toml deploy_configs/{{ _env }}/gateway.env

      args:

        chdir: "{{ _deploy_dir }}"



    - name: Remove legacy dag.env on remote

      file:

        path: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/dag.env"

        state: absent



    - name: Copy connections.json to remote

      copy:

        src: "{{ playbook_dir }}/../deploy_configs/{{ _env }}/connections.json"

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/connections.json"

      ignore_errors: yes



    - name: Render docker compose extra_hosts override (optional)

      template:

        src: templates/docker-compose.extra-hosts.override.yml.j2

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml"

      vars:

        extra_hosts_list: "{{ _extra_hosts_list }}"

      when: _extra_hosts_list | length > 0



    - name: Render docker compose Harbor API auth override (optional)

      template:

        src: templates/docker-compose.harbor-api-auth.override.yml.j2

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/docker-compose.harbor-api-auth.override.yml"

        mode: '0600'

      vars:

        harbor_api_username: "{{ _harbor_api_username }}"

        harbor_api_password: "{{ _harbor_api_password }}"

      when:

        - _harbor_api_username | length > 0

        - _harbor_api_password | length > 0

      no_log: true



    - name: Build task-portal image (optional)

      shell: |

        export HARBOR_PROJECT="{{ harbor_project | default('') }}"

        export IMAGE_TAG="{{ image_tag | default('latest') }}"

        export BUILD_NUMBER="{{ build_number | default(image_tag | default('dev')) }}"

        export BUILD_TIME="{{ build_time | default(ansible_facts['date_time']['iso8601']) }}"

        export MASTER_IP="{{ hostvars[groups['master'][0]]['ansible_host'] | default('127.0.0.1') }}"

        export JWT_SECRET="{{ jwt_secret | default('') }}"

        export ENV="{{ _env }}"



        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env build task-portal

      args:

        chdir: "{{ _deploy_dir }}"

      when: _build_task_portal | bool



    - name: Build api-gateway image (optional)

      shell: |

        export HARBOR_PROJECT="{{ harbor_project | default('') }}"

        export IMAGE_TAG="{{ image_tag | default('latest') }}"

        export BUILD_NUMBER="{{ build_number | default(image_tag | default('dev')) }}"

        export BUILD_TIME="{{ build_time | default(ansible_facts['date_time']['iso8601']) }}"

        export MASTER_IP="{{ hostvars[groups['master'][0]]['ansible_host'] | default('127.0.0.1') }}"

        export JWT_SECRET="{{ jwt_secret | default('') }}"

        export ENV="{{ _env }}"



        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env build api-gateway

      args:

        chdir: "{{ _deploy_dir }}"

      when: _build_api_gateway | bool



    - name: Start Master services (postgres, redis, elasticsearch, airflow core, api, etc.)

      shell: |

        export HARBOR_PROJECT="{{ harbor_project | default('') }}"

        export IMAGE_TAG="{{ image_tag | default('latest') }}"

        export MASTER_IP="{{ hostvars[groups['master'][0]]['ansible_host'] | default('127.0.0.1') }}"

        export JWT_SECRET="{{ jwt_secret | default('eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhaXJmbG93IiwiaWF0IjoxNzU1NDQ3NjMxLCJleHAiOjE3ODY5ODM2MzF9.CbxQayEt370iHhvYXhHMESsk1IWmi8QLMOcctIlbicXhybNtBbfboDUdUDMClwvkSuTjC0AOeSHL23pGyy0plQ') }}"

        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env pull || true

        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env --profile master up -d --remove-orphans

        export ENV="{{ _env }}"



        COMPOSE_FILES="-f docker-compose.yaml"

        if [ -f deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml ]; then

          COMPOSE_FILES="$COMPOSE_FILES -f deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml"

        fi

        if [ -f deploy_configs/{{ _env }}/docker-compose.harbor-api-auth.override.yml ]; then

          COMPOSE_FILES="$COMPOSE_FILES -f deploy_configs/{{ _env }}/docker-compose.harbor-api-auth.override.yml"

        fi



        {{ docker_compose_cmd }} $COMPOSE_FILES --env-file deploy_configs/airflow.env pull || true

        {{ docker_compose_cmd }} $COMPOSE_FILES --env-file deploy_configs/airflow.env --profile master up -d --remove-orphans

      args:

        chdir: "{{ _deploy_dir }}"
@@ -107,9 +193,9 @@ 


    - name: Import Airflow Variables from variables.toml (if exists)

      shell: |

        python3 -c 'import sys, json; import tomllib; json.dump(tomllib.load(sys.stdin.buffer), sys.stdout)' < deploy_configs/{{ _env }}/variables.toml > deploy_configs/{{ _env }}/variables_parsed.json

        python3 scripts/variables_toml_tool.py airflow-variables deploy_configs/{{ _env }}/variables.toml deploy_configs/{{ _env }}/variables_parsed.json

        docker exec csst-airflow-airflow-apiserver-1 airflow variables import deploy_configs/{{ _env }}/variables_parsed.json

        rm deploy_configs/{{ _env }}/variables_parsed.json

        rm -f deploy_configs/{{ _env }}/variables_parsed.json

      args:

        chdir: "{{ _deploy_dir }}"

      ignore_errors: yes
@@ -127,6 +213,7 @@ 
  vars:

    _env: "{{ env | default('p368') }}"

    _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}"

    _extra_hosts_list: "{{ (extra_hosts | default('') ).split(',') | map('trim') | reject('equalto','') | list }}"

  tasks:

    - name: Ensure deploy_configs directory exists

      file:
@@ -138,21 +225,47 @@ 
        src: "{{ playbook_dir }}/../deploy_configs/airflow.env"

        dest: "{{ _deploy_dir }}/deploy_configs/airflow.env"



    - name: Copy gateway.env to remote

    - name: Copy variables.toml to worker remote

      copy:

        src: "{{ playbook_dir }}/../deploy_configs/{{ _env }}/gateway.env"

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/gateway.env"

        src: "{{ playbook_dir }}/../deploy_configs/{{ _env }}/variables.toml"

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/variables.toml"

      ignore_errors: yes



    - name: Render gateway.env from variables.toml on worker

      shell: |

        python3 scripts/variables_toml_tool.py gateway-env deploy_configs/{{ _env }}/variables.toml deploy_configs/{{ _env }}/gateway.env

      args:

        chdir: "{{ _deploy_dir }}"



    - name: Render docker compose extra_hosts override on worker (optional)

      template:

        src: templates/docker-compose.extra-hosts.override.yml.j2

        dest: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml"

      vars:

        extra_hosts_list: "{{ _extra_hosts_list }}"

      when: _extra_hosts_list | length > 0



    - name: Remove legacy dag.env on worker remote

      file:

        path: "{{ _deploy_dir }}/deploy_configs/{{ _env }}/dag.env"

        state: absent



    - name: Start Worker services (celery-worker, filebeat, monitoring)

      shell: |

        export HARBOR_PROJECT="{{ harbor_project | default('') }}"

        export IMAGE_TAG="{{ image_tag | default('latest') }}"

        export MASTER_IP="{{ hostvars[groups['master'][0]]['ansible_host'] | default('127.0.0.1') }}"

        export JWT_SECRET="{{ jwt_secret | default('eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhaXJmbG93IiwiaWF0IjoxNzU1NDQ3NjMxLCJleHAiOjE3ODY5ODM2MzF9.CbxQayEt370iHhvYXhHMESsk1IWmi8QLMOcctIlbicXhybNtBbfboDUdUDMClwvkSuTjC0AOeSHL23pGyy0plQ') }}"

        export AIRFLOW__CELERY__WORKER_CONCURRENCY={{ worker_concurrency | default(16) }}

        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env pull || true

        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env --profile worker up -d --remove-orphans

        export ENV="{{ _env }}"

        export WORKER_CONCURRENCY="{{ worker_concurrency | default(16) }}"



        COMPOSE_FILES="-f docker-compose.yaml"

        if [ -f deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml ]; then

          COMPOSE_FILES="$COMPOSE_FILES -f deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml"

        fi



        {{ docker_compose_cmd }} $COMPOSE_FILES --env-file deploy_configs/airflow.env pull || true

        {{ docker_compose_cmd }} $COMPOSE_FILES --env-file deploy_configs/airflow.env --profile worker up -d --remove-orphans

      args:

        chdir: "{{ _deploy_dir }}"
@@ -162,10 +275,21 @@ 
  vars:

    _env: "{{ env | default('p368') }}"

    _deploy_dir: "{{ deploy_dir | default('/opt/csst-airflow') }}"

    _extra_hosts_list: "{{ (extra_hosts | default('') ).split(',') | map('trim') | reject('equalto','') | list }}"

  tasks:

    - name: Restart Flower and API Gateway to ensure they detect new workers

      shell: |

        {{ docker_compose_cmd }} --env-file deploy_configs/airflow.env restart flower api-gateway task-portal

        export ENV="{{ _env }}"



        COMPOSE_FILES="-f docker-compose.yaml"

        if [ -f deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml ]; then

          COMPOSE_FILES="$COMPOSE_FILES -f deploy_configs/{{ _env }}/docker-compose.extra-hosts.override.yml"

        fi

        if [ -f deploy_configs/{{ _env }}/docker-compose.harbor-api-auth.override.yml ]; then

          COMPOSE_FILES="$COMPOSE_FILES -f deploy_configs/{{ _env }}/docker-compose.harbor-api-auth.override.yml"

        fi



        {{ docker_compose_cmd }} $COMPOSE_FILES --env-file deploy_configs/airflow.env restart flower api-gateway task-portal

      args:

        chdir: "{{ _deploy_dir }}"

      ignore_errors: yes

docker-celery-3.0.5/ansible/templates/docker-compose.extra-hosts.override.yml.j2

0 → 100644
+11 −0
Original line number Diff line number Diff line 
services:

  api-gateway:

    extra_hosts:

{% for host in extra_hosts_list %}

      - "{{ host }}"

{% endfor %}

  airflow-worker:

    extra_hosts:

{% for host in extra_hosts_list %}

      - "{{ host }}"

{% endfor %}

docker-celery-3.0.5/ansible/templates/docker-compose.harbor-api-auth.override.yml.j2

0 → 100644
+5 −0
Original line number Diff line number Diff line 
services:

  api-gateway:

    environment:

      HARBOR_API_USERNAME: "{{ harbor_api_username }}"

      HARBOR_API_PASSWORD: "{{ harbor_api_password }}"

docker-celery-3.0.5/ansible/update.yml

+135 −17

File changed.

Preview size limit exceeded, changes collapsed.