Skip to content
GitLab
Switch branch/tag
Find file Normal viewHistoryPermalink
README.md 14.3 KB
Newer Older
BO ZHANG's avatar
tweaks    
BO ZHANG committed
1 
# `csst-dag`
BO ZHANG's avatar
Initial commit  
BO ZHANG committed
2
BO ZHANG's avatar
tweak    
BO ZHANG committed
3
4
5
6 
- `csst-dag` 是 CSST 数据处理的 DAG 任务匹配工具;
- 通过指定数据和数据处理相关的参数得到一批可处理的 DAG 实例;
- 用户需要通过 [`csst-pipeline-cli`](https://csst-tb.bao.ac.cn/code/csst-pipeline/csst-pipeline-cli) 来触发 DAG 任务的匹配和执行。
BO ZHANG's avatar
tweak    
BO ZHANG committed
7 
## Installation
BO ZHANG's avatar
tweak    
BO ZHANG committed
8
9
10 

安装
BO ZHANG's avatar
tweak    
BO ZHANG committed
11
12
13
14
15
16 
```shell
pip install git+https://csst-tb.bao.ac.cn/code/csst-cicd/csst-dag.git
```

## Usage
BO ZHANG's avatar
tweak    
BO ZHANG committed
17
18 
基本用法
BO ZHANG's avatar
rewrite csst-dag as a dependency package of csst-pipeline-cli    
BO ZHANG committed
19
20
21
22
23 
```python
from csst_dag import CSST_DAGS

# define parameters
test_kwargs = dict(
BO ZHANG's avatar
tweak    
BO ZHANG committed
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46 
    dag_group="default-dag-group",      # DAG组名称，用于标识一组相关的DAG任务，相同组名的任务会被认为是同一组处理
    batch_id="default-batch",           # 批次ID，用于标识一次处理批次，相同批次ID的任务会被认为是同一批次处理
    priority=1,                         # 处理优先级，数字越大优先级越高，支持1-9
    # plan filter
    dataset="csst-msc-c9-25sqdeg-v3",   # 数据集名称
    instrument="MSC",                   # 设备
    obs_type="WIDE",                    # 观测类型
    obs_group="W1",                     # 观测组
    obs_id="10100100412",               # 观测ID
    proposal_id=None,                   # 观测申请ID
    # data filter
    detector = None,                    # 探测器名
    filter = None,                      # 滤光片名称
    prc_status = None,                  # 数据的被处理状态，未处理为-1024
    qc_status = None,                   # 数据的QC状态
    # prc paramters
    pmapname="",                        # CCDS pmap名称
    ref_cat="",                         # 参考星表，DFS中的数据库表名
    extra_kwargs={"a": 1},              # 额外的流水线参数，可用于配置、开关等
    # additional parameters
    return_data_list=True,              # True会返回具体包含的data_list，用于从GUI进行任务编辑、重处理, 默认True
    force_success=False,                # True会强制成功（即便文件没到齐），仅用于测试
    return_details=False,               # True会返回dag_run的详细信息,实际上是task而不是dag_run，仅用于测试
BO ZHANG's avatar
rewrite csst-dag as a dependency package of csst-pipeline-cli    
BO ZHANG committed
47
48
49
50
51
52
53
54
55
56 
)
# get DAG
dag = CSST_DAGS.get("csst-msc-l1-mbi")
# run with parameters
dag_group_run, dag_run_list = dag.run(
    **test_kwargs,
)

# `dag_group_run` is a dict containing information of the DAG group run
print(dag_group_run)
BO ZHANG's avatar
tweak    
BO ZHANG committed
57 
# {'dag_group': 'default-dag-group', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'batch_id': 'default-batch', 'priority': 1, 'created_time': '2025-09-22T07:15:26.033'}
BO ZHANG's avatar
rewrite csst-dag as a dependency package of csst-pipeline-cli    
BO ZHANG committed
58
59
60
61 

# `dag_run_list` is a list[dict], whose elements are `dag_run`s
for dag_run in dag_run_list:
    print(dag_run)
BO ZHANG's avatar
tweak    
BO ZHANG committed
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79 
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '06', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': 'd1216a68fe30bda0b2533bd0111d6b3db22e88cf', 'priority': 1, 'data_list': ['68ceaa82a440bba188724928'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '07', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '2c2d1e4fb26a92adbf0ce271bdff8e8290b678c1', 'priority': 1, 'data_list': ['68ceaa7a6f99de8a9a1cd0c0'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '08', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '36f8ba8d1259f9a80cab0fba8153c4867c49613e', 'priority': 1, 'data_list': ['68ceaa7d6f99de8a9a1cd0c1'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '09', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '1d768baeabc3cd76a4452b4cc96896f0b2163692', 'priority': 1, 'data_list': ['68ceaa77c038bd8c2ff52e43'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '11', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '084d8b175371be8f7aa5ee3e2f8e09765a14b6ce', 'priority': 1, 'data_list': ['68ceaa81c038bd8c2ff52e45'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '12', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '508f4638fdb81ec73a25b59d357fbc134ec4b043', 'priority': 1, 'data_list': ['68ceaa8154c587893d8826bd'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '13', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '620332d257b8ef63ad7f8e11122df0ba32dc59a8', 'priority': 1, 'data_list': ['68ceaa80fbe660f3856fb94b'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '14', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '4119eb27b5e804290ef29bc2336cc13ca64b3078', 'priority': 1, 'data_list': ['68ceaa806f99de8a9a1cd0c2'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '15', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '3c2e11af81c83ce1076e874d7743c39b7d81f63c', 'priority': 1, 'data_list': ['68ceaa8354c587893d8826be'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '16', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': 'd6c004645761f1393db09b02b9c44f05220ef3c2', 'priority': 1, 'data_list': ['68ceaa7e54c587893d8826bc'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '17', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': 'a7024ac9bdc7ad92b724b0d4073f24530b250700', 'priority': 1, 'data_list': ['68ceaa786f99de8a9a1cd0bf'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '18', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '4938154a75d1d903b5d1fddb642a02a434478763', 'priority': 1, 'data_list': ['68ceaa82fbe660f3856fb94c'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '19', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '017676cf7cfd3b081a4300a009e5d355636a8676', 'priority': 1, 'data_list': ['68ceaa84013fe10c488bd735'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '20', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '928da8b2c30be81934e0b3c146b39b09f7285bd5', 'priority': 1, 'data_list': ['68ceaa83a440bba188724929'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '22', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '9365d9aa5b55b756c6a664fb9f0053e9f8c54d17', 'priority': 1, 'data_list': ['68ceaa7fc038bd8c2ff52e44'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '23', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '843dc51e5081f9d0df7b19a7046611e43d422a15', 'priority': 1, 'data_list': ['68ceaa82634453078af601a4'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '24', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': 'ee72839e8585bbe5847e716d6d9f9af75973453e', 'priority': 1, 'data_list': ['68ceaa85013fe10c488bd736'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
# {'dataset': 'csst-msc-c9-25sqdeg-v3', 'instrument': 'MSC', 'obs_type': 'WIDE', 'obs_group': 'W1', 'obs_id': '10100100412', 'detector': '25', 'filter': '', 'custom_id': '', 'batch_id': 'default-batch', 'pmapname': '', 'ref_cat': '', 'dag_group': 'default-dag-group', 'dag': 'csst-msc-l1-mbi', 'dag_group_run': '522d69478694679fd3351e63f0ec76ade53af5ad', 'dag_run': '1a88713fb9ba9b3322c9dea74f2760091d9489bf', 'priority': 1, 'data_list': ['68ceaa81013fe10c488bd734'], 'extra_kwargs': {'a': 1}, 'created_time': '2025-09-22T07:15:26.033', 'rerun': -1, 'status_code': -1024, 'n_file_expected': 1, 'n_file_found': 1, 'object': '', 'proposal_id': ''}
BO ZHANG's avatar
rewrite csst-dag as a dependency package of csst-pipeline-cli    
BO ZHANG committed
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94 

# currently defined DAGs
for dag in CSST_DAGS.keys():
    print(dag)
# csst-msc-l1-qc0
# csst-msc-l1-mbi
# csst-msc-l1-ast
# csst-msc-l1-sls
# csst-msc-l1-ooc
# csst-mci-l1
# csst-mci-l1-qc0
# csst-ifs-l1
# csst-cpic-l1
# csst-cpic-l1-qc0
# csst-hstdm-l1
BO ZHANG's avatar
update README.md    
BO ZHANG committed
95 
```