**astropy 需升级至 5.3** **老写法同时兼容本地nas和云上s3,只要读写路径以s3:// 协议开头会自动识别** 如果需要读写S3时,需要传入s3的密钥和endpoint等配置,有两种方法可选 ### 方法1 环境变量 执行下面三个环境变量,本文档下面介绍到的所有方法都会尝试读取环境变量以获取配置 ```python s3_options = { 'key': os.getenv('S3_KEY'), 'secret': os.getenv('S3_SECRET'), 'endpoint_url': os.getenv('S3_ENDPOINT_URL') } ``` ### 方法2 每次调用方法时传入 s3_options ``` 在第一个kwargs参数位置指定s3_options, s3_options示例: ```json s3_options = { "key": "minioadmin", "secret": "minioadmin", "endpoint_url": "http://localhost:9000" } ``` ## 本地到s3的上传与下载 ### 上传 ```python from csst_fs import s3_fs # single file,s3_options from env s3_fs.put('requirements.txt', 's3://csst-prod/gaia/test/requirements.txt') # single file,s3_options from function parameter s3_fs.put('requirements.txt', 's3://csst-prod/gaia/test/requirements.txt', s3_options=s3_options) # folder,to s3 s3://csst-prod/common s3_fs.put('./common', 's3://csst-prod/', recursive=True) s3_fs.put('./common', 's3://csst-prod/', s3_options=s3_options, recursive=True) ``` ### 下载 ```python from csst_fs import s3_fs # single file s3_fs.get('s3://csst-prod/gaia/test/requirements.txt', 'requirements.txt') s3_fs.get('s3://csst-prod/gaia/test/requirements.txt', 'requirements.txt', s3_options=s3_options) # folder s3_fs.get('s3://csst-prod/gaia/data', './', recursive=True) s3_fs.get('s3://csst-prod/gaia/data', './', s3_options=s3_options, recursive=True) ``` ## astropy直接读写s3的写法适配 ### fits.open #### 老写法 ```python fits.open(path) ``` usage reference: [https://docs.astropy.org/en/stable/io/fits/api/files.html#open](https://docs.astropy.org/en/stable/io/fits/api/files.html#open) #### 新写法 ```python from csst_fs import fsspec_fits fsspec_fits.open("s3://csst-prod/gaia/xx.fits") fsspec_fits.open("s3://csst-prod/gaia/xx.fits", s3_options=s3_options) ``` ### fits.getheader #### 老写法 ```python fits.getheader(filename=in_image_path, ext=1) ``` usage reference: [https://docs.astropy.org/en/stable/io/fits/api/files.html#getheader](https://docs.astropy.org/en/stable/io/fits/api/files.html#getheader) #### 新写法 ```python from csst_fs import fsspec_fits fsspec_fits.getheader(filename=in_image_path, ext=1) fsspec_fits.getheader(filename=in_image_path, ext=1, s3_options=s3_options) ``` ### fits.getdata #### 老写法 ```python fits.getdata(in_ref_flat) fits.getdata( in_ref_shutter, 1) ``` usage reference: [https://docs.astropy.org/en/stable/io/fits/api/files.html#getdata](https://docs.astropy.org/en/stable/io/fits/api/files.html#getdata) #### 新写法 ```python from csst_fs import fsspec_fits fsspec_fits.getdata(in_ref_flat) fsspec_fits.getdata(in_ref_flat, s3_options=s3_options) fsspec_fits.getdata( in_ref_shutter, 1) fsspec_fits.getdata( in_ref_shutter, 1, s3_options=s3_options) ``` ### header.tofile #### 老写法 ```python header.tofile(out_head_path) ``` usage reference: [https://docs.astropy.org/en/stable/io/fits/api/headers.html#astropy.io.fits.Header.tofile](https://docs.astropy.org/en/stable/io/fits/api/headers.html#astropy.io.fits.Header.tofile) #### 新写法 ```python from csst_fs import fsspec_header fsspec_header.tofile(header, out_head_path) fsspec_header.tofile(header, out_head_path, s3_options=s3_options) ``` ### header.fromfile #### 老写法 ```python header.fromfile(filename) ``` usage reference: [https://docs.astropy.org/en/stable/io/fits/api/headers.html#astropy.io.fits.Header.fromfile](https://docs.astropy.org/en/stable/io/fits/api/headers.html#astropy.io.fits.Header.fromfile) #### 新写法 ```python from csst_fs import fsspec_header fsspec_header.fromfile(filename) fsspec_header.fromfile(filename, s3_options=s3_options) ``` ### HDUList.writeto #### 老写法 ```python hdul_img.writeto(hdul_img, out_combined_img, overwrite=True) ``` usage reference: [https://docs.astropy.org/en/stable/io/fits/api/hdulists.html#astropy.io.fits.HDUList.writeto](https://docs.astropy.org/en/stable/io/fits/api/hdulists.html#astropy.io.fits.HDUList.writeto) #### 新写法 ```python from csst_fs import fsspec_HDUList fsspec_HDUList.writeto(hdul_img, out_combined_img, overwrite=True) fsspec_HDUList.writeto(hdul_img, out_combined_img, s3_options=s3_options, overwrite=True) ``` ### table.Table.read #### 老写法 ```python from astropy import table table.Table.read(out_gaia_ldac, hdu=2) ``` usage reference: [https://docs.astropy.org/en/stable/api/astropy.table.Table.html#astropy.table.Table.read](https://docs.astropy.org/en/stable/api/astropy.table.Table.html#astropy.table.Table.read ) #### 新写法 ```python from csst_fs import fsspec_table fsspec_table.read(out_gaia_ldac, hdu=2) fsspec_table.read(out_gaia_ldac, s3_options=s3_options, hdu=2) ``` ### table.Table.read #### 老写法 ```python ps.write(ref, format='fits', overwrite=True) ``` usage reference: [https://docs.astropy.org/en/stable/api/astropy.table.Table.html#astropy.table.Table.write](https://docs.astropy.org/en/stable/api/astropy.table.Table.html#astropy.table.Table.write) #### 新写法 ```python from csst_fs import fsspec_table fsspec_table.write(ps, ref, format='fits', overwrite=True) fsspec_table.write(ps, ref, format='fits', s3_options=s3_options, overwrite=True) ```