Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
csst-pipeline
csst_fs
Commits
0543231b
Commit
0543231b
authored
Aug 21, 2025
by
Matthias Weidenthaler
Browse files
Fixed details in readme for planned api changes
parent
6312188a
Changes
1
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
0543231b
...
@@ -7,7 +7,7 @@ This repository provides the following functionalities:
...
@@ -7,7 +7,7 @@ This repository provides the following functionalities:
# 1. Read or Download a File from S3 storage
# 1. Read or Download a File from S3 storage
Supported are two distinct ways of reading from s3 storage.
Supported are two distinct ways of reading from s3 storage.
1)
[
Download to a local file
](
#
下载
)
1)
[
Download to a local file
](
#
从s3下载到本地
)
2)
[
use open() to get a file object
](
#open-for-read
)
2)
[
use open() to get a file object
](
#open-for-read
)
## Configuration
## Configuration
...
@@ -15,47 +15,56 @@ Supported are two distinct ways of reading from s3 storage.
...
@@ -15,47 +15,56 @@ Supported are two distinct ways of reading from s3 storage.
**老写法同时兼容本地nas和云上s3,只要读路径以s3:// 协议开头会自动识别**
**老写法同时兼容本地nas和云上s3,只要读路径以s3:// 协议开头会自动识别**
如果需要读S3时,需要传入s3的密钥和endpoint等配置,有两种方法可选
如果需要读S3时,需要传入s3的密钥和endpoint等配置,有两种方法可选
The used s3 bucket is configured through an env variable.
### 方法1 环境变量
执行下面三个环境变量,本文档下面介绍到的所有方法都会尝试读取环境变量以获取配置
```
python
s3_options
=
{
'key'
:
os
.
getenv
(
'S3_KEY'
),
'secret'
:
os
.
getenv
(
'S3_SECRET'
),
'endpoint_url'
:
os
.
getenv
(
'S3_ENDPOINT_URL'
)
}
```
### 方法2 每次调用方法时传入 s3_options
在第一个kwargs参数位置指定s3_options, s3_options示例:
```
python
s3_options
=
{
"key"
:
"minioadmin"
,
"secret"
:
"minioadmin"
,
"endpoint_url"
:
"http://localhost:9000"
}
```
## 从s3下载到本地
## 从s3下载到本地
### 下载
```
python
```
python
def
get
(
key
:
str
,
local_path
:
str
):
"""
Download a file/folder from s3 to local storage.
Args:
key: s3 key
local_path: Local path that will be downloaded to.
"""
def
info
(
key
:
str
):
"""
Get information about s3 file.
Args:
key: s3 key
"""
# Example:
from
csst_fs
import
s3_fs
from
csst_fs
import
s3_fs
# single file
# single file
s3_fs
.
get
(
's3://csst-prod/gaia/test/requirements.txt'
,
'requirements.txt'
)
s3_fs
.
get
(
'gaia/test/requirements.txt'
,
'requirements.txt'
)
s3_fs
.
get
(
's3://csst-prod/gaia/test/requirements.txt'
,
'requirements.txt'
,
s3_options
=
s3_options
)
# folder
# folder
s3_fs
.
get
(
's3://csst-prod/gaia/data'
,
'./'
,
recursive
=
True
)
s3_fs
.
get
(
'gaia/data'
,
'./'
,
recursive
=
True
)
s3_fs
.
get
(
's3://csst-prod/gaia/data'
,
'./'
,
s3_options
=
s3_options
,
recursive
=
True
)
# get file or folder info
# get file or folder info
s3_fs
.
info
(
's3://csst-prod/gaia/data'
)
s3_fs
.
info
(
'gaia/data'
)
s3_fs
.
info
(
's3://csst-prod/gaia/test/requirements.txt'
,
s3_options
=
s3_options
)
```
```
##
#
Open for read
## Open for read
```
python
```
python
def
open
(
key
:
str
):
"""
Get a readonly file object from a file on s3.
Args:
key: s3 key
Returns:
File object of the s3 file.
"""
# Example:
from
csst_fs
import
s3_fs
from
csst_fs
import
s3_fs
# open single file (s3 or local)
# open single file (s3 or local)
with
s3_fs
.
open
(
'
s3://csst-prod/
gaia/data'
)
as
file
:
with
s3_fs
.
open
(
'gaia/data'
)
as
file
:
file
.
read
()
file
.
read
()
```
```
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment