Commit 0543231b authored by Matthias Weidenthaler's avatar Matthias Weidenthaler
Browse files

Fixed details in readme for planned api changes

parent 6312188a
...@@ -7,7 +7,7 @@ This repository provides the following functionalities: ...@@ -7,7 +7,7 @@ This repository provides the following functionalities:
# 1. Read or Download a File from S3 storage # 1. Read or Download a File from S3 storage
Supported are two distinct ways of reading from s3 storage. Supported are two distinct ways of reading from s3 storage.
1) [Download to a local file](#下载) 1) [Download to a local file](#从s3下载到本地)
2) [use open() to get a file object](#open-for-read) 2) [use open() to get a file object](#open-for-read)
## Configuration ## Configuration
...@@ -15,47 +15,56 @@ Supported are two distinct ways of reading from s3 storage. ...@@ -15,47 +15,56 @@ Supported are two distinct ways of reading from s3 storage.
**老写法同时兼容本地nas和云上s3,只要读路径以s3:// 协议开头会自动识别** **老写法同时兼容本地nas和云上s3,只要读路径以s3:// 协议开头会自动识别**
如果需要读S3时,需要传入s3的密钥和endpoint等配置,有两种方法可选 如果需要读S3时,需要传入s3的密钥和endpoint等配置,有两种方法可选
The used s3 bucket is configured through an env variable.
### 方法1 环境变量
执行下面三个环境变量,本文档下面介绍到的所有方法都会尝试读取环境变量以获取配置
```python
s3_options = {
'key': os.getenv('S3_KEY'),
'secret': os.getenv('S3_SECRET'),
'endpoint_url': os.getenv('S3_ENDPOINT_URL')
}
```
### 方法2 每次调用方法时传入 s3_options
在第一个kwargs参数位置指定s3_options, s3_options示例:
```python
s3_options = {
"key": "minioadmin",
"secret": "minioadmin",
"endpoint_url": "http://localhost:9000"
}
```
## 从s3下载到本地 ## 从s3下载到本地
### 下载
```python ```python
def get(key: str, local_path: str):
"""
Download a file/folder from s3 to local storage.
Args:
key: s3 key
local_path: Local path that will be downloaded to.
"""
def info(key: str):
"""
Get information about s3 file.
Args:
key: s3 key
"""
# Example:
from csst_fs import s3_fs from csst_fs import s3_fs
# single file # single file
s3_fs.get('s3://csst-prod/gaia/test/requirements.txt', 'requirements.txt') s3_fs.get('gaia/test/requirements.txt', 'requirements.txt')
s3_fs.get('s3://csst-prod/gaia/test/requirements.txt', 'requirements.txt', s3_options=s3_options)
# folder # folder
s3_fs.get('s3://csst-prod/gaia/data', './', recursive=True) s3_fs.get('gaia/data', './', recursive=True)
s3_fs.get('s3://csst-prod/gaia/data', './', s3_options=s3_options, recursive=True)
# get file or folder info # get file or folder info
s3_fs.info('s3://csst-prod/gaia/data') s3_fs.info('gaia/data')
s3_fs.info('s3://csst-prod/gaia/test/requirements.txt', s3_options=s3_options)
``` ```
### Open for read ## Open for read
```python ```python
def open(key: str):
"""
Get a readonly file object from a file on s3.
Args:
key: s3 key
Returns:
File object of the s3 file.
"""
# Example:
from csst_fs import s3_fs from csst_fs import s3_fs
# open single file (s3 or local) # open single file (s3 or local)
with s3_fs.open('s3://csst-prod/gaia/data') as file: with s3_fs.open('gaia/data') as file:
file.read() file.read()
``` ```
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment