Skip to content

feat(functions): add python s3 form data upload example #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,11 @@ Table of Contents:
| **[Node MultiPart Upload to S3](functions/node-upload-file-s3-multipart/README.md)** <br/> A function to upload file from form-data to S3. | node19 | [Serverless Framework] |
| **[Python ChatBot](functions/python-dependencies/README.md)** <br/> A chatbot example with ChatterBot. | python310 | [Serverless Framework] |
| **[Python Dependencies](functions/python-dependencies/README.md)** <br/> Example showing how to use Python requirements with Serverless Framework. | python310 | [Serverless Framework] |
| **[Python MultiPart Upload to S3](functions/python-upload-file-s3-multipart/README.md)** <br/> A function to upload file from form-data to S3. | python311 | [Python API Framework] |
| **[Redis TLS](functions/redis-tls/README.md)** <br/> How to connect a function to a Scaleway Redis cluster with TLS enabled. | python310 | [Terraform] |
| **[Rust MNIST](functions/rust-mnist/README.md)** <br/> A Rust function to recognize hand-written digits with a simple neural network. | rust165 | [Serverless Framework] |
| **[Terraform Python](functions/terraform-python-example/README.md)** <br/> A Python function deployed with Terraform. | python310 | [Terraform] |
| **[PostgeSQL Node](functions/postgre-sql-node/README.md)** <br/> A Node function to connect and interact with PostgreSQL database. | node18 | [Serverless Framework] |
| **[PostgeSQL Node](functions/postgre-sql-node/README.md)** <br/> A Node function to connect and interact with PostgreSQL database. | node18 | [Serverless Framework] |

### 📦 Containers

Expand All @@ -61,6 +62,7 @@ Table of Contents:

[Serverless Framework]: https://github.com/scaleway/serverless-scaleway-functions
[Terraform]: https://registry.terraform.io/providers/scaleway/scaleway/latest/docs
[Python API Framework]: https://github.com/scaleway/serverless-api-project

## Contributing

Expand Down
12 changes: 8 additions & 4 deletions functions/node-upload-file-s3-multipart/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Node function to read files and upload to S3

This function does the following steps :
This function does the following steps:

* Read a file from an HTTP request
* Send the file to an S3 bucket

## Requirements

If you want to enable S3 upload, ensure to create a bucket and have the following secrets variables available in your environment:
Ensure to create a bucket and have the following secrets variables available in your environment:

```env
S3_REGION = # Default: fr-par
Expand All @@ -16,8 +16,12 @@ SCW_SECRET_KEY =
BUCKET_NAME =
```

## Running

To call the function (replace `README.md` with the file you want to upload):

```sh
curl -X POST -F "data=@README.md" -F "another=@package.json" http://localhost:8080
```console
serverless deploy

curl -X POST -F "data=@README.md" -F "another=@package.json"
```
45 changes: 45 additions & 0 deletions functions/python-upload-file-s3-multipart/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Python function to upload files to S3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Very useful example.


This function does the following steps:

* Read a file from an HTTP request form
* Send the file to long-term storage with Glacier for S3

## Requirements

This example uses the [Python API Framework](https://github.com/scaleway/serverless-api-project) to deploy the function.

If needed, create a bucket and provide the following variables in your environment:

```env
export SCW_ACCESS_KEY =
export SCW_SECRET_KEY =
export BUCKET_NAME =
```

## Running

### Running locally

This examples uses [Serverless Functions Python Framework](https://github.com/scaleway/serverless-functions-python) and can be executed locally:

```console
pip install -r requirements-dev.txt

python app.py
```

The upload endpoint allows you to upload files to Glacier via the `file` form-data key:

```console
echo -e 'Hello world!\n My contents will be stored in a bunker!' > myfile.dat
curl -F file=@myfile.dat localhost:8080
```

### Deploying with the API Framework

Deployment can be done with `scw_serverless`:

```console
scw_serverless deploy app.py
```
75 changes: 75 additions & 0 deletions functions/python-upload-file-s3-multipart/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
from typing import TYPE_CHECKING
import logging
import os

from scw_serverless import Serverless
if TYPE_CHECKING:
from scaleway_functions_python.framework.v1.hints import Context, Event, Response

import boto3
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import ValueTarget

SCW_ACCESS_KEY = os.environ["SCW_ACCESS_KEY"]
SCW_SECRET_KEY = os.environ["SCW_SECRET_KEY"]
BUCKET_NAME = os.environ["BUCKET_NAME"]

# Files will be uploaded to cold storage
# See: https://www.scaleway.com/en/glacier-cold-storage/
STORAGE_CLASS = "GLACIER"

app = Serverless(
"s3-utilities",
secret={
"SCW_ACCESS_KEY": SCW_ACCESS_KEY,
"SCW_SECRET_KEY": SCW_SECRET_KEY,
},
env={
"BUCKET_NAME": BUCKET_NAME,
"PYTHONUNBUFFERED": "1",
},
)

s3 = boto3.resource(
"s3",
region_name="fr-par",
use_ssl=True,
endpoint_url="https://s3.fr-par.scw.cloud",
aws_access_key_id=SCW_ACCESS_KEY,
aws_secret_access_key=SCW_SECRET_KEY,
)

bucket = s3.Bucket(BUCKET_NAME)

logging.basicConfig(level=logging.INFO)


@app.func()
def upload(event: "Event", _context: "Context") -> "Response":
"""Upload form data to S3 Glacier."""

headers = event["headers"]
parser = StreamingFormDataParser(headers=headers)

# Defined a target to handle files uploaded with the "file" key
target = ValueTarget()
parser.register("file", target)

body: str = event["body"]
parser.data_received(body.encode("utf-8"))

if not (len(target.value) > 0 and target.multipart_filename):
return {"statusCode": 400}

name = target.multipart_filename

logging.info("Uploading file %s to Glacier on %s", name, bucket.name)
bucket.put_object(Key=name, Body=target.value, StorageClass=STORAGE_CLASS)

return {"statusCode": 200, "body": f"Successfully uploaded {name} to bucket!"}


if __name__ == "__main__":
from scaleway_functions_python import local

local.serve_handler(upload)
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-r requirements.txt

scaleway-functions-python~=0.1.0
3 changes: 3 additions & 0 deletions functions/python-upload-file-s3-multipart/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
scw-serverless~=0.0.4
boto3~=1.26
streaming_form_data~=1.11.0