From 0ee16364882ee480ec055b23d17dfc42e4f2edc4 Mon Sep 17 00:00:00 2001 From: Sahdev Zala Date: Fri, 10 Nov 2023 00:37:03 -0500 Subject: [PATCH 1/2] Tutorial on using TorchServe on Vertex AI This new tutorial is based on the Vertex AI documentation. The PR provides a similar doc as a tutorial hosted locally. Fixes #2346 Signed-off-by: Sahdev Zala --- index.rst | 8 + .../torchserve_vertexai_tutorial.rst | 141 ++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 intermediate_source/torchserve_vertexai_tutorial.rst diff --git a/index.rst b/index.rst index 21c13022685..e2c0012e487 100644 --- a/index.rst +++ b/index.rst @@ -637,6 +637,13 @@ What's new in PyTorch tutorials? :link: beginner/knowledge_distillation_tutorial.html :tags: Model-Optimization,Image/Video +.. customcarditem:: + :header: Deploying a PyTorch Stable Diffusion model as a Vertex AI Endpoint + :card_description: Learn how to deploy model in Vertex AI with TorchServe + :image: _static/img/thumbnails/cropped/generic-pytorch-logo.png + :link: intermediate/torchserve_vertexai_tutorial.rst + :tags: Model-Optimization,Production + .. Parallel-and-Distributed-Training @@ -1042,6 +1049,7 @@ Additional Resources intermediate/inductor_debug_cpu intermediate/scaled_dot_product_attention_tutorial beginner/knowledge_distillation_tutorial + intermediate/torchserve_vertexai_tutorial.rst .. toctree:: :maxdepth: 2 diff --git a/intermediate_source/torchserve_vertexai_tutorial.rst b/intermediate_source/torchserve_vertexai_tutorial.rst new file mode 100644 index 00000000000..516a2b4e34b --- /dev/null +++ b/intermediate_source/torchserve_vertexai_tutorial.rst @@ -0,0 +1,141 @@ +Deploying a PyTorch Stable Diffusion model as a Vertex AI Endpoint +================================================================== + +Deploying large models, like Stable Diffusion, can be challenging and time-consuming. +In this tutorial, we will show how you can streamline the deployment of a PyTorch Stable Diffusion +model by leveraging Vertex AI. PyTorch is the framework used by Stability AI on Stable +Diffusion v1.5. Vertex AI is a fully-managed machine learning platform with tools and +infrastructure designed to help ML practitioners accelerate and scale ML in production with +the benefit of open-source frameworks like PyTorch. In four steps you can deploy a PyTorch +Stable Diffusion model (v1.5). + +Deploying your Stable Diffusion model on a Vertex AI Endpoint can be done in four steps: + +* Create a custom TorchServe handler. + +* Upload model artifacts to Google Cloud Storage (GCS). + +* Create a Vertex AI model with the model artifacts and a prebuilt PyTorch container image. + +* Deploy the Vertex AI model onto an endpoint. + +Let’s have a look at each step in more detail. You can follow and implement the steps using the +`Notebook example `__. + +NOTE: please keep in mind that this tutorial requires a billable Vertex AI as explained in more details in the notebook example. + +Create a custom TorchServe handler +---------------------------------- + +TorchServe is an easy and flexible tool for serving PyTorch models. The model deployed to Vertex AI +uses TorchServe to handle requests and return responses from the model. You must create a custom +TorchServe handler to include in the model artifacts uploaded to Vertex AI. Include the handler file in the +directory with the other model artifacts, like this: `model_artifacts/handler.py`. + +After creating the handler file, you must package the handler as a model archiver (MAR) file. +The output file must be named `model.mar`. + + +.. code:: shell + + !torch-model-archiver \ + -f \ + --model-name \ + --version 1.0 \ + --handler model_artifacts/handler.py \ + --export-path model_artifacts + +Upload model artifacts to Google Cloud Storage (GCS) +---------------------------------------------------- + +In this step we are uploading +`model artifacts `__ +to GCS, like the model file or handler. The advantage of storing your artifacts on GCS is that you can +track the artifacts in a central bucket. + + +.. code:: shell + + BUCKET_NAME = "your-bucket-name-unique" # @param {type:"string"} + BUCKET_URI = f"gs://{BUCKET_NAME}/" + + # Will copy the artifacts into the bucket + !gsutil cp -r model_artifacts $BUCKET_URI + +Create a Vertex AI model with the model artifacts and a prebuilt PyTorch container image +---------------------------------------------------------------------------------------- + +Once you've uploaded the model artifacts into a GCS bucket, you can upload your PyTorch model to +`Vertex AI Model Registry `__. +From the Vertex AI Model Registry, you have an overview of your models +so you can better organize, track, and train new versions. For this you can use the +`Vertex AI SDK `__ +and this +`pre-built PyTorch container `__. + + +.. code:: shell + + from google.cloud import aiplatform as vertexai + PYTORCH_PREDICTION_IMAGE_URI = ( + "us-docker.pkg.dev/vertex-ai/prediction/pytorch-gpu.1-12:latest" + ) + MODEL_DISPLAY_NAME = "stable_diffusion_1_5-unique" + MODEL_DESCRIPTION = "stable_diffusion_1_5 container" + + vertexai.init(project='your_project', location='us-central1', staging_bucket=BUCKET_NAME) + + model = aiplatform.Model.upload( + display_name=MODEL_DISPLAY_NAME, + description=MODEL_DESCRIPTION, + serving_container_image_uri=PYTORCH_PREDICTION_IMAGE_URI, + artifact_uri=BUCKET_URI, + ) + +Deploy the Vertex AI model onto an endpoint +------------------------------------------- + +Once the model has been uploaded to Vertex AI Model Registry you can then take it and deploy +it to an Vertex AI Endpoint. For this you can use the Console or the Vertex AI SDK. In this +example you will deploy the model on a NVIDIA Tesla P100 GPU and n1-standard-8 machine. You can +specify your machine type. + + +.. code:: shell + + endpoint = aiplatform.Endpoint.create(display_name=ENDPOINT_DISPLAY_NAME) + + model.deploy( + endpoint=endpoint, + deployed_model_display_name=MODEL_DISPLAY_NAME, + machine_type="n1-standard-8", + accelerator_type="NVIDIA_TESLA_P100", + accelerator_count=1, + traffic_percentage=100, + deploy_request_timeout=1200, + sync=True, + ) + +If you follow the +`notebook `__ +you can also get online predictions using the Vertex AI SDK as shown in the following snippet. + + +.. code:: shell + + instances = [{"prompt": "An examplePup dog with a baseball jersey."}] + response = endpoint.predict(instances=instances) + + with open("img.jpg", "wb") as g: + g.write(base64.b64decode(response.predictions[0])) + + display.Image("img.jpg") + +Create a Vertex AI model with the model artifacts and a prebuilt PyTorch container image + +More resources +-------------- + +This tutorial was created using the vendor documentation. To refer to the original documentation on the vendor site, please see +`torchserve example `__. + From 60edb06fd9e709b8c44d326cea0c24d398a74754 Mon Sep 17 00:00:00 2001 From: sekyonda <127536312+sekyondaMeta@users.noreply.github.com> Date: Tue, 14 Nov 2023 13:45:41 -0500 Subject: [PATCH 2/2] Move to recipes Moving this tutorial to recipes folder as it is more of a recipe. --- index.rst | 11 ++------ recipes_source/recipes_index.rst | 9 +++++++ .../torchserve_vertexai_tutorial.rst | 25 +++++++++++-------- 3 files changed, 25 insertions(+), 20 deletions(-) rename {intermediate_source => recipes_source}/torchserve_vertexai_tutorial.rst (89%) diff --git a/index.rst b/index.rst index e2c0012e487..91006129411 100644 --- a/index.rst +++ b/index.rst @@ -286,7 +286,7 @@ What's new in PyTorch tutorials? :header: Introduction to ONNX Registry :card_description: Demonstrate end-to-end how to address unsupported operators by using ONNX Registry. :image: _static/img/thumbnails/cropped/Exporting-PyTorch-Models-to-ONNX-Graphs.png - :link: advanced/onnx_registry_tutorial.html + :link: advanced/onnx_registry_tutorial.html :tags: Production,ONNX,Backends .. Reinforcement Learning @@ -637,13 +637,6 @@ What's new in PyTorch tutorials? :link: beginner/knowledge_distillation_tutorial.html :tags: Model-Optimization,Image/Video -.. customcarditem:: - :header: Deploying a PyTorch Stable Diffusion model as a Vertex AI Endpoint - :card_description: Learn how to deploy model in Vertex AI with TorchServe - :image: _static/img/thumbnails/cropped/generic-pytorch-logo.png - :link: intermediate/torchserve_vertexai_tutorial.rst - :tags: Model-Optimization,Production - .. Parallel-and-Distributed-Training @@ -1049,7 +1042,7 @@ Additional Resources intermediate/inductor_debug_cpu intermediate/scaled_dot_product_attention_tutorial beginner/knowledge_distillation_tutorial - intermediate/torchserve_vertexai_tutorial.rst + .. toctree:: :maxdepth: 2 diff --git a/recipes_source/recipes_index.rst b/recipes_source/recipes_index.rst index 10a6ca3fe33..5dc55874459 100644 --- a/recipes_source/recipes_index.rst +++ b/recipes_source/recipes_index.rst @@ -324,6 +324,15 @@ Recipes are bite-sized, actionable examples of how to use specific PyTorch featu :link: ../recipes/DCP_tutorial.html :tags: Distributed-Training +.. TorchServe + +.. customcarditem:: + :header: Deploying a PyTorch Stable Diffusion model as a Vertex AI Endpoint + :card_description: Learn how to deploy model in Vertex AI with TorchServe + :image: ../_static/img/thumbnails/cropped/generic-pytorch-logo.png + :link: ../recipes/torchserve_vertexai_tutorial.html + :tags: Production + .. End of tutorial card section .. raw:: html diff --git a/intermediate_source/torchserve_vertexai_tutorial.rst b/recipes_source/torchserve_vertexai_tutorial.rst similarity index 89% rename from intermediate_source/torchserve_vertexai_tutorial.rst rename to recipes_source/torchserve_vertexai_tutorial.rst index 516a2b4e34b..9c748e7b8c1 100644 --- a/intermediate_source/torchserve_vertexai_tutorial.rst +++ b/recipes_source/torchserve_vertexai_tutorial.rst @@ -2,12 +2,16 @@ Deploying a PyTorch Stable Diffusion model as a Vertex AI Endpoint ================================================================== Deploying large models, like Stable Diffusion, can be challenging and time-consuming. -In this tutorial, we will show how you can streamline the deployment of a PyTorch Stable Diffusion -model by leveraging Vertex AI. PyTorch is the framework used by Stability AI on Stable + +In this recipe, we will show how you can streamline the deployment of a PyTorch Stable Diffusion +model by leveraging Vertex AI. + +PyTorch is the framework used by Stability AI on Stable Diffusion v1.5. Vertex AI is a fully-managed machine learning platform with tools and infrastructure designed to help ML practitioners accelerate and scale ML in production with -the benefit of open-source frameworks like PyTorch. In four steps you can deploy a PyTorch -Stable Diffusion model (v1.5). +the benefit of open-source frameworks like PyTorch. + +In four steps you can deploy a PyTorch Stable Diffusion model (v1.5). Deploying your Stable Diffusion model on a Vertex AI Endpoint can be done in four steps: @@ -22,14 +26,14 @@ Deploying your Stable Diffusion model on a Vertex AI Endpoint can be done in fou Let’s have a look at each step in more detail. You can follow and implement the steps using the `Notebook example `__. -NOTE: please keep in mind that this tutorial requires a billable Vertex AI as explained in more details in the notebook example. +NOTE: Please keep in mind that this recipe requires a billable Vertex AI as explained in more details in the notebook example. Create a custom TorchServe handler ---------------------------------- TorchServe is an easy and flexible tool for serving PyTorch models. The model deployed to Vertex AI -uses TorchServe to handle requests and return responses from the model. You must create a custom -TorchServe handler to include in the model artifacts uploaded to Vertex AI. Include the handler file in the +uses TorchServe to handle requests and return responses from the model. +You must create a custom TorchServe handler to include in the model artifacts uploaded to Vertex AI. Include the handler file in the directory with the other model artifacts, like this: `model_artifacts/handler.py`. After creating the handler file, you must package the handler as a model archiver (MAR) file. @@ -70,7 +74,7 @@ Once you've uploaded the model artifacts into a GCS bucket, you can upload your From the Vertex AI Model Registry, you have an overview of your models so you can better organize, track, and train new versions. For this you can use the `Vertex AI SDK `__ -and this +and this `pre-built PyTorch container `__. @@ -97,7 +101,7 @@ Deploy the Vertex AI model onto an endpoint Once the model has been uploaded to Vertex AI Model Registry you can then take it and deploy it to an Vertex AI Endpoint. For this you can use the Console or the Vertex AI SDK. In this -example you will deploy the model on a NVIDIA Tesla P100 GPU and n1-standard-8 machine. You can +example you will deploy the model on a NVIDIA Tesla P100 GPU and n1-standard-8 machine. You can specify your machine type. @@ -116,7 +120,7 @@ specify your machine type. sync=True, ) -If you follow the +If you follow this `notebook `__ you can also get online predictions using the Vertex AI SDK as shown in the following snippet. @@ -138,4 +142,3 @@ More resources This tutorial was created using the vendor documentation. To refer to the original documentation on the vendor site, please see `torchserve example `__. -