language/examples/reference-architectures/product_description_generator_image.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7d9bbf86da5e"
   },
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2bd716bf3e39"
   },
   "source": [
    "# Product Description Generator From Image\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/language/examples/reference-architectures/product_description_generator_image.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/reference-architectures/product_description_generator_image.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/reference-architectures/product_description_generator_image.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d8cd12648da4"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook is a use-case demonstration of creating product descriptions from images. In this notebook, you will use [Fashion Image Dataset](https://github.com/alexeygrigorev/clothing-dataset-small) to create product descriptions for the clothing images.\n",
    "As an initial step, you will deploy the pre-trained [BLIP Image Captioning](https://huggingface.co/Salesforce/blip-image-captioning-base) model on Vertex AI for online prediction. Then you will use the model to caption the images. Later, you will use the captions of product images to produce descriptions using the PaLM model for marketing collaterals. \n",
    "\n",
    "### Objective\n",
    "\n",
    "- Upload the model to [Model Registry](https://cloud.google.com/vertex-ai/docs/model-registry/introduction).\n",
    "- Deploy the model on [Endpoint](https://cloud.google.com/vertex-ai/docs/predictions/using-private-endpoints).\n",
    "- Run online predictions for image captioning.\n",
    "- Run online predictions for PaLM model with the image captions to produce product descriptions.\n",
    "\n",
    "### Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* Vertex AI\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "988ce40b7407"
   },
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b4e98da3ac28"
   },
   "source": [
    "### Install Vertex AI SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cf1e8556f27a"
   },
   "outputs": [],
   "source": [
    "!pip install google-cloud-aiplatform --upgrade --quiet --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b0248b091b6e"
   },
   "source": [
    "***Colab only***: Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "17a3adf1ce04"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d73ffa0c0b83"
   },
   "source": [
    "### Authenticating your notebook environment\n",
    "* If you are using **Colab** to run this notebook, uncomment the cell below and continue.\n",
    "* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cCxVCgQp4YgL"
   },
   "outputs": [],
   "source": [
    "# from google.colab import auth as google_auth\n",
    "# google_auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c2a8dfbe5bfa"
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "13da1828cd44"
   },
   "source": [
    "**Colab only:** Uncomment the following cell to initialize the Vertex AI SDK. For Vertex AI Workbench, you don't need to run this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "32d3683b3260"
   },
   "outputs": [],
   "source": [
    "# import vertexai\n",
    "\n",
    "# PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
    "# vertexai.init(project=PROJECT_ID, location=\"us-central1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0fb2f1b3ce06"
   },
   "outputs": [],
   "source": [
    "import io\n",
    "import os\n",
    "import base64\n",
    "import requests\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "from datetime import datetime\n",
    "from google.cloud import storage\n",
    "from google.cloud import aiplatform\n",
    "from vertexai.language_models import TextGenerationModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ac378ede6e22"
   },
   "source": [
    "### Import models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dada301d0216"
   },
   "outputs": [],
   "source": [
    "generation_model = TextGenerationModel.from_pretrained(\"text-bison@001\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0f826ff482a2"
   },
   "source": [
    "### The following GCS Bucket contains some fashion product image dataset samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9db30f827a65"
   },
   "outputs": [],
   "source": [
    "GCS_BUCKET = \"github-repo\"\n",
    "!gsutil ls gs://$GCS_BUCKET/product_img/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6ca48b699d17"
   },
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "de9882ea89ea"
   },
   "outputs": [],
   "source": [
    "# The pre-built serving docker image. It contains serving scripts and models.\n",
    "SERVE_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai-restricted/vertex-vision-model-garden-dockers/pytorch-transformers-serve\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "10188266a5cd"
   },
   "source": [
    "### Define common functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cac4478ae098"
   },
   "outputs": [],
   "source": [
    "def create_job_name(prefix):\n",
    "    user = os.environ.get(\"USER\")\n",
    "    now = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
    "    job_name = f\"{prefix}-{user}-{now}\"\n",
    "    return job_name\n",
    "\n",
    "\n",
    "def download_image(url):\n",
    "    response = requests.get(url)\n",
    "    return Image.open(BytesIO(response.content))\n",
    "\n",
    "\n",
    "def image_to_base64(image, format=\"JPEG\"):\n",
    "    buffer = BytesIO()\n",
    "    image.save(buffer, format=format)\n",
    "    image_str = base64.b64encode(buffer.getvalue()).decode(\"utf-8\")\n",
    "    return image_str\n",
    "\n",
    "\n",
    "def base64_to_image(image_str):\n",
    "    image = Image.open(BytesIO(base64.b64decode(image_str)))\n",
    "    return image\n",
    "\n",
    "\n",
    "def image_grid(imgs, rows=2, cols=2):\n",
    "    w, h = imgs[0].size\n",
    "    grid = Image.new(\"RGB\", size=(cols * w, rows * h))\n",
    "    for i, img in enumerate(imgs):\n",
    "        grid.paste(img, box=(i % cols * w, i // cols * h))\n",
    "    return grid\n",
    "\n",
    "\n",
    "def deploy_model(model_id, task):\n",
    "    model_name = \"blip-image-captioning\"\n",
    "    endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
    "    serving_env = {\n",
    "        \"MODEL_ID\": model_id,\n",
    "        \"TASK\": task,\n",
    "    }\n",
    "    # If the model_id is a GCS path, use artifact_uri to pass it to serving docker.\n",
    "    artifact_uri = model_id if model_id.startswith(\"gs://\") else None\n",
    "    model = aiplatform.Model.upload(\n",
    "        display_name=model_name,\n",
    "        serving_container_image_uri=SERVE_DOCKER_URI,\n",
    "        serving_container_ports=[7080],\n",
    "        serving_container_predict_route=\"/predictions/transformers_serving\",\n",
    "        serving_container_health_route=\"/ping\",\n",
    "        serving_container_environment_variables=serving_env,\n",
    "        artifact_uri=artifact_uri,\n",
    "    )\n",
    "    model.deploy(\n",
    "        endpoint=endpoint,\n",
    "        machine_type=\"n1-standard-8\",\n",
    "        accelerator_type=\"NVIDIA_TESLA_T4\",\n",
    "        accelerator_count=1,\n",
    "        deploy_request_timeout=1800,\n",
    "    )\n",
    "    return model, endpoint\n",
    "\n",
    "\n",
    "def read_jpeg_image_from_gcs(bucket_name, image_name):\n",
    "    \"\"\"Reads a JPEG image from a Google Cloud Storage (GCS) bucket.\n",
    "\n",
    "    Args:\n",
    "    bucket_name: The name of the GCS bucket that contains the image file.\n",
    "    image_name: The name of the image file in the GCS bucket.\n",
    "\n",
    "    Returns:\n",
    "    The image file as a PIL Image object.\n",
    "    \"\"\"\n",
    "\n",
    "    # Import the Google Cloud Storage client library.\n",
    "\n",
    "    # Create a storage client.\n",
    "    client = storage.Client()\n",
    "\n",
    "    # Get the bucket object.\n",
    "    bucket = client.bucket(bucket_name)\n",
    "\n",
    "    # Get the blob object.\n",
    "    blob = bucket.blob(image_name)\n",
    "\n",
    "    # Read the blob to a bytestring.\n",
    "    image_data = blob.download_as_bytes()\n",
    "\n",
    "    # Decode the bytestring to a PIL Image object.\n",
    "    image = Image.open(io.BytesIO(image_data))\n",
    "\n",
    "    # Return the PIL Image object.\n",
    "    return image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d2d72ecdb8c9"
   },
   "source": [
    "## Upload and deploy models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9448c5f545fa"
   },
   "source": [
    "This section uploads the pre-trained model to Model Registry and deploys it on the Endpoint with 1 T4 GPU.\n",
    "\n",
    "The model deployment step will take ~15 minutes to complete.\n",
    "\n",
    "Once deployed, you can send images to get descriptions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** **Run this cell only once.** As this is a PrivateEndpoint-A maximum of one model can be deployed to each private Endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b4b46c28d8b1"
   },
   "outputs": [],
   "source": [
    "model, endpoint = deploy_model(\n",
    "    model_id=\"Salesforce/blip-image-captioning-base\", task=\"image-to-text\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0c4be5a41cd7"
   },
   "source": [
    "Now you will write a function that will take the image caption generated by the BLIP model and sends it to our PaLM 2 text generation model. Through your prompt, you are expecting it to return a catchy product description that can be used in marketing collaterals. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-iam8Fc5tt-Q"
   },
   "outputs": [],
   "source": [
    "def generate_product_description(model, image_caption, temperature=0):\n",
    "    \"\"\"Ideation example with a Large Language Model\"\"\"\n",
    "    prompt_prefix = \"Imagine you are a digital marketer working for a retail organization. \\\n",
    "                    You are an expert in building detailed and catchy descriptions fro the retail fashion products on your website.\\\n",
    "                    Generate a product description using the following short caption that describes the apparel\"\n",
    "    prompt = prompt_prefix + image_caption\n",
    "    response = model.predict(\n",
    "        prompt,\n",
    "        temperature=temperature,\n",
    "        max_output_tokens=256,\n",
    "        top_k=40,\n",
    "        top_p=0.8,\n",
    "    )\n",
    "    return response.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b6066d4f2ef3"
   },
   "source": [
    "Once the model is defined, test the images and their captions to see how descriptions are generated. You can write a simple for loop that goes through each photo in our source folder and calls our model by passing the caption. You can read through the descriptions of each image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6be655247cb1"
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "for i in range(1, 9):\n",
    "    image_data = read_jpeg_image_from_gcs(\n",
    "        GCS_BUCKET, \"product_img/fashion\" + str(i) + \".jpeg\"\n",
    "    )\n",
    "    # Display the image\n",
    "    plt.imshow(image_data)\n",
    "    plt.axis(\"off\")\n",
    "    plt.show()\n",
    "\n",
    "    instances = [\n",
    "        {\"image\": image_to_base64(image_data)},\n",
    "    ]\n",
    "    preds = endpoint.predict(instances=instances).predictions\n",
    "    print(preds)\n",
    "    product_description = generate_product_description(\n",
    "        model=generation_model, image_caption=preds[0]\n",
    "    )\n",
    "    print(product_description)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "product_description_generator.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-11.m108",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-11:m108"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}