examples/langchain_azure_openai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to connect to Azure OpenAI service via PromptSail and Langchain SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "First you need to get or create an Azure OpenAI resource. \n",
    "You can do it here: https://portal.azure.com/#view/Microsoft_Azure_ProjectOxford/CognitiveServicesHub/~/OpenAI\n",
    "\n",
    "Click on the chosen resource and than the \"Develop\" button in the middle of the screen.\n",
    "\n",
    "Copy the API key and endpoint and paste it into `.env` file in the `examples` folder. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the connection without PromptSail\n",
    "\n",
    "Import the OpenAI Python SDK and load your API key from the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure OpenAI api key=9b2...d67b3\n",
      "Azure OpenAI api endpoint=https://openai-pr...\n",
      "Azure OpenAI deployment name=gpt-35T...\n",
      "Azure OpenAI api version=2023-07-01-preview\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv, dotenv_values\n",
    "from pprint import pprint\n",
    "\n",
    "\n",
    "config = dotenv_values(\".env\")\n",
    "\n",
    "\n",
    "azure_oai_key = config[\"AZURE_OPENAI_API_KEY\"]\n",
    "api_base_url = config[\"AZURE_OPENAI_ENDPOINT\"]\n",
    "\n",
    "deployment_name = config[\"AZURE_LLM_DEPLOYMENT_NAME\"]\n",
    "api_version = config[\"AZURE_OPENAI_API_VERSION\"]\n",
    "\n",
    "print(\n",
    "    f\"Azure OpenAI api key={azure_oai_key[0:3]}...{azure_oai_key[-5:]}\"\n",
    ")\n",
    "print(\n",
    "    f\"Azure OpenAI api endpoint={api_base_url[0:17]}...\"\n",
    ")\n",
    "print(f\"Azure OpenAI deployment name={deployment_name[0:7]}...\")\n",
    "print(f\"Azure OpenAI api version={api_version}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the direct connection to Azure OpenAI and Langchain SDK.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "#from langchain_community.chat_models import AzureChatOpenAI\n",
    "\n",
    "from langchain.prompts.chat import (\n",
    "    ChatPromptTemplate,\n",
    "    HumanMessagePromptTemplate,\n",
    "    SystemMessagePromptTemplate,\n",
    ")\n",
    "from langchain.schema import HumanMessage, SystemMessage, AIMessage\n",
    "\n",
    "\n",
    "message = [\n",
    "    SystemMessage(\n",
    "        content=\"You are a helpful assistant that help rewirte an jira ticket.\"\n",
    "    ),\n",
    "    HumanMessage(\n",
    "        content=\"Give meaningful title to this bug, RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU X; X GiB total capacity; X GiB already allocated; X MiB free; X cached)\"\n",
    "    ),\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "message_followup = [\n",
    "    SystemMessage(\n",
    "        content=\"You are a helpful assistant that help rewirte an jira ticket.\"\n",
    "    ),\n",
    "    HumanMessage(\n",
    "        content=\"Give meaningful title to this bug, RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU X; X GiB total capacity; X GiB already allocated; X MiB free; X cached)\"\n",
    "    ),\n",
    "    AIMessage(\n",
    "        content=\"RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU X; X GiB total capacity; X GiB already allocated; X MiB free; X cached)\"\n",
    "    ),\n",
    "    HumanMessage(\n",
    "        content=\"The error message is too technical, could you make it more user friendly and poetic?\"\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "chat = AzureChatOpenAI(\n",
    "    openai_api_key=azure_oai_key,\n",
    "    deployment_name=deployment_name,\n",
    "    api_version=api_version,\n",
    "    azure_endpoint=api_base_url,\n",
    "    #base_url=api_base_url,\n",
    "    #model=deployment_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='Bug: CUDA out of memory error occurs when allocating X MiB on GPU X with X GiB total capacity\\n\\nDescription: When attempting to allocate memory on GPU X, a RuntimeError with the message \"CUDA out of memory\" is encountered. The error message provides information about the GPU\\'s total capacity, already allocated memory, free memory, and cached memory. \\n\\nSeverity: High\\n\\nSteps to Reproduce:\\n1. Perform actions that require GPU memory allocation\\n2. Monitor memory usage on GPU X\\n3. Observe RuntimeError: CUDA out of memory when allocating X MiB\\n\\nExpected Results: The memory allocation should succeed without encountering the CUDA out of memory error.\\n\\nActual Results: The RuntimeError: CUDA out of memory error is thrown when trying to allocate X MiB on GPU X.\\n\\nNotes:\\n- Ensure that the system has sufficient free memory to accommodate the allocation\\n- Consider reducing memory usage or optimizing memory allocation in the application to avoid running out of memory on GPU X.')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a request to the AzureOpenAI via PromptSail proxy\n",
    "\n",
    "Run the docker and go to PromptSail UI http://localhost/\n",
    "\n",
    "Create new project with you `project_slug`or edit existing one.\n",
    "\n",
    "Add your own Azure OpenAI provider by editing the project settings, this will map the Azure OpenAI endpoint to new proxy prompt sail URL. \n",
    "\n",
    "Set the `api base url` to your Azure OpenAI endpoint like\n",
    " \n",
    "'https://**azure_openai_resource**.openai.azure.com/'\n",
    " \n",
    " and add meaningfull `deployment name`.\n",
    "\n",
    "\n",
    "In mongo it will create new entry in `ai_providers` array, similar to this one\n",
    "\n",
    "```bash\n",
    "{\n",
    "     ai_providers: [\n",
    "        {\n",
    "            deployment_name: 'azure US deployment',\n",
    "            slug: 'azure-us-deployment',\n",
    "            api_base: 'https://[azure_openai_resource].openai.azure.com/',\n",
    "            description: '',\n",
    "            provider_name: 'Azure OpenAI'\n",
    "        }\n",
    "    ],\n",
    "}\n",
    "```\n",
    "\n",
    "In this case we will use the default `project 2` settings:\n",
    "* with project_slug -> 'project2' \n",
    "* deployment_name -> 'azure-us-deployment'\n",
    "resulting in promptsail proxy url like this: \n",
    "\n",
    "**\"http://localhost:8000/project2/azure-us-deployment\"**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps_api_base = \"http://localhost:8000/project2/azure-prompt-sail-eus2\"\n",
    "\n",
    "\n",
    "chat = AzureChatOpenAI(\n",
    "    openai_api_key=azure_oai_key,\n",
    "    deployment_name=deployment_name,\n",
    "    api_version=api_version,\n",
    "    azure_endpoint=ps_api_base,\n",
    "\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='Bug: RuntimeError: Insufficient CUDA memory. Attempted to allocate X MiB (GPU X; X GiB total capacity; X GiB already allocated; X MiB free; X cached)')"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Azure OpenAI costs calculation and model versions\n",
    "\n",
    "\n",
    "The Azure OpenAI responses do not include the version of the model used to generate the response, request contains the `deployment_name` and response returns just model famili eg, `gpt-3.5-turbo` and not `gpt-3.5-turbo-1106`, this makes impossible to track and calculate costs accurately. To address this issue, you can pass the model_version parameter to the AzureChatOpenAI class, which will append the version to the model name in the output. This allows for easy differentiation between different versions of the model.\n",
    "\n",
    "\n",
    "Ensure that your deployment supports the chosen model version, as different regions support various models and versions. You can verify the supported models and versions in each region by following this\n",
    "[Azure OpenAI Model summary table and region availability](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "ps_api_base_model_version_tag = \"http://localhost:8000/project2/azure-prompt-sail-eus2/?tags=examples,langchain-v0.0.354&target_path=\"\n",
    "\n",
    "#ps_api_base_model_version_tag = \"http://localhost:8000/project2/azure-prompt-sail-eus2\"\n",
    "\n",
    "chat_0613 = AzureChatOpenAI(\n",
    "    openai_api_key=azure_oai_key,\n",
    "    deployment_name=deployment_name,\n",
    "    api_version=api_version,\n",
    "    azure_endpoint=ps_api_base_model_version_tag,\n",
    "    #model_version=\"0613\",\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "ename": "NotFoundError",
     "evalue": "Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNotFoundError\u001b[0m                             Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[41], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43mchat_0613\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmessage_followup\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\langchain_core\\language_models\\chat_models.py:642\u001b[0m, in \u001b[0;36mBaseChatModel.__call__\u001b[1;34m(self, messages, stop, callbacks, **kwargs)\u001b[0m\n\u001b[0;32m    635\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__call__\u001b[39m(\n\u001b[0;32m    636\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[0;32m    637\u001b[0m     messages: List[BaseMessage],\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    640\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any,\n\u001b[0;32m    641\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m BaseMessage:\n\u001b[1;32m--> 642\u001b[0m     generation \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgenerate(\n\u001b[0;32m    643\u001b[0m         [messages], stop\u001b[38;5;241m=\u001b[39mstop, callbacks\u001b[38;5;241m=\u001b[39mcallbacks, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs\n\u001b[0;32m    644\u001b[0m     )\u001b[38;5;241m.\u001b[39mgenerations[\u001b[38;5;241m0\u001b[39m][\u001b[38;5;241m0\u001b[39m]\n\u001b[0;32m    645\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(generation, ChatGeneration):\n\u001b[0;32m    646\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m generation\u001b[38;5;241m.\u001b[39mmessage\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\langchain_core\\language_models\\chat_models.py:383\u001b[0m, in \u001b[0;36mBaseChatModel.generate\u001b[1;34m(self, messages, stop, callbacks, tags, metadata, run_name, **kwargs)\u001b[0m\n\u001b[0;32m    381\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m run_managers:\n\u001b[0;32m    382\u001b[0m             run_managers[i]\u001b[38;5;241m.\u001b[39mon_llm_error(e, response\u001b[38;5;241m=\u001b[39mLLMResult(generations\u001b[38;5;241m=\u001b[39m[]))\n\u001b[1;32m--> 383\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[0;32m    384\u001b[0m flattened_outputs \u001b[38;5;241m=\u001b[39m [\n\u001b[0;32m    385\u001b[0m     LLMResult(generations\u001b[38;5;241m=\u001b[39m[res\u001b[38;5;241m.\u001b[39mgenerations], llm_output\u001b[38;5;241m=\u001b[39mres\u001b[38;5;241m.\u001b[39mllm_output)\n\u001b[0;32m    386\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m res \u001b[38;5;129;01min\u001b[39;00m results\n\u001b[0;32m    387\u001b[0m ]\n\u001b[0;32m    388\u001b[0m llm_output \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_combine_llm_outputs([res\u001b[38;5;241m.\u001b[39mllm_output \u001b[38;5;28;01mfor\u001b[39;00m res \u001b[38;5;129;01min\u001b[39;00m results])\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\langchain_core\\language_models\\chat_models.py:373\u001b[0m, in \u001b[0;36mBaseChatModel.generate\u001b[1;34m(self, messages, stop, callbacks, tags, metadata, run_name, **kwargs)\u001b[0m\n\u001b[0;32m    370\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i, m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(messages):\n\u001b[0;32m    371\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m    372\u001b[0m         results\u001b[38;5;241m.\u001b[39mappend(\n\u001b[1;32m--> 373\u001b[0m             \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_generate_with_cache(\n\u001b[0;32m    374\u001b[0m                 m,\n\u001b[0;32m    375\u001b[0m                 stop\u001b[38;5;241m=\u001b[39mstop,\n\u001b[0;32m    376\u001b[0m                 run_manager\u001b[38;5;241m=\u001b[39mrun_managers[i] \u001b[38;5;28;01mif\u001b[39;00m run_managers \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[0;32m    377\u001b[0m                 \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[0;32m    378\u001b[0m             )\n\u001b[0;32m    379\u001b[0m         )\n\u001b[0;32m    380\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m    381\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m run_managers:\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\langchain_core\\language_models\\chat_models.py:529\u001b[0m, in \u001b[0;36mBaseChatModel._generate_with_cache\u001b[1;34m(self, messages, stop, run_manager, **kwargs)\u001b[0m\n\u001b[0;32m    525\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[0;32m    526\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAsked to cache, but no cache found at `langchain.cache`.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    527\u001b[0m     )\n\u001b[0;32m    528\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m new_arg_supported:\n\u001b[1;32m--> 529\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_generate(\n\u001b[0;32m    530\u001b[0m         messages, stop\u001b[38;5;241m=\u001b[39mstop, run_manager\u001b[38;5;241m=\u001b[39mrun_manager, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs\n\u001b[0;32m    531\u001b[0m     )\n\u001b[0;32m    532\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    533\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_generate(messages, stop\u001b[38;5;241m=\u001b[39mstop, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\langchain_community\\chat_models\\openai.py:435\u001b[0m, in \u001b[0;36mChatOpenAI._generate\u001b[1;34m(self, messages, stop, run_manager, stream, **kwargs)\u001b[0m\n\u001b[0;32m    429\u001b[0m message_dicts, params \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_create_message_dicts(messages, stop)\n\u001b[0;32m    430\u001b[0m params \u001b[38;5;241m=\u001b[39m {\n\u001b[0;32m    431\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mparams,\n\u001b[0;32m    432\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39m({\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mstream\u001b[39m\u001b[38;5;124m\"\u001b[39m: stream} \u001b[38;5;28;01mif\u001b[39;00m stream \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m {}),\n\u001b[0;32m    433\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[0;32m    434\u001b[0m }\n\u001b[1;32m--> 435\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcompletion_with_retry(\n\u001b[0;32m    436\u001b[0m     messages\u001b[38;5;241m=\u001b[39mmessage_dicts, run_manager\u001b[38;5;241m=\u001b[39mrun_manager, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mparams\n\u001b[0;32m    437\u001b[0m )\n\u001b[0;32m    438\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_create_chat_result(response)\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\langchain_community\\chat_models\\openai.py:352\u001b[0m, in \u001b[0;36mChatOpenAI.completion_with_retry\u001b[1;34m(self, run_manager, **kwargs)\u001b[0m\n\u001b[0;32m    350\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Use tenacity to retry the completion call.\"\"\"\u001b[39;00m\n\u001b[0;32m    351\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m is_openai_v1():\n\u001b[1;32m--> 352\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclient\u001b[38;5;241m.\u001b[39mcreate(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m    354\u001b[0m retry_decorator \u001b[38;5;241m=\u001b[39m _create_retry_decorator(\u001b[38;5;28mself\u001b[39m, run_manager\u001b[38;5;241m=\u001b[39mrun_manager)\n\u001b[0;32m    356\u001b[0m \u001b[38;5;129m@retry_decorator\u001b[39m\n\u001b[0;32m    357\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_completion_with_retry\u001b[39m(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\openai\\_utils\\_utils.py:272\u001b[0m, in \u001b[0;36mrequired_args.<locals>.inner.<locals>.wrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    270\u001b[0m             msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mMissing required argument: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mquote(missing[\u001b[38;5;241m0\u001b[39m])\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    271\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(msg)\n\u001b[1;32m--> 272\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m func(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\openai\\resources\\chat\\completions.py:645\u001b[0m, in \u001b[0;36mCompletions.create\u001b[1;34m(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)\u001b[0m\n\u001b[0;32m    596\u001b[0m \u001b[38;5;129m@required_args\u001b[39m([\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmessages\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m], [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmessages\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mstream\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[0;32m    597\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcreate\u001b[39m(\n\u001b[0;32m    598\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    643\u001b[0m     timeout: \u001b[38;5;28mfloat\u001b[39m \u001b[38;5;241m|\u001b[39m httpx\u001b[38;5;241m.\u001b[39mTimeout \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m|\u001b[39m NotGiven \u001b[38;5;241m=\u001b[39m NOT_GIVEN,\n\u001b[0;32m    644\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ChatCompletion \u001b[38;5;241m|\u001b[39m Stream[ChatCompletionChunk]:\n\u001b[1;32m--> 645\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_post\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    646\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m/chat/completions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[0;32m    647\u001b[0m \u001b[43m        \u001b[49m\u001b[43mbody\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmaybe_transform\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    648\u001b[0m \u001b[43m            \u001b[49m\u001b[43m{\u001b[49m\n\u001b[0;32m    649\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmessages\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmessages\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    650\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmodel\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    651\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfrequency_penalty\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfrequency_penalty\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    652\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfunction_call\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunction_call\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    653\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfunctions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunctions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    654\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mlogit_bias\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogit_bias\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    655\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mlogprobs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    656\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmax_tokens\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    657\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mn\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mn\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    658\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mpresence_penalty\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mpresence_penalty\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    659\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mresponse_format\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mresponse_format\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    660\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mseed\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mseed\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    661\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstop\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstop\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    662\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstream\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    663\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtemperature\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtemperature\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    664\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtool_choice\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtool_choice\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    665\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtools\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtools\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    666\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_logprobs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_logprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    667\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_p\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_p\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    668\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43muser\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43muser\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    669\u001b[0m \u001b[43m            \u001b[49m\u001b[43m}\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    670\u001b[0m \u001b[43m            \u001b[49m\u001b[43mcompletion_create_params\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mCompletionCreateParams\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    671\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    672\u001b[0m \u001b[43m        \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmake_request_options\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    673\u001b[0m \u001b[43m            \u001b[49m\u001b[43mextra_headers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_headers\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mextra_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mextra_body\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_body\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\n\u001b[0;32m    674\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    675\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mChatCompletion\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    676\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[0;32m    677\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mStream\u001b[49m\u001b[43m[\u001b[49m\u001b[43mChatCompletionChunk\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    678\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\openai\\_base_client.py:1088\u001b[0m, in \u001b[0;36mSyncAPIClient.post\u001b[1;34m(self, path, cast_to, body, options, files, stream, stream_cls)\u001b[0m\n\u001b[0;32m   1074\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpost\u001b[39m(\n\u001b[0;32m   1075\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[0;32m   1076\u001b[0m     path: \u001b[38;5;28mstr\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1083\u001b[0m     stream_cls: \u001b[38;5;28mtype\u001b[39m[_StreamT] \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[0;32m   1084\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ResponseT \u001b[38;5;241m|\u001b[39m _StreamT:\n\u001b[0;32m   1085\u001b[0m     opts \u001b[38;5;241m=\u001b[39m FinalRequestOptions\u001b[38;5;241m.\u001b[39mconstruct(\n\u001b[0;32m   1086\u001b[0m         method\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpost\u001b[39m\u001b[38;5;124m\"\u001b[39m, url\u001b[38;5;241m=\u001b[39mpath, json_data\u001b[38;5;241m=\u001b[39mbody, files\u001b[38;5;241m=\u001b[39mto_httpx_files(files), \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39moptions\n\u001b[0;32m   1087\u001b[0m     )\n\u001b[1;32m-> 1088\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m cast(ResponseT, \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mopts\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m)\u001b[49m)\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\openai\\_base_client.py:853\u001b[0m, in \u001b[0;36mSyncAPIClient.request\u001b[1;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[0;32m    844\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrequest\u001b[39m(\n\u001b[0;32m    845\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[0;32m    846\u001b[0m     cast_to: Type[ResponseT],\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    851\u001b[0m     stream_cls: \u001b[38;5;28mtype\u001b[39m[_StreamT] \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[0;32m    852\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ResponseT \u001b[38;5;241m|\u001b[39m _StreamT:\n\u001b[1;32m--> 853\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    854\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    855\u001b[0m \u001b[43m        \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    856\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    857\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    858\u001b[0m \u001b[43m        \u001b[49m\u001b[43mremaining_retries\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mremaining_retries\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    859\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\krzys\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\promptsail-hDSOLtZB-py3.10\\lib\\site-packages\\openai\\_base_client.py:930\u001b[0m, in \u001b[0;36mSyncAPIClient._request\u001b[1;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[0;32m    927\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mis_closed:\n\u001b[0;32m    928\u001b[0m         err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mread()\n\u001b[1;32m--> 930\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_make_status_error_from_response(err\u001b[38;5;241m.\u001b[39mresponse) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m    932\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_process_response(\n\u001b[0;32m    933\u001b[0m     cast_to\u001b[38;5;241m=\u001b[39mcast_to,\n\u001b[0;32m    934\u001b[0m     options\u001b[38;5;241m=\u001b[39moptions,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    937\u001b[0m     stream_cls\u001b[38;5;241m=\u001b[39mstream_cls,\n\u001b[0;32m    938\u001b[0m )\n",
      "\u001b[1;31mNotFoundError\u001b[0m: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}"
     ]
    }
   ],
   "source": [
    "chat_0613(message_followup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "prompt-sail-hDSOLtZB-py3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}