Web_scraping_with_Large_Language_Models_LLM_AnthropicAI_LangChainAI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checkout my [Twitter(@rohanpaul_ai)](https://twitter.com/rohanpaul_ai) for daily LLM bits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatAnthropic\n",
    "from langchain.agents.agent_toolkits import PlayWrightBrowserToolkit\n",
    "\n",
    "# A synchronous browser is available, though it isn't compatible with jupyter.\n",
    "from langchain.tools.playwright.utils import (\n",
    "    create_sync_playwright_browser,\n",
    ")\n",
    "from langchain.agents import initialize_agent, AgentType\n",
    "\n",
    "# In a seperate .env file add your environment variables\n",
    "# ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxxx\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "# Following line generates the chat response using @LangChainAI's ChatAnthropic()\n",
    "# To use ChatAnthropic you should have the @anthropic-ai/sdk package installed,\n",
    "# with the ANTHROPIC_API_KEY environment variable set.\n",
    "llm = ChatAnthropic (model=\"claude-1\", temperature=0, max_tokens_to_sample=2000)\n",
    "\n",
    "# For the above line, I could use any other LLM, e.g., ChatOpenAI(), OpenAI()\n",
    "\n",
    "def main():\n",
    "    sync_browser = create_sync_playwright_browser()\n",
    "    # It's always recommended to instantiate using the from_browser method\n",
    "    toolkit = PlayWrightBrowserToolkit.from_browser (sync_browser=sync_browser)\n",
    "\n",
    "    tools = toolkit.get_tools()\n",
    "\n",
    "    agent_chain = initialize_agent(\n",
    "        tools,\n",
    "        agent = AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
    "        verbose=True\n",
    "    )\n",
    "\n",
    "    result = agent_chain.run(\"Get all links from https://en.wikipedia.org/wiki/Large_language_model \"\n",
    "                         \"only return links starting with https and \"\n",
    "                         \"omit this portion https://en.wikipedia.org\")\n",
    "\n",
    "\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "👉 @AnthropicAI + @LangChainAI\n",
    "\n",
    "💡1) Ask Langchain Agent to scrape data\n",
    "\n",
    "💡2) Get back data in a list\n",
    "\n",
    "And the magical thing is with 100k context window of @AnthropicAI, you dont need to use vectordb as well here.\n",
    "\n",
    "------------\n",
    "\n",
    "➡️ PlayWright Browser Toolkit\n",
    "\n",
    "This toolkit is used to interact with the browser. While other tools (like the Requests tools) are fine for static sites, Browser toolkits let your agent navigate the web and interact with dynamically rendered sites.\n",
    "\n",
    "```py\n",
    "!pip install playwright > /dev/null\n",
    "!pip install  lxml\n",
    "\n",
    "```\n",
    "If this is your first time using playwright, you'll have to install a browser executable.\n",
    "\n",
    "Running `playwright install` by default installs a chromium browser executable.\n",
    "\n",
    "📌 Several of the browser tools are StructuredTool's, meaning they expect multiple arguments. These aren't compatible (out of the box) with agents older than the `STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION`"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}