Merge branch 'screenshot-scraper-fix' into pre/beta

Author: VinciGit00

CHANGELOG.md

@@ -1,4 +1,129 @@
-## [1.15.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.1-beta.1...v1.15.0-beta.1) (2024-08-23)
+## [1.16.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.1...v1.16.0-beta.1) (2024-08-28)
+
+
+### Features
+
+* add togheterai ([8f615ad](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f615adef320dacdd214a184981384dd05df8171))
+
+
+### chore
+
+* **examples:** create Together AI examples ([34942de](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/34942deca514df53e8aa1c7f96f812ee78b994bf))
+
+## [1.15.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0...v1.15.1) (2024-08-28)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+* update README.md ([5f562b8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5f562b89bd63eba1300afe98572f152a0621b370))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+
+### CI
+
+* **release:** 1.15.0-beta.4 [skip ci] ([c1ce9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c1ce9c69d4ba746d488891d18fa64460e76124bf))
+* **release:** 1.15.0-beta.5 [skip ci] ([22ab45f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/22ab45f6bda3a12ab01c743fd124448a2e26cd46))
+* **release:** 1.15.0-beta.6 [skip ci] ([050fa3f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/050fa3faa02cb2a86ce7c0f61c99e4fa8cf3f9a5))
+* **release:** 1.15.0-beta.7 [skip ci] ([be3f1ec](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/be3f1ec58d6354d583401f51f310f6aac987a393))
+* **release:** 1.15.0-beta.8 [skip ci] ([dbec550](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/dbec55064feac8dfe01290bf82b5b47b013b589d))
+* **release:** 1.15.1-beta.1 [skip ci] ([8f38a6b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f38a6bf15c2138471d7bdb9e0236f02389d93bb))
+
+## [1.15.1-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0...v1.15.1-beta.1) (2024-08-28)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+* update README.md ([5f562b8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5f562b89bd63eba1300afe98572f152a0621b370))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+
+### CI
+
+* **release:** 1.15.0-beta.4 [skip ci] ([c1ce9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c1ce9c69d4ba746d488891d18fa64460e76124bf))
+* **release:** 1.15.0-beta.5 [skip ci] ([22ab45f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/22ab45f6bda3a12ab01c743fd124448a2e26cd46))
+* **release:** 1.15.0-beta.6 [skip ci] ([050fa3f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/050fa3faa02cb2a86ce7c0f61c99e4fa8cf3f9a5))
+* **release:** 1.15.0-beta.7 [skip ci] ([be3f1ec](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/be3f1ec58d6354d583401f51f310f6aac987a393))
+* **release:** 1.15.0-beta.8 [skip ci] ([dbec550](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/dbec55064feac8dfe01290bf82b5b47b013b589d))
+
+## [1.15.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.7...v1.15.0-beta.8) (2024-08-28)
+
+
+### Bug Fixes
+
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+
+## [1.15.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.6...v1.15.0-beta.7) (2024-08-27)
+
+
+### Bug Fixes
+
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+
+## [1.15.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.5...v1.15.0-beta.6) (2024-08-27)
+
+
+### Bug Fixes
+
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+## [1.15.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.4...v1.15.0-beta.5) (2024-08-26)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+
+## [1.15.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.3...v1.15.0-beta.4) (2024-08-26)
+
+## [1.15.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.1...v1.15.0) (2024-08-26)
 
 
 ### Features
@@ -8,15 +133,52 @@
 
 ### Bug Fixes
 
+* abstract graph ([cf1fada](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf1fada36a6716cb0e24bbc5da7509446a964145))
+* **models_tokens:** add llama2 and llama3 sizes explicitly ([b05ec16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b05ec16b252d00c9c9ee7c6d4605b420851c7754))
 * Azure OpenAI issue ([a92b9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a92b9c6970049a4ba9dbdf8eff3eeb7f98c6c639))
+* update abstract graph ([86fe5fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/86fe5fcaf1a6ba28786678874378f07fba1db40f))
 
-## [1.14.1-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0...v1.14.1-beta.1) (2024-08-21)
+
+### CI
+
+* **release:** 1.14.1-beta.1 [skip ci] ([1b48871](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1b488715e698888423eb65f43fdf768bb0729602))
+* **release:** 1.15.0-beta.1 [skip ci] ([06dc640](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/06dc640d44449d1b394829e546a64e38a3d3629e))
+* **release:** 1.15.0-beta.2 [skip ci] ([ab21576](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ab215764353773c5303b88743c6cca4fa7e1b52e))
+* **release:** 1.15.0-beta.3 [skip ci] ([132ee5b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/132ee5b7daf36ef376bfbc63bc6dc7f2332fdd6b))
 
 
 ### Bug Fixes
 
-* **models_tokens:** add llama2 and llama3 sizes explicitly ([b05ec16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b05ec16b252d00c9c9ee7c6d4605b420851c7754))
+* add claude3.5 sonnet ([ee8f8b3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee8f8b31ecfe4ffd311528d2f48cb055e4609d99))
+
+
+### CI
+
+* **release:** 1.14.1 [skip ci] ([88e76ce](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/88e76ceedb39dc1b41222e9a5cb8a6f0d81cadf4))
+
+## [1.14.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0...v1.14.1) (2024-08-24)
+
+
+
+### Bug Fixes
+
+
+* update abstract graph ([86fe5fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/86fe5fcaf1a6ba28786678874378f07fba1db40f))
+
+## [1.15.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.1...v1.15.0-beta.2) (2024-08-23)
+
+
+### Bug Fixes
+
+* abstract graph ([cf1fada](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf1fada36a6716cb0e24bbc5da7509446a964145))
+
+
+### Docs
+
+* added sponsors ([b3a2d0d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b3a2d0d65a41f6e645fac3fc84f702fdf64b951c))
+
 
+#
 ## [1.14.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.3...v1.14.0) (2024-08-20)
 
 

README.md

@@ -32,6 +32,28 @@ playwright install
 
 **Note**: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries 🐱
 
+By the way if you to use not mandatory modules it is necessary to install by yourself with the following command:
+
+### Installing "Other Language Models"
+
+This group allows you to use additional language models like Fireworks, Groq, Anthropic, Together AI, Hugging Face, and Nvidia AI Endpoints.
+```bash
+pip install scrapegraphai[other-language-models]
+
+```
+### Installing "More Semantic Options"
+
+This group includes tools for advanced semantic processing, such as Graphviz.
+```bash
+pip install scrapegraphai[more-semantic-options]
+```
+### Installing "More Browser Options"
+
+This group includes additional browser management options, such as BrowserBase.
+```bash
+pip install scrapegraphai[more-browser-options]
+```
+
 ## 💻 Usage
 There are multiple standard scraping pipelines that can be used to extract information from a website (or local file).
 
@@ -46,7 +68,7 @@ from scrapegraphai.graphs import SmartScraperGraph
 graph_config = {
     "llm": {
         "api_key": "YOUR_OPENAI_APIKEY",
-        "model": "gpt-4o-mini",
+        "model": "openai/gpt-4o-mini",
     },
     "verbose": True,
     "headless": False,
@@ -104,6 +126,15 @@ The documentation for ScrapeGraphAI can be found [here](https://scrapegraph-ai.r
 
 Check out also the Docusaurus [here](https://scrapegraph-doc.onrender.com/).
 
+## 🏆 Sponsors
+<div style="text-align: center;">
+  <a href="https://2ly.link/1zNiz">
+    <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/serp_api_logo.png" alt="SerpAPI" style="width: 10%;">
+  </a>
+  <a href="https://2ly.link/1zNj1">
+    <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/transparent_stat.png" alt="Stats" style="width: 15%;">
+  </a>
+</div>
 
 ## 🤝 Contributing
 

docs/README.md

@@ -33,7 +33,6 @@ markmap:
     - Make graphs run in parallel
     - Scrape only relevant URLs from user prompt
     - Use the multi dimensional DOM tree of the website for retrieval
-  - [Issue #112](https://github.com/VinciGit00/Scrapegraph-ai/issues/112)
 
 - Crawler graph
     - Scrape all the URLs with the same domain in all the pages
@@ -61,4 +60,4 @@ markmap:
 
 - Automatic generation of scraping pipelines from a given prompt
 
-- Create API for the library
+- Create API for the library

examples/anthropic/csv_scraper_graph_multi_haiku.py

@@ -26,8 +26,8 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000},
+        "model": "anthropic/claude-3-haiku-20240307",
+    },
 }
 
 # ************************************************

examples/anthropic/csv_scraper_haiku.py

@@ -32,9 +32,8 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "anthropic/claude-3-haiku-20240307",
+    },
 }
 
 # ************************************************

examples/anthropic/custom_graph_haiku.py

@@ -5,10 +5,9 @@
 import os
 from dotenv import load_dotenv
 
-from langchain_openai import OpenAIEmbeddings
-from langchain_openai import ChatOpenAI
+from langchain_anthropic import ChatAnthropic
 from scrapegraphai.graphs import BaseGraph
-from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
+from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode, RobotsNode
 load_dotenv()
 
 # ************************************************
@@ -19,16 +18,14 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************
 # Define the graph nodes
 # ************************************************
 
-llm_model = OpenAI(graph_config["llm"])
-embedder = OpenAIEmbeddings(api_key=llm_model.openai_api_key)
+llm_model = ChatAnthropic(graph_config["llm"])
 
 # define the nodes for the graph
 robot_node = RobotsNode(
@@ -57,15 +54,6 @@
         "verbose": True,
     }
 )
-rag_node = RAGNode(
-    input="user_prompt & (parsed_doc | doc)",
-    output=["relevant_chunks"],
-    node_config={
-        "llm_model": llm_model,
-        "embedder_model": embedder,
-        "verbose": True,
-    }
-)
 generate_answer_node = GenerateAnswerNode(
     input="user_prompt & (relevant_chunks | parsed_doc | doc)",
     output=["answer"],
@@ -84,14 +72,12 @@
         robot_node,
         fetch_node,
         parse_node,
-        rag_node,
         generate_answer_node,
     ],
     edges=[
         (robot_node, fetch_node),
         (fetch_node, parse_node),
-        (parse_node, rag_node),
-        (rag_node, generate_answer_node)
+        (parse_node, generate_answer_node)
     ],
     entry_point=robot_node
 )

examples/anthropic/json_scraper_haiku.py

@@ -26,9 +26,8 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "anthropic/claude-3-haiku-20240307",
+    },
 }
 
 # ************************************************

examples/anthropic/json_scraper_multi_haiku.py

@@ -11,9 +11,8 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "anthropic/claude-3-haiku-20240307",
+    },
 }
 
 FILE_NAME = "inputs/example.json"

examples/anthropic/pdf_scraper_graph_haiku.py

@@ -14,9 +14,8 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "anthropic/claude-3-haiku-20240307",
+    },
 }
 
 source = """

examples/anthropic/pdf_scraper_multi_haiku.py

@@ -11,9 +11,8 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "anthropic/claude-3-haiku-20240307",
+    },
 }
 
 # ***************