ScrapeGraphAI · f-aguzzi · Aug 28, 2024 · Aug 25, 2024 · Aug 25, 2024 · Aug 25, 2024
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,81 @@
+## [1.15.1-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0...v1.15.1-beta.1) (2024-08-28)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+* update README.md ([5f562b8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5f562b89bd63eba1300afe98572f152a0621b370))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+
+### CI
+
+* **release:** 1.15.0-beta.4 [skip ci] ([c1ce9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c1ce9c69d4ba746d488891d18fa64460e76124bf))
+* **release:** 1.15.0-beta.5 [skip ci] ([22ab45f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/22ab45f6bda3a12ab01c743fd124448a2e26cd46))
+* **release:** 1.15.0-beta.6 [skip ci] ([050fa3f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/050fa3faa02cb2a86ce7c0f61c99e4fa8cf3f9a5))
+* **release:** 1.15.0-beta.7 [skip ci] ([be3f1ec](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/be3f1ec58d6354d583401f51f310f6aac987a393))
+* **release:** 1.15.0-beta.8 [skip ci] ([dbec550](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/dbec55064feac8dfe01290bf82b5b47b013b589d))
+
+## [1.15.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.7...v1.15.0-beta.8) (2024-08-28)
+
+
+### Bug Fixes
+
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+
+## [1.15.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.6...v1.15.0-beta.7) (2024-08-27)
+
+
+### Bug Fixes
+
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+
+## [1.15.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.5...v1.15.0-beta.6) (2024-08-27)
+
+
+### Bug Fixes
+
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+## [1.15.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.4...v1.15.0-beta.5) (2024-08-26)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+
+## [1.15.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.3...v1.15.0-beta.4) (2024-08-26)
+
 ## [1.15.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.1...v1.15.0) (2024-08-26)
 
 
@@ -21,47 +99,39 @@
 * **release:** 1.15.0-beta.2 [skip ci] ([ab21576](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ab215764353773c5303b88743c6cca4fa7e1b52e))
 * **release:** 1.15.0-beta.3 [skip ci] ([132ee5b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/132ee5b7daf36ef376bfbc63bc6dc7f2332fdd6b))
 
-## [1.15.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.2...v1.15.0-beta.3) (2024-08-24)
-
-
 
 ### Bug Fixes
 
-* update abstract graph ([86fe5fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/86fe5fcaf1a6ba28786678874378f07fba1db40f))
-
-## [1.15.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.1...v1.15.0-beta.2) (2024-08-23)
-
+* add claude3.5 sonnet ([ee8f8b3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee8f8b31ecfe4ffd311528d2f48cb055e4609d99))
 
-### Bug Fixes
 
-* abstract graph ([cf1fada](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf1fada36a6716cb0e24bbc5da7509446a964145))
+### CI
 
+* **release:** 1.14.1 [skip ci] ([88e76ce](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/88e76ceedb39dc1b41222e9a5cb8a6f0d81cadf4))
 
+## [1.14.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0...v1.14.1) (2024-08-24)
 
-### Docs
 
-* added sponsors ([b3a2d0d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b3a2d0d65a41f6e645fac3fc84f702fdf64b951c))
 
-## [1.15.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.1-beta.1...v1.15.0-beta.1) (2024-08-23)
+### Bug Fixes
 
 
-### Features
+* update abstract graph ([86fe5fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/86fe5fcaf1a6ba28786678874378f07fba1db40f))
 
-* ligthweigthing the library ([62f32e9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/62f32e994bcb748dfef4f7e1b2e5213a989c33cc))
+## [1.15.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.1...v1.15.0-beta.2) (2024-08-23)
 
 
 ### Bug Fixes
 
-* Azure OpenAI issue ([a92b9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a92b9c6970049a4ba9dbdf8eff3eeb7f98c6c639))
-
-## [1.14.1-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0...v1.14.1-beta.1) (2024-08-21)
+* abstract graph ([cf1fada](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf1fada36a6716cb0e24bbc5da7509446a964145))
 
 
-### Bug Fixes
+### Docs
 
-* **models_tokens:** add llama2 and llama3 sizes explicitly ([b05ec16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b05ec16b252d00c9c9ee7c6d4605b420851c7754))
+* added sponsors ([b3a2d0d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b3a2d0d65a41f6e645fac3fc84f702fdf64b951c))
 
 
+#
 ## [1.14.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.3...v1.14.0) (2024-08-20)
 
 

diff --git a/...opic/csv_scraper_graph_multi_anthropic.py → ...nthropic/csv_scraper_graph_multi_haiku.py b/...opic/csv_scraper_graph_multi_anthropic.py → ...nthropic/csv_scraper_graph_multi_haiku.py
@@ -27,7 +27,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000},
+    },
 }
 
 # ************************************************

diff --git a/examples/anthropic/csv_scraper_anthropic.py → examples/anthropic/csv_scraper_haiku.py b/examples/anthropic/csv_scraper_anthropic.py → examples/anthropic/csv_scraper_haiku.py
@@ -33,8 +33,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/examples/anthropic/custom_graph_anthropic.py → examples/anthropic/custom_graph_haiku.py b/examples/anthropic/custom_graph_anthropic.py → examples/anthropic/custom_graph_haiku.py
@@ -5,10 +5,9 @@
 import os
 from dotenv import load_dotenv
 
-from langchain_openai import OpenAIEmbeddings
-from langchain_openai import ChatOpenAI
+from langchain_anthropic import ChatAnthropic
 from scrapegraphai.graphs import BaseGraph
-from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
+from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode, RobotsNode
 load_dotenv()
 
 # ************************************************
@@ -18,17 +17,15 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "claude-3-haiku-20240307",
+    },
 }
 
 # ************************************************
 # Define the graph nodes
 # ************************************************
 
-llm_model = OpenAI(graph_config["llm"])
-embedder = OpenAIEmbeddings(api_key=llm_model.openai_api_key)
+llm_model = ChatAnthropic(graph_config["llm"])
 
 # define the nodes for the graph
 robot_node = RobotsNode(
@@ -57,15 +54,6 @@
         "verbose": True,
     }
 )
-rag_node = RAGNode(
-    input="user_prompt & (parsed_doc | doc)",
-    output=["relevant_chunks"],
-    node_config={
-        "llm_model": llm_model,
-        "embedder_model": embedder,
-        "verbose": True,
-    }
-)
 generate_answer_node = GenerateAnswerNode(
     input="user_prompt & (relevant_chunks | parsed_doc | doc)",
     output=["answer"],
@@ -84,14 +72,12 @@
         robot_node,
         fetch_node,
         parse_node,
-        rag_node,
         generate_answer_node,
     ],
     edges=[
         (robot_node, fetch_node),
         (fetch_node, parse_node),
-        (parse_node, rag_node),
-        (rag_node, generate_answer_node)
+        (parse_node, generate_answer_node)
     ],
     entry_point=robot_node
 )

diff --git a/examples/anthropic/json_scraper_anthropic.py → examples/anthropic/json_scraper_haiku.py b/examples/anthropic/json_scraper_anthropic.py → examples/anthropic/json_scraper_haiku.py
@@ -27,8 +27,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/...anthropic/json_scraper_multi_anthropic.py → ...les/anthropic/json_scraper_multi_haiku.py b/...anthropic/json_scraper_multi_anthropic.py → ...les/anthropic/json_scraper_multi_haiku.py
@@ -12,8 +12,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 FILE_NAME = "inputs/example.json"

diff --git a/.../anthropic/pdf_scraper_graph_anthropic.py → ...ples/anthropic/pdf_scraper_graph_haiku.py b/.../anthropic/pdf_scraper_graph_anthropic.py → ...ples/anthropic/pdf_scraper_graph_haiku.py
@@ -15,8 +15,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 source = """

diff --git a/.../anthropic/pdf_scraper_multi_anthropic.py → ...ples/anthropic/pdf_scraper_multi_haiku.py b/.../anthropic/pdf_scraper_multi_anthropic.py → ...ples/anthropic/pdf_scraper_multi_haiku.py
@@ -12,8 +12,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ***************

diff --git a/.../anthropic/scrape_plain_text_anthropic.py → ...ples/anthropic/scrape_plain_text_haiku.py b/.../anthropic/scrape_plain_text_anthropic.py → ...ples/anthropic/scrape_plain_text_haiku.py
@@ -29,8 +29,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/...s/anthropic/script_generator_anthropic.py → examples/anthropic/script_generator_haiku.py b/...s/anthropic/script_generator_anthropic.py → examples/anthropic/script_generator_haiku.py
@@ -17,8 +17,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/...ropic/script_multi_generator_anthropic.py → ...anthropic/script_multi_generator_haiku.py b/...ropic/script_multi_generator_anthropic.py → ...anthropic/script_multi_generator_haiku.py
@@ -17,9 +17,8 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
-        "library": "beautifulsoup"
+    },
+    "library": "beautifulsoup"
 }
 
 # ************************************************

diff --git a/examples/anthropic/search_graph_anthropic.py → examples/anthropic/search_graph_haiku.py b/examples/anthropic/search_graph_anthropic.py → examples/anthropic/search_graph_haiku.py
@@ -16,8 +16,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/...nthropic/search_graph_schema_anthropic.py → ...es/anthropic/search_graph_schema_haiku.py b/...nthropic/search_graph_schema_anthropic.py → ...es/anthropic/search_graph_schema_haiku.py
@@ -28,8 +28,7 @@ class Dishes(BaseModel):
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/.../anthropic/search_link_graph_anthropic.py → ...ples/anthropic/search_link_graph_haiku.py b/.../anthropic/search_link_graph_anthropic.py → ...ples/anthropic/search_link_graph_haiku.py
@@ -14,26 +14,11 @@
 
 load_dotenv()
 
-llm_model_instance = AzureChatOpenAI(
-    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
-    azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
-)
-
-embedder_model_instance = AzureOpenAIEmbeddings(
-    azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
-    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
-)
-
-# ************************************************
-# Create the SmartScraperGraph instance and run it
-# ************************************************
-
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

diff --git a/...ples/anthropic/smart_scraper_anthropic.py → examples/anthropic/smart_scraper_haiku.py b/...ples/anthropic/smart_scraper_anthropic.py → examples/anthropic/smart_scraper_haiku.py
@@ -20,8 +20,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 smart_scraper_graph = SmartScraperGraph(

diff --git a/...nthropic/smart_scraper_multi_anthropic.py → ...es/anthropic/smart_scraper_multi_haiku.py b/...nthropic/smart_scraper_multi_anthropic.py → ...es/anthropic/smart_scraper_multi_haiku.py
@@ -18,8 +18,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # *******************************************************

diff --git a/...thropic/smart_scraper_schema_anthropic.py → ...s/anthropic/smart_scraper_schema_haiku.py b/...thropic/smart_scraper_schema_anthropic.py → ...s/anthropic/smart_scraper_schema_haiku.py
@@ -34,7 +34,7 @@ class Projects(BaseModel):
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000},
+    },
 }
 
 smart_scraper_graph = SmartScraperGraph(

diff --git a/...opic/xml_scraper_graph_multi_anthropic.py → ...nthropic/xml_scraper_graph_multi_haiku.py b/...opic/xml_scraper_graph_multi_anthropic.py → ...nthropic/xml_scraper_graph_multi_haiku.py
@@ -27,7 +27,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000},
+    },
 }
 
 # ************************************************

diff --git a/examples/anthropic/xml_scraper_anthropic.py → examples/anthropic/xml_scraper_haiku.py b/examples/anthropic/xml_scraper_anthropic.py → examples/anthropic/xml_scraper_haiku.py
@@ -27,8 +27,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************