Merge branch 'main' into pre/beta

Author: f-aguzzi

CHANGELOG.md

@@ -1,3 +1,60 @@
+## [1.12.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.3...v1.12.0) (2024-08-06)
+
+
+### Features
+
+* add generate_answer node paralellization ([0c4b290](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/0c4b2908d98efbb2b0a6faf68618a801d726bb5f))
+* add integration in the abstract grapgh ([5ecdbe7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5ecdbe715f4bb223fa1be834fda07ccea2a51cb9))
+* fix tests ([1db164e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1db164e9e682eefbc1414989a043fefa2e9009c2))
+* intregration of firebase ([4caed54](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4caed545e5030460b2d5e46f9ad90546ce36f0ee))
+* pdate models_tokens.py ([377d679](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/377d679eecd62611c0c9a3cba8202c6f0719ed31))
+* refactoring of the code ([9355507](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9355507a2dc73342f325b6649e871df48ae13567))
+
+
+### Bug Fixes
+
+* abstract_graph and removed unused embeddings ([0b4cfd6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/0b4cfd6522dcad0eb418f0badd0f7824a1efd534))
+* add llama 3.1 ([f336c95](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f336c95c2d1833d1f829d61ae7fa415ac2caf250))
+* fixed bug on fetch_node ([968c69e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/968c69e217d9c180b9b8c2aa52ca59b9a1733525))
+* **AbstractGraph:** instantiation of Azure GPT models ([ade28fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ade28fca2c3fdf40f28a80854e3b8435a52a6930)), closes [#498](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/498)
+* pyproject.toml ([e90fad4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e90fad44ce53e34a73270619255cc392eed81a06))
+* rebuild pyproject, requirements and lockfiles ([1193984](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1193984434dea0ad70ff6b975ac778d56d2e1688))
+
+
+### chore
+
+* rebuild requirements ([2edad66](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2edad66788cbd92f197e3b37db13c44bfa39e36a))
+* remove unused import ([88710f1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/88710f1a7c7d50f57108456112da30d1a12a1ba1))
+* set dependency version for vertexai ([971cc2d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/971cc2da04e331ebca1f93048c78bc58b452d30a))
+* update pyproject, rebuild lockfiles ([d6312bf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d6312bfc9b2d68370727645b1ce5010ff7a626c0))
+
+
+### Refactor
+
+* **Ollama:** integrate new LangChain chat init ([d177afb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d177afb68be036465ede1f567d2562b145d77d36))
+* **OpenAI:** integrate new LangChain chat init ([5e3eb6e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5e3eb6e43df4bd4c452d34b49f254235e9ff1b22))
+* move embeddings code from AbstractGraph to RAGNode ([a94ebcd](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a94ebcde0078d66d33e67f7e0a87850efb92d408))
+* remove LangChain wrappers ([2c5f934](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2c5f934f101e319ec4e61009d4c464ca4626c1ff))
+* remove LangChain wrappers for Ollama ([25066b2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/25066b2bc51517e50058231664230b8edef365b9))
+* remove redundant LangChain wrappers ([9275486](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/927548624034b3c30eca60011d216720102d1815))
+* remove redundant wrappers for Ernie and Nvidia ([bc2c996](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bc2c9967d2f13ade6eeb7b23e9b423f6e79aa890))
+* reuse code for common interface models ([bb73d91](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bb73d916a1a7b378438038ec928eeda6d8f6ac9d))
+
+
+### CI
+
+* **release:** 1.11.0-beta.1 [skip ci] ([7080a0a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7080a0afd527a34ada33ee2d3ace8e724d879df7))
+* **release:** 1.11.0-beta.10 [skip ci] ([ee30a83](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee30a83f8a77958be6881ca0a94b02d278f37a61)), closes [#498](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/498)
+* **release:** 1.11.0-beta.2 [skip ci] ([bf6d487](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bf6d487bbb26187b32f5985433b54025f6437af5))
+* **release:** 1.11.0-beta.3 [skip ci] ([66f9421](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/66f9421fc216f0984d5a393101d1c109b08eaa33))
+* **release:** 1.11.0-beta.4 [skip ci] ([51db43a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/51db43a129ef05c050b6de017598a664119594ba))
+* **release:** 1.11.0-beta.5 [skip ci] ([b15fd9f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b15fd9f4dc3643c9904a2cbaa5f392a6805c9762))
+* **release:** 1.11.0-beta.6 [skip ci] ([74ed8d0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/74ed8d06c5db4f9734521c2f84f4379b18b7308f))
+* **release:** 1.11.0-beta.7 [skip ci] ([55f706f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/55f706f3d5f4a8afe9dd8fc9ce9bd527f8a11894))
+* **release:** 1.11.0-beta.8 [skip ci] ([3e07f62](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e07f6273fae667b2f663be1cdd5e9c068f4c59f))
+* **release:** 1.11.0-beta.9 [skip ci] ([4440790](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4440790f00c1ddd416add7af895756ab42c30bf3))
+
+
 ## [1.11.0-beta.12](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.0-beta.11...v1.11.0-beta.12) (2024-08-06)
 
 

docs/chinese.md

@@ -11,7 +11,7 @@ ScrapeGraphAI 是一个*网络爬虫* Python 库，使用大型语言模型和
 只需告诉库您想提取哪些信息，它将为您完成！
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/scrapegraphai_logo.png" alt="Scrapegraph-ai Logo" style="width: 50%;">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
 </p>
 
 ## 🚀 快速安装

docs/japanese.md

@@ -11,7 +11,7 @@ ScrapeGraphAIは、大規模言語モデルと直接グラフロジックを使
 クロールしたい情報をライブラリに伝えるだけで、残りはすべてライブラリが行います！
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/scrapegraphai_logo.png" alt="Scrapegraph-ai Logo" style="width: 50%;">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
 </p>
 
 ## 🚀 インストール方法

docs/korean.md

@@ -6,7 +6,7 @@ ScrapeGraphAI는 웹 사이트와 로컬 문서(XML, HTML, JSON 등)에 대한 
 추출하려는 정보를 말하기만 하면 라이브러리가 알아서 처리해 줍니다!
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/scrapegraphai_logo.png" alt="Scrapegraph-ai Logo" style="width: 50%;">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
 </p>
 
 ## 🚀 빠른 설치

examples/local_models/smart_scraper_ollama.py

@@ -9,7 +9,7 @@
 
 graph_config = {
     "llm": {
-        "model": "ollama/llama3",
+        "model": "ollama/llama3.1",
         "temperature": 0,
         "format": "json",  # Ollama needs the format to be specified explicitly
         # "base_url": "http://localhost:11434", # set ollama URL arbitrarily

examples/openai/md_scraper_openai.py

@@ -1,5 +1,5 @@
 """
-Basic example of scraping pipeline using MDScraperGraph from XML documents
+Basic example of scraping pipeline using MDScraperGraph from MD documents
 """
 
 import os
@@ -9,7 +9,7 @@
 load_dotenv()
 
 # ************************************************
-# Read the XML file
+# Read the MD file
 # ************************************************
 
 FILE_NAME = "inputs/markdown_example.md"

pyproject.toml

@@ -1,9 +1,11 @@
 [project]
 name = "scrapegraphai"
 
-version = "1.11.0b12"
+version = "1.12.0"
+
 
 description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines."
+
 authors = [
     { name = "Marco Vinciguerra", email = "[email protected]" },
     { name = "Marco Perini", email = "[email protected]" },
@@ -12,8 +14,10 @@ authors = [
 
 dependencies = [
     "langchain>=0.2.10",
+    "langchain-fireworks>=0.1.3",
+    "langchain_community>=0.2.9",
     "langchain-google-genai>=1.0.7",
-    "langchain-google-vertexai",
+    "langchain-google-vertexai>=1.0.7",
     "langchain-openai>=0.1.17",
     "langchain-groq>=0.1.3",
     "langchain-aws>=0.1.3",
@@ -36,7 +40,7 @@ dependencies = [
     "langchain-fireworks>=0.1.3",
     "langchain-community>=0.2.9",
     "langchain-huggingface>=0.0.3",
-    "browserbase==0.3.0"
+    "browserbase>=0.3.0"
 ]
 
 license = "MIT"

requirements-dev.lock

@@ -218,7 +218,7 @@ httpx==0.27.0
     # via openai
 httpx-sse==0.4.0
     # via fireworks-ai
-huggingface-hub==0.24.0
+huggingface-hub==0.24.1
     # via langchain-huggingface
     # via sentence-transformers
     # via tokenizers
@@ -231,7 +231,7 @@ idna==3.7
     # via yarl
 imagesize==1.4.1
     # via sphinx
-importlib-metadata==8.0.0
+importlib-metadata==8.1.0
     # via sphinx
 importlib-resources==6.4.0
     # via matplotlib
@@ -263,16 +263,16 @@ jsonschema-specifications==2023.12.1
     # via jsonschema
 kiwisolver==1.4.5
     # via matplotlib
-langchain==0.2.10
+langchain==0.2.11
     # via langchain-community
     # via scrapegraphai
 langchain-anthropic==0.1.20
     # via scrapegraphai
 langchain-aws==0.1.12
     # via scrapegraphai
-langchain-community==0.2.9
+langchain-community==0.2.10
     # via scrapegraphai
-langchain-core==0.2.22
+langchain-core==0.2.23
     # via langchain
     # via langchain-anthropic
     # via langchain-aws
@@ -295,7 +295,7 @@ langchain-groq==0.1.6
     # via scrapegraphai
 langchain-huggingface==0.0.3
     # via scrapegraphai
-langchain-nvidia-ai-endpoints==0.1.6
+langchain-nvidia-ai-endpoints==0.1.7
     # via scrapegraphai
 langchain-openai==0.1.17
     # via scrapegraphai
@@ -386,7 +386,7 @@ pillow==10.4.0
     # via streamlit
 platformdirs==4.2.2
     # via pylint
-playwright==1.45.0
+playwright==1.45.1
     # via browserbase
     # via scrapegraphai
     # via undetected-playwright

requirements.lock

@@ -162,7 +162,7 @@ httpx==0.27.0
     # via openai
 httpx-sse==0.4.0
     # via fireworks-ai
-huggingface-hub==0.24.0
+huggingface-hub==0.24.1
     # via langchain-huggingface
     # via sentence-transformers
     # via tokenizers
@@ -185,16 +185,16 @@ jsonpatch==1.33
     # via langchain-core
 jsonpointer==3.0.0
     # via jsonpatch
-langchain==0.2.10
+langchain==0.2.11
     # via langchain-community
     # via scrapegraphai
 langchain-anthropic==0.1.20
     # via scrapegraphai
 langchain-aws==0.1.12
     # via scrapegraphai
-langchain-community==0.2.9
+langchain-community==0.2.10
     # via scrapegraphai
-langchain-core==0.2.22
+langchain-core==0.2.23
     # via langchain
     # via langchain-anthropic
     # via langchain-aws
@@ -217,7 +217,7 @@ langchain-groq==0.1.6
     # via scrapegraphai
 langchain-huggingface==0.0.3
     # via scrapegraphai
-langchain-nvidia-ai-endpoints==0.1.6
+langchain-nvidia-ai-endpoints==0.1.7
     # via scrapegraphai
 langchain-openai==0.1.17
     # via scrapegraphai
@@ -278,7 +278,7 @@ pillow==10.4.0
     # via fireworks-ai
     # via langchain-nvidia-ai-endpoints
     # via sentence-transformers
-playwright==1.45.0
+playwright==1.45.1
     # via browserbase
     # via scrapegraphai
     # via undetected-playwright

requirements.txt

@@ -1,6 +1,8 @@
 langchain>=0.2.10
+langchain-fireworks>=0.1.3
+langchain_community>=0.2.9
 langchain-google-genai>=1.0.7
-langchain-google-vertexai
+langchain-google-vertexai>=1.0.7
 langchain-openai>=0.1.17
 langchain-groq>=0.1.3
 langchain-aws>=0.1.3
@@ -23,4 +25,4 @@ semchunk>=1.0.1
 langchain-fireworks>=0.1.3
 langchain-community>=0.2.9
 langchain-huggingface>=0.0.3
-browserbase==0.3.0
+browserbase>=0.3.0

scrapegraphai/nodes/fetch_node.py

@@ -168,7 +168,10 @@ def execute(self, state):
             parsed_content = source
 
             if isinstance(self.llm_model, ChatOpenAI) and not self.script_creator or self.force and not self.script_creator:
+
                 parsed_content = convert_to_md(source)
+            else:
+                parsed_content = source
 
             compressed_document = [
                 Document(page_content=parsed_content, metadata={"source": "local_dir"})
@@ -187,6 +190,7 @@ def execute(self, state):
                 if  (isinstance(self.llm_model, ChatOpenAI)
                      and not self.script_creator) or (self.force and not self.script_creator):
                     parsed_content = convert_to_md(source, input_data[0])
+
                 compressed_document = [Document(page_content=parsed_content)]
             else:
                 self.logger.warning(
@@ -217,6 +221,7 @@ def execute(self, state):
             if  isinstance(self.llm_model, ChatOpenAI) and not self.script_creator or self.force and not self.script_creator and not self.openai_md_enabled:
                 parsed_content = convert_to_md(document[0].page_content, input_data[0])
 
+
             compressed_document = [
                 Document(page_content=parsed_content, metadata={"source": "html file"})
             ]

scrapegraphai/utils/convert_to_md.py

@@ -20,11 +20,12 @@ def convert_to_md(html: str, url: str = None) -> str:
 
     Note: All the styles and links are ignored during the conversion. """
 
-    if url:
-        parsed_url = urlparse(url)
-        domain = f"{parsed_url.scheme}://{parsed_url.netloc}"
     h = html2text.HTML2Text()
     h.ignore_links = False
-    h.baseurl = domain
     h.body_width = 0
+    if url is not None:
+        parsed_url = urlparse(url)
+        domain = f"{parsed_url.scheme}://{parsed_url.netloc}"
+        h.baseurl = domain
+    
     return h.handle(html)