Merge branch 'main' into temp-1

Author: VinciGit00

CHANGELOG.md

@@ -25,10 +25,15 @@
 * fixed bug on fetch_node ([968c69e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/968c69e217d9c180b9b8c2aa52ca59b9a1733525))
 
 ## [1.11.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.0-beta.6...v1.11.0-beta.7) (2024-08-01)
+## [1.10.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.0-beta.6...v1.10.0-beta.7) (2024-07-23)
+
+## [1.11.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.2...v1.11.3) (2024-07-25)
+
 
 
 ### Bug Fixes
 
+
 * abstract_graph and removed unused embeddings ([0b4cfd6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/0b4cfd6522dcad0eb418f0badd0f7824a1efd534))
 
 
@@ -79,6 +84,16 @@
 * rebuild requirements ([2edad66](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2edad66788cbd92f197e3b37db13c44bfa39e36a))
 
 ## [1.11.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.0-beta.2...v1.11.0-beta.3) (2024-07-25)
+=======
+* add llama 3.1 ([f872bdd](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f872bdd24f9874660eea04f9ade570c96b6e7e93))
+
+
+### Docs
+
+* prev version ([5c08eea](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c08eea189d7ede6f29399a67d897aa3b3f6a7b0))
+
+
+## [1.11.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.1...v1.11.2) (2024-07-23)
 
 
 ### Bug Fixes
@@ -93,6 +108,16 @@
 * pdate models_tokens.py ([377d679](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/377d679eecd62611c0c9a3cba8202c6f0719ed31))
 
 ## [1.11.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.4...v1.11.0-beta.1) (2024-07-23)
+* md conversion ([1d41f6e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1d41f6eafe8ed0e191bb6a258d54c6388ff283c6))
+
+## [1.11.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.11.0...v1.11.1) (2024-07-23)
+
+
+### Bug Fixes
+
+* md conversion ([5a45e9f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5a45e9f2d86a1c58b8ea321e3df9718bc00f9c28))
+
+## [1.11.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.4...v1.11.0) (2024-07-23)
 
 
 ### Features
@@ -181,8 +206,11 @@
 
 
 
+
 ### Features
 
+* add nvidia connection ([fc0dadb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/fc0dadb8f812dfd636dec856921a971b58695ce3))
+
 
 * add new toml ([fcb3220](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/fcb3220868e7ef1127a7a47f40d0379be282e6eb))
 
@@ -201,6 +229,24 @@
 
 ### chore
 
+* **dependecies:** add script to auto-update requirements ([3289c7b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3289c7bf5ec58ac3d04e9e5e8e654af9abcee228))
+* **ci:** set up workflow for requirements auto-update ([295fc28](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/295fc28ceb02c78198f7fbe678352503b3259b6b))
+* update requirements.txt ([c7bac98](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c7bac98d2e79e5ab98fa65d7efa858a2cdda1622))
+
+## [1.10.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.0-beta.5...v1.10.0-beta.6) (2024-07-22)
+
+
+### Features
+
+* add new toml ([fcb3220](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/fcb3220868e7ef1127a7a47f40d0379be282e6eb))
+* add gpt4o omni ([431edb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/431edb7bb2504f4c1335c3ae3ce2f91867fa7222))
+* add searchngx integration ([5c92186](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c9218608140bf694fbfd96aa90276bc438bb475))
+* refactoring_to_md function ([602dd00](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/602dd00209ee1d72a1223fc4793759450921fcf9))
+
+
+
+
+### chore
 
 * **pyproject:** upgrade dependencies ([0425124](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/0425124c570f765b98fcf67ba6649f4f9fe76b15))
 * correct search engine name ([7ba2f6a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7ba2f6ae0b9d2e9336e973e1f57ab8355c739e27))
@@ -209,11 +255,13 @@
 * upgrade tiktoken ([7314bc3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7314bc383068db590662bf7e512f799529308991))
 
 
+
 ### Docs
 
 * **gpt-4o-mini:** added new gpt, fixed chromium lazy loading, ([99dc849](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/99dc8497d85289759286a973e4aecc3f924d3ada))
 
 
+
 ### CI
 
 * **release:** 1.10.0-beta.1 [skip ci] ([8f619de](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f619de23540216934b53bcf3426702e56c48f31))
@@ -554,7 +602,7 @@
 * **release:** 1.6.1 [skip ci] ([44fbd71](https://github.com/VinciGit00/Scrapegraph-ai/commit/44fbd71742a57a4b10f22ed33781bb67aa77e58d))
 
 ## [1.6.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.0...v1.6.1) (2024-06-15)
-=======
+
 
 
 ### Bug Fixes

docs/chinese.md

@@ -11,7 +11,7 @@ ScrapeGraphAI 是一个*网络爬虫* Python 库，使用大型语言模型和
 只需告诉库您想提取哪些信息，它将为您完成！
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/scrapegraphai_logo.png" alt="Scrapegraph-ai Logo" style="width: 50%;">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
 </p>
 
 ## 🚀 快速安装

docs/japanese.md

@@ -11,7 +11,7 @@ ScrapeGraphAIは、大規模言語モデルと直接グラフロジックを使
 クロールしたい情報をライブラリに伝えるだけで、残りはすべてライブラリが行います！
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/scrapegraphai_logo.png" alt="Scrapegraph-ai Logo" style="width: 50%;">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
 </p>
 
 ## 🚀 インストール方法

docs/korean.md

@@ -6,7 +6,7 @@ ScrapeGraphAI는 웹 사이트와 로컬 문서(XML, HTML, JSON 등)에 대한 
 추출하려는 정보를 말하기만 하면 라이브러리가 알아서 처리해 줍니다!
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/scrapegraphai_logo.png" alt="Scrapegraph-ai Logo" style="width: 50%;">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
 </p>
 
 ## 🚀 빠른 설치

examples/local_models/smart_scraper_ollama.py

@@ -9,7 +9,7 @@
 
 graph_config = {
     "llm": {
-        "model": "ollama/llama3",
+        "model": "ollama/llama3.1",
         "temperature": 0,
         "format": "json",  # Ollama needs the format to be specified explicitly
         # "base_url": "http://localhost:11434", # set ollama URL arbitrarily

examples/openai/md_scraper_openai.py

@@ -1,5 +1,5 @@
 """
-Basic example of scraping pipeline using MDScraperGraph from XML documents
+Basic example of scraping pipeline using MDScraperGraph from MD documents
 """
 
 import os
@@ -9,7 +9,7 @@
 load_dotenv()
 
 # ************************************************
-# Read the XML file
+# Read the MD file
 # ************************************************
 
 FILE_NAME = "inputs/markdown_example.md"

pyproject.toml

@@ -4,16 +4,20 @@ name = "scrapegraphai"
 version = "1.11.0b10"
 
 description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines."
+
 authors = [
     { name = "Marco Vinciguerra", email = "[email protected]" },
     { name = "Marco Perini", email = "[email protected]" },
     { name = "Lorenzo Padoan", email = "[email protected]" }
 ]
 
-dependencies = [
     "langchain>=0.2.10",
+
+    "langchain-fireworks>=0.1.3",
+    "langchain_community>=0.2.9",
+
     "langchain-google-genai>=1.0.7",
-    "langchain-google-vertexai",
+    "langchain-google-vertexai>=1.0.7",
     "langchain-openai>=0.1.17",
     "langchain-groq>=0.1.3",
     "langchain-aws>=0.1.3",

requirements-dev.lock

@@ -221,6 +221,8 @@ httpx-sse==0.4.0
 huggingface-hub==0.24.0
     # via langchain-huggingface
     # via sentence-transformers
+huggingface-hub==0.24.1
+
     # via tokenizers
     # via transformers
 idna==3.7
@@ -231,7 +233,7 @@ idna==3.7
     # via yarl
 imagesize==1.4.1
     # via sphinx
-importlib-metadata==8.0.0
+importlib-metadata==8.1.0
     # via sphinx
 importlib-resources==6.4.0
     # via matplotlib
@@ -263,16 +265,16 @@ jsonschema-specifications==2023.12.1
     # via jsonschema
 kiwisolver==1.4.5
     # via matplotlib
-langchain==0.2.10
+langchain==0.2.11
     # via langchain-community
     # via scrapegraphai
 langchain-anthropic==0.1.20
     # via scrapegraphai
 langchain-aws==0.1.12
     # via scrapegraphai
-langchain-community==0.2.9
+langchain-community==0.2.10
     # via scrapegraphai
-langchain-core==0.2.22
+langchain-core==0.2.23
     # via langchain
     # via langchain-anthropic
     # via langchain-aws
@@ -296,6 +298,8 @@ langchain-groq==0.1.6
 langchain-huggingface==0.0.3
     # via scrapegraphai
 langchain-nvidia-ai-endpoints==0.1.6
+
+langchain-nvidia-ai-endpoints==0.1.7
     # via scrapegraphai
 langchain-openai==0.1.17
     # via scrapegraphai
@@ -386,8 +390,8 @@ pillow==10.4.0
     # via streamlit
 platformdirs==4.2.2
     # via pylint
-playwright==1.45.0
-    # via browserbase
+
+playwright==1.45.1
     # via scrapegraphai
     # via undetected-playwright
 pluggy==1.5.0

requirements.lock

@@ -162,9 +162,7 @@ httpx==0.27.0
     # via openai
 httpx-sse==0.4.0
     # via fireworks-ai
-huggingface-hub==0.24.0
-    # via langchain-huggingface
-    # via sentence-transformers
+huggingface-hub==0.24.1
     # via tokenizers
     # via transformers
 idna==3.7
@@ -185,16 +183,16 @@ jsonpatch==1.33
     # via langchain-core
 jsonpointer==3.0.0
     # via jsonpatch
-langchain==0.2.10
+langchain==0.2.11
     # via langchain-community
     # via scrapegraphai
 langchain-anthropic==0.1.20
     # via scrapegraphai
 langchain-aws==0.1.12
     # via scrapegraphai
-langchain-community==0.2.9
+langchain-community==0.2.10
     # via scrapegraphai
-langchain-core==0.2.22
+langchain-core==0.2.23
     # via langchain
     # via langchain-anthropic
     # via langchain-aws
@@ -217,7 +215,8 @@ langchain-groq==0.1.6
     # via scrapegraphai
 langchain-huggingface==0.0.3
     # via scrapegraphai
-langchain-nvidia-ai-endpoints==0.1.6
+langchain-nvidia-ai-endpoints==0.1.7
+
     # via scrapegraphai
 langchain-openai==0.1.17
     # via scrapegraphai
@@ -271,15 +270,14 @@ packaging==24.1
     # via huggingface-hub
     # via langchain-core
     # via marshmallow
-    # via transformers
+
 pandas==2.2.2
     # via scrapegraphai
 pillow==10.4.0
     # via fireworks-ai
     # via langchain-nvidia-ai-endpoints
-    # via sentence-transformers
-playwright==1.45.0
-    # via browserbase
+
+playwright==1.45.1
     # via scrapegraphai
     # via undetected-playwright
 proto-plus==1.24.0
@@ -420,7 +418,7 @@ typing-extensions==4.12.2
     # via pydantic-core
     # via pyee
     # via sqlalchemy
-    # via torch
+
     # via typing-inspect
 typing-inspect==0.9.0
     # via dataclasses-json

requirements.txt

@@ -1,6 +1,12 @@
 langchain>=0.2.10
 langchain-google-genai>=1.0.7
 langchain-google-vertexai
+
+langchain-fireworks>=0.1.3
+langchain_community>=0.2.9
+langchain-google-genai>=1.0.7
+langchain-google-vertexai>=1.0.7
+
 langchain-openai>=0.1.17
 langchain-groq>=0.1.3
 langchain-aws>=0.1.3
@@ -24,3 +30,4 @@ langchain-fireworks>=0.1.3
 langchain-community>=0.2.9
 langchain-huggingface>=0.0.3
 browserbase==0.3.0
+

scrapegraphai/nodes/fetch_node.py

@@ -168,7 +168,10 @@ def execute(self, state):
             parsed_content = source
 
             if isinstance(self.llm_model, ChatOpenAI) and not self.script_creator or self.force and not self.script_creator:
+
                 parsed_content = convert_to_md(source)
+            else:
+                parsed_content = source
 
             compressed_document = [
                 Document(page_content=parsed_content, metadata={"source": "local_dir"})
@@ -187,6 +190,7 @@ def execute(self, state):
                 if  (isinstance(self.llm_model, ChatOpenAI)
                      and not self.script_creator) or (self.force and not self.script_creator):
                     parsed_content = convert_to_md(source, input_data[0])
+
                 compressed_document = [Document(page_content=parsed_content)]
             else:
                 self.logger.warning(
@@ -217,6 +221,7 @@ def execute(self, state):
             if  isinstance(self.llm_model, ChatOpenAI) and not self.script_creator or self.force and not self.script_creator and not self.openai_md_enabled:
                 parsed_content = convert_to_md(document[0].page_content, input_data[0])
 
+
             compressed_document = [
                 Document(page_content=parsed_content, metadata={"source": "html file"})
             ]

scrapegraphai/utils/convert_to_md.py

@@ -20,11 +20,12 @@ def convert_to_md(html: str, url: str = None) -> str:
 
     Note: All the styles and links are ignored during the conversion. """
 
-    if url:
-        parsed_url = urlparse(url)
-        domain = f"{parsed_url.scheme}://{parsed_url.netloc}"
     h = html2text.HTML2Text()
     h.ignore_links = False
-    h.baseurl = domain
     h.body_width = 0
+    if url is not None:
+        parsed_url = urlparse(url)
+        domain = f"{parsed_url.scheme}://{parsed_url.netloc}"
+        h.baseurl = domain
+    
     return h.handle(html)