Skip to content

Commit 76493d2

Browse files
f-aguzzif-aguzzi
authored
Merge pull request #565 from ScrapeGraphAI/pre/beta
Merge pre/beta into main, release ScrapeGraphAI 1.14
2 parents 3e6a037 + f4dbe5b commit 76493d2

File tree

171 files changed

+2714
-766
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

171 files changed

+2714
-766
lines changed

‎CHANGELOG.md

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,125 @@
1+
## [1.14.0-beta.13](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.12...v1.14.0-beta.13) (2024-08-20)
2+
3+
4+
### Features
5+
6+
* add async call ([f60aa3a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f60aa3acde3c9bead2250e81eb8fc77d2e1e450c))
7+
* refactoring of the code ([5eb3cff](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5eb3cff64f5becf7e107325117364b67b5fe7348))
8+
9+
## [1.14.0-beta.12](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.11...v1.14.0-beta.12) (2024-08-20)
10+
11+
12+
### Bug Fixes
13+
14+
* **SearchNode:** prompt ([052f7d5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/052f7d5e66436c97e17491c00b86c382642490b6))
15+
16+
## [1.14.0-beta.11](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.10...v1.14.0-beta.11) (2024-08-19)
17+
18+
19+
### Features
20+
21+
* add structured output format ([7d2fc67](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7d2fc672c8c3c05b0f0beac46316ce16c16bcd02))
22+
* **GenerateAnswerNode:** built-in structured output through LangChain ([d29338b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d29338b7c2ef0b13535a2e4edae4a4aab08f1825))
23+
24+
25+
### Bug Fixes
26+
27+
* **ParseNode:** leave room for LLM reply in context window ([683bf57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/683bf57d895d8f6847fdd64e8936ffa1aa91926a))
28+
29+
## [1.14.0-beta.10](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.9...v1.14.0-beta.10) (2024-08-19)
30+
31+
32+
### Features
33+
34+
* Implemented a filter logic in search_link_node.py ([08e9d9d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/08e9d9d6a09f450a9f512ac2789287819ced9641))
35+
36+
## [1.14.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.8...v1.14.0-beta.9) (2024-08-17)
37+
38+
39+
### Features
40+
41+
* update model tokens dict ([0aca287](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/0aca28732b249ffaedf5b665cbfb5b1255c0cc74))
42+
43+
## [1.14.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.7...v1.14.0-beta.8) (2024-08-17)
44+
45+
46+
### Bug Fixes
47+
48+
* browser-base integration ([1d7f30b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1d7f30b65b24b80113cd898c1cfbfd5de5f240b5))
49+
50+
## [1.14.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.6...v1.14.0-beta.7) (2024-08-16)
51+
52+
53+
### Bug Fixes
54+
55+
* model count ([faef318](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/faef3186f795e950ade14bc8b6d8d1cea3afd327))
56+
57+
## [1.14.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.5...v1.14.0-beta.6) (2024-08-16)
58+
59+
60+
### Features
61+
62+
* add integration for new module of gpt4o ([982150e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/982150e81fbaa4241c725aaa9dfcd553f8b86978))
63+
64+
## [1.14.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.4...v1.14.0-beta.5) (2024-08-16)
65+
66+
67+
### Features
68+
69+
* Add new feature to support gpt-4o variant models with different pricing ([8551448](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/855144876d796ceebb0930fec45ead6cc3834f14))
70+
71+
## [1.14.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.3...v1.14.0-beta.4) (2024-08-15)
72+
73+
74+
### Features
75+
76+
* update abstract graph ([c77231c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c77231c983bd6e154eefd26422cd156da4c8b7bb))
77+
78+
## [1.14.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.2...v1.14.0-beta.3) (2024-08-13)
79+
80+
81+
### Bug Fixes
82+
83+
* **models_tokens:** incorrect provider names ([cb6b353](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cb6b35397e56c6785553480200aa948053d9904b))
84+
85+
86+
### chore
87+
88+
* **examples:** add vertex examples, rename genai examples ([1aa9c6e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1aa9c6e73bfa26b83010cf8d980cdf5f572cde5a))
89+
* **examples:** update provider names to match tokens dictionary ([ee078cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee078cb102ad922a900228ebe5ea45724712a960))
90+
91+
## [1.14.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.1...v1.14.0-beta.2) (2024-08-12)
92+
93+
94+
### Bug Fixes
95+
96+
* **AbstractGraph:** pass kwargs to Ernie and Nvidia models ([e6bedb6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e6bedb6701601e87a6dff99eabec9c3494280411))
97+
98+
99+
### chore
100+
101+
* **examples:** fix import bug in image2text demo ([71438a1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71438a1e8696aee51d054f9df7243665497fc35c))
102+
* **requirements:** update requirements.txt ([7fe181f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7fe181f69b3178d2d9d41a00fd660a98e04b777e))
103+
104+
## [1.14.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.3...v1.14.0-beta.1) (2024-08-11)
105+
106+
107+
### Features
108+
109+
* add refactoring of default temperature ([6c3b37a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c3b37ab001b80c09ea9ffb56d4c3df338e33a7a))
110+
111+
112+
### Bug Fixes
113+
114+
* broken node ([1272273](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/127227349915deeb0dede34aa575ad269ed7cbe3))
115+
* merge_anwser prompt import ([f17cef9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f17cef94bb39349d40cc520d93b51ac4e629db32))
116+
117+
118+
### CI
119+
120+
* **release:** 1.13.0-beta.8 [skip ci] ([b470d97](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b470d974cf3fdb3a75ead46fceb8c21525e2e616))
121+
* **release:** 1.13.0-beta.9 [skip ci] ([d4c1a1c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d4c1a1c58a54740ff50aa87b1d1d3500b61ea088))
122+
1123
## [1.13.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.2...v1.13.3) (2024-08-10)
2124

3125

@@ -25,6 +147,8 @@
25147
* conditional node ([ce00345](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ce003454953e5785d4746223c252de38cd5d07ea))
26148

27149
## [1.13.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.12.2...v1.13.0) (2024-08-09)
150+
## [1.13.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.8...v1.13.0-beta.9) (2024-08-10)
151+
28152

29153

30154
### Features
@@ -65,6 +189,15 @@
65189
* **release:** 1.13.0-beta.5 [skip ci] ([2eba73b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2eba73b784ee443260117e98ab7c943934b3018d)), closes [#513](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/513)
66190
* **release:** 1.13.0-beta.6 [skip ci] ([e75b574](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e75b574b67040e127599da9ee1b0eee13d234cb9))
67191
* **release:** 1.13.0-beta.7 [skip ci] ([6e56925](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6e56925355c424edae290c70fd98646ab5f420ee))
192+
* add refactoring of default temperature ([6c3b37a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c3b37ab001b80c09ea9ffb56d4c3df338e33a7a))
193+
194+
## [1.13.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.7...v1.13.0-beta.8) (2024-08-09)
195+
196+
197+
### Bug Fixes
198+
199+
* broken node ([1272273](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/127227349915deeb0dede34aa575ad269ed7cbe3))
200+
68201

69202
## [1.13.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.6...v1.13.0-beta.7) (2024-08-09)
70203

‎Dockerfile

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
FROM python:3.11-slim
22

3-
RUN apt-get update && apt-get upgrade -y && \
4-
useradd -m -s /bin/bash app
5-
6-
USER app
3+
RUN apt-get update && apt-get upgrade -y
74

85
RUN pip install scrapegraphai
6+
RUN pip install scrapegraphai[burr]
7+
8+
RUN python3 -m playwright install-deps
9+
RUN python3 -m playwright install

‎examples/anthropic/custom_graph_haiku.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
from dotenv import load_dotenv
77

88
from langchain_openai import OpenAIEmbeddings
9-
from scrapegraphai.models import OpenAI
9+
from langchain_openai import ChatOpenAI
1010
from scrapegraphai.graphs import BaseGraph
1111
from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
1212
load_dotenv()

‎examples/anthropic/search_graph_schema_haiku.py

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,12 @@
33
"""
44

55
import os
6+
from typing import List
67
from dotenv import load_dotenv
7-
load_dotenv()
8-
8+
from pydantic import BaseModel, Field
99
from scrapegraphai.graphs import SearchGraph
1010

11-
from pydantic import BaseModel, Field
12-
from typing import List
11+
load_dotenv()
1312

1413
# ************************************************
1514
# Define the output schema for the graph

‎examples/azure/csv_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
graph_config = {
2626
"llm": {
2727
"api_key": os.environ["AZURE_OPENAI_KEY"],
28-
"model": "azure/gpt-3.5-turbo",
28+
"model": "azure_openai/gpt-3.5-turbo",
2929
},
3030
"verbose": True,
3131
"headless": False

‎examples/azure/csv_scraper_graph_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
graph_config = {
2626
"llm": {
2727
"api_key": os.environ["AZURE_OPENAI_KEY"],
28-
"model": "azure/gpt-3.5-turbo",
28+
"model": "azure_openai/gpt-3.5-turbo",
2929
},
3030
"verbose": True,
3131
"headless": False

‎examples/azure/json_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
graph_config = {
2424
"llm": {
2525
"api_key": os.environ["AZURE_OPENAI_KEY"],
26-
"model": "azure/gpt-3.5-turbo",
26+
"model": "azure_openai/gpt-3.5-turbo",
2727
},
2828
"verbose": True,
2929
"headless": False

‎examples/azure/json_scraper_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
graph_config = {
1313
"llm": {
1414
"api_key": os.environ["AZURE_OPENAI_KEY"],
15-
"model": "azure/gpt-3.5-turbo",
15+
"model": "azure_openai/gpt-3.5-turbo",
1616
},
1717
"verbose": True,
1818
"headless": False

‎examples/azure/pdf_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
graph_config = {
1111
"llm": {
1212
"api_key": os.environ["AZURE_OPENAI_KEY"],
13-
"model": "azure/gpt-3.5-turbo",
13+
"model": "azure_openai/gpt-3.5-turbo",
1414
},
1515
"verbose": True,
1616
"headless": False

‎examples/azure/scrape_plain_text_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
graph_config = {
2929
"llm": {
3030
"api_key": os.environ["AZURE_OPENAI_KEY"],
31-
"model": "azure/gpt-3.5-turbo",
31+
"model": "azure_openai/gpt-3.5-turbo",
3232
},
3333
"verbose": True,
3434
"headless": False

‎examples/azure/script_generator_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
graph_config = {
1616
"llm": {
1717
"api_key": os.environ["AZURE_OPENAI_KEY"],
18-
"model": "azure/gpt-3.5-turbo",
18+
"model": "azure_openai/gpt-3.5-turbo",
1919
},
2020
"verbose": True,
2121
"headless": False

‎examples/azure/script_multi_generator_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
graph_config = {
1717
"llm": {
1818
"api_key": os.environ["AZURE_OPENAI_KEY"],
19-
"model": "azure/gpt-3.5-turbo",
19+
"model": "azure_openai/gpt-3.5-turbo",
2020
},
2121
"verbose": True,
2222
"headless": False

‎examples/azure/search_graph_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
graph_config = {
2323
"llm": {
2424
"api_key": os.environ["AZURE_OPENAI_KEY"],
25-
"model": "azure/gpt-3.5-turbo",
25+
"model": "azure_openai/gpt-3.5-turbo",
2626
},
2727
"verbose": True,
2828
"headless": False

‎examples/azure/search_graph_schema_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ class Dishes(BaseModel):
3030
graph_config = {
3131
"llm": {
3232
"api_key": os.environ["AZURE_OPENAI_KEY"],
33-
"model": "azure/gpt-3.5-turbo",
33+
"model": "azure_openai/gpt-3.5-turbo",
3434
},
3535
"verbose": True,
3636
"headless": False

‎examples/azure/search_link_graph_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
graph_config = {
1616
"llm": {
1717
"api_key": os.environ["AZURE_OPENAI_KEY"],
18-
"model": "azure/gpt-3.5-turbo",
18+
"model": "azure_openai/gpt-3.5-turbo",
1919
},
2020
"verbose": True,
2121
"headless": False

‎examples/azure/smart_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
graph_config = {
2727
"llm": {
2828
"api_key": os.environ["AZURE_OPENAI_KEY"],
29-
"model": "azure/gpt-3.5-turbo",
29+
"model": "azure_openai/gpt-3.5-turbo",
3030
},
3131
"verbose": True,
3232
"headless": False

‎examples/azure/smart_scraper_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
graph_config = {
1515
"llm": {
1616
"api_key": os.environ["AZURE_OPENAI_KEY"],
17-
"model": "azure/gpt-3.5-turbo",
17+
"model": "azure_openai/gpt-3.5-turbo",
1818
},
1919
"verbose": True,
2020
"headless": False

‎examples/azure/smart_scraper_schema_azure.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
Basic example of scraping pipeline using SmartScraper with schema
33
"""
44

5-
import os, json
5+
import os
6+
import json
67
from typing import List
78
from pydantic import BaseModel, Field
89
from dotenv import load_dotenv
@@ -28,7 +29,7 @@ class Projects(BaseModel):
2829
graph_config = {
2930
"llm": {
3031
"api_key": os.environ["AZURE_OPENAI_KEY"],
31-
"model": "azure/gpt-3.5-turbo",
32+
"model": "azure_openai/gpt-3.5-turbo",
3233
},
3334
"verbose": True,
3435
"headless": False

‎examples/azure/xml_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
graph_config = {
2525
"llm": {
2626
"api_key": os.environ["AZURE_OPENAI_KEY"],
27-
"model": "azure/gpt-3.5-turbo",
27+
"model": "azure_openai/gpt-3.5-turbo",
2828
},
2929
"verbose": True,
3030
"headless": False

‎examples/azure/xml_scraper_graph_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
graph_config = {
2626
"llm": {
2727
"api_key": os.environ["AZURE_OPENAI_KEY"],
28-
"model": "azure/gpt-3.5-turbo",
28+
"model": "azure_openai/gpt-3.5-turbo",
2929
},
3030
"verbose": True,
3131
"headless": False

‎examples/ernie/custom_graph_ernie.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
from dotenv import load_dotenv
77

88
from langchain_openai import OpenAIEmbeddings
9-
from scrapegraphai.models import OpenAI
9+
from langchain_openai import ChatOpenAI
1010
from scrapegraphai.graphs import BaseGraph
1111
from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
1212

‎examples/extras/.env.example

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
OPENAI_API_KEY="OPENAI_API_KEY"
2+
BROWSER_BASE_PROJECT_ID="BROWSER_BASE_PROJECT_ID"
3+
BROWSER_BASE_API_KEY="BROWSERBASE_API_KEY"

‎examples/extras/browser_base_integration.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@
1818
graph_config = {
1919
"llm": {
2020
"api_key": os.getenv("OPENAI_API_KEY"),
21-
"model": "gpt-3.5-turbo",
21+
"model": "gpt-4o",
2222
},
2323
"browser_base": {
2424
"api_key": os.getenv("BROWSER_BASE_API_KEY"),
25-
"project_id": os.getenv("BROWSER_BASE_API_KEY"),
25+
"project_id": os.getenv("BROWSER_BASE_PROJECT_ID"),
2626
},
2727
"verbose": True,
2828
"headless": False,

‎examples/gemini/.env.example renamed to ‎examples/google_genai/.env.example

File renamed without changes.

‎examples/gemini/csv_scraper_gemini.py renamed to ‎examples/google_genai/csv_scraper_gemini.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
graph_config = {
2525
"llm": {
2626
"api_key": gemini_key,
27-
"model": "gemini-pro",
27+
"model": "google_genai/gemini-pro",
2828
},
2929
}
3030

‎examples/gemini/csv_scraper_graph_multi_gemini.py renamed to ‎examples/google_genai/csv_scraper_graph_multi_gemini.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
graph_config = {
2929
"llm": {
3030
"api_key": gemini_key,
31-
"model": "gemini-pro",
31+
"model": "google_genai/gemini-pro",
3232
},
3333
}
3434

0 commit comments

Comments
 (0)