Skip to content

Commit dcd12d8

Browse files
f-aguzzif-aguzzi
committed
Merge branch 'pre/beta' into pr/563
2 parents 31931aa + d617750 commit dcd12d8

File tree

164 files changed

+2361
-765
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

164 files changed

+2361
-765
lines changed

‎CHANGELOG.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,110 @@
1+
## [1.14.0-beta.11](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.10...v1.14.0-beta.11) (2024-08-19)
2+
3+
4+
### Features
5+
6+
* add structured output format ([7d2fc67](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7d2fc672c8c3c05b0f0beac46316ce16c16bcd02))
7+
* **GenerateAnswerNode:** built-in structured output through LangChain ([d29338b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d29338b7c2ef0b13535a2e4edae4a4aab08f1825))
8+
9+
10+
### Bug Fixes
11+
12+
* **ParseNode:** leave room for LLM reply in context window ([683bf57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/683bf57d895d8f6847fdd64e8936ffa1aa91926a))
13+
14+
## [1.14.0-beta.10](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.9...v1.14.0-beta.10) (2024-08-19)
15+
16+
17+
### Features
18+
19+
* Implemented a filter logic in search_link_node.py ([08e9d9d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/08e9d9d6a09f450a9f512ac2789287819ced9641))
20+
21+
## [1.14.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.8...v1.14.0-beta.9) (2024-08-17)
22+
23+
24+
### Features
25+
26+
* update model tokens dict ([0aca287](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/0aca28732b249ffaedf5b665cbfb5b1255c0cc74))
27+
28+
## [1.14.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.7...v1.14.0-beta.8) (2024-08-17)
29+
30+
31+
### Bug Fixes
32+
33+
* browser-base integration ([1d7f30b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1d7f30b65b24b80113cd898c1cfbfd5de5f240b5))
34+
35+
## [1.14.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.6...v1.14.0-beta.7) (2024-08-16)
36+
37+
38+
### Bug Fixes
39+
40+
* model count ([faef318](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/faef3186f795e950ade14bc8b6d8d1cea3afd327))
41+
42+
## [1.14.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.5...v1.14.0-beta.6) (2024-08-16)
43+
44+
45+
### Features
46+
47+
* add integration for new module of gpt4o ([982150e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/982150e81fbaa4241c725aaa9dfcd553f8b86978))
48+
49+
## [1.14.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.4...v1.14.0-beta.5) (2024-08-16)
50+
51+
52+
### Features
53+
54+
* Add new feature to support gpt-4o variant models with different pricing ([8551448](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/855144876d796ceebb0930fec45ead6cc3834f14))
55+
56+
## [1.14.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.3...v1.14.0-beta.4) (2024-08-15)
57+
58+
59+
### Features
60+
61+
* update abstract graph ([c77231c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c77231c983bd6e154eefd26422cd156da4c8b7bb))
62+
63+
## [1.14.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.2...v1.14.0-beta.3) (2024-08-13)
64+
65+
66+
### Bug Fixes
67+
68+
* **models_tokens:** incorrect provider names ([cb6b353](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cb6b35397e56c6785553480200aa948053d9904b))
69+
70+
71+
### chore
72+
73+
* **examples:** add vertex examples, rename genai examples ([1aa9c6e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1aa9c6e73bfa26b83010cf8d980cdf5f572cde5a))
74+
* **examples:** update provider names to match tokens dictionary ([ee078cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee078cb102ad922a900228ebe5ea45724712a960))
75+
76+
## [1.14.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.1...v1.14.0-beta.2) (2024-08-12)
77+
78+
79+
### Bug Fixes
80+
81+
* **AbstractGraph:** pass kwargs to Ernie and Nvidia models ([e6bedb6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e6bedb6701601e87a6dff99eabec9c3494280411))
82+
83+
84+
### chore
85+
86+
* **examples:** fix import bug in image2text demo ([71438a1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71438a1e8696aee51d054f9df7243665497fc35c))
87+
* **requirements:** update requirements.txt ([7fe181f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7fe181f69b3178d2d9d41a00fd660a98e04b777e))
88+
89+
## [1.14.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.3...v1.14.0-beta.1) (2024-08-11)
90+
91+
92+
### Features
93+
94+
* add refactoring of default temperature ([6c3b37a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c3b37ab001b80c09ea9ffb56d4c3df338e33a7a))
95+
96+
97+
### Bug Fixes
98+
99+
* broken node ([1272273](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/127227349915deeb0dede34aa575ad269ed7cbe3))
100+
* merge_anwser prompt import ([f17cef9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f17cef94bb39349d40cc520d93b51ac4e629db32))
101+
102+
103+
### CI
104+
105+
* **release:** 1.13.0-beta.8 [skip ci] ([b470d97](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b470d974cf3fdb3a75ead46fceb8c21525e2e616))
106+
* **release:** 1.13.0-beta.9 [skip ci] ([d4c1a1c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d4c1a1c58a54740ff50aa87b1d1d3500b61ea088))
107+
1108
## [1.13.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.2...v1.13.3) (2024-08-10)
2109

3110

@@ -25,6 +132,8 @@
25132
* conditional node ([ce00345](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ce003454953e5785d4746223c252de38cd5d07ea))
26133

27134
## [1.13.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.12.2...v1.13.0) (2024-08-09)
135+
## [1.13.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.8...v1.13.0-beta.9) (2024-08-10)
136+
28137

29138

30139
### Features
@@ -65,6 +174,15 @@
65174
* **release:** 1.13.0-beta.5 [skip ci] ([2eba73b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2eba73b784ee443260117e98ab7c943934b3018d)), closes [#513](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/513)
66175
* **release:** 1.13.0-beta.6 [skip ci] ([e75b574](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e75b574b67040e127599da9ee1b0eee13d234cb9))
67176
* **release:** 1.13.0-beta.7 [skip ci] ([6e56925](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6e56925355c424edae290c70fd98646ab5f420ee))
177+
* add refactoring of default temperature ([6c3b37a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c3b37ab001b80c09ea9ffb56d4c3df338e33a7a))
178+
179+
## [1.13.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.7...v1.13.0-beta.8) (2024-08-09)
180+
181+
182+
### Bug Fixes
183+
184+
* broken node ([1272273](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/127227349915deeb0dede34aa575ad269ed7cbe3))
185+
68186

69187
## [1.13.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.6...v1.13.0-beta.7) (2024-08-09)
70188

‎Dockerfile

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
FROM python:3.11-slim
22

3-
RUN apt-get update && apt-get upgrade -y && \
4-
useradd -m -s /bin/bash app
5-
6-
USER app
3+
RUN apt-get update && apt-get upgrade -y
74

85
RUN pip install scrapegraphai
6+
RUN pip install scrapegraphai[burr]
7+
8+
RUN python3 -m playwright install-deps
9+
RUN python3 -m playwright install

‎examples/anthropic/custom_graph_haiku.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
from dotenv import load_dotenv
77

88
from langchain_openai import OpenAIEmbeddings
9-
from scrapegraphai.models import OpenAI
9+
from langchain_openai import ChatOpenAI
1010
from scrapegraphai.graphs import BaseGraph
1111
from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
1212
load_dotenv()

‎examples/anthropic/search_graph_schema_haiku.py

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,12 @@
33
"""
44

55
import os
6+
from typing import List
67
from dotenv import load_dotenv
7-
load_dotenv()
8-
8+
from pydantic import BaseModel, Field
99
from scrapegraphai.graphs import SearchGraph
1010

11-
from pydantic import BaseModel, Field
12-
from typing import List
11+
load_dotenv()
1312

1413
# ************************************************
1514
# Define the output schema for the graph

‎examples/azure/csv_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
graph_config = {
2626
"llm": {
2727
"api_key": os.environ["AZURE_OPENAI_KEY"],
28-
"model": "azure/gpt-3.5-turbo",
28+
"model": "azure_openai/gpt-3.5-turbo",
2929
},
3030
"verbose": True,
3131
"headless": False

‎examples/azure/csv_scraper_graph_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
graph_config = {
2626
"llm": {
2727
"api_key": os.environ["AZURE_OPENAI_KEY"],
28-
"model": "azure/gpt-3.5-turbo",
28+
"model": "azure_openai/gpt-3.5-turbo",
2929
},
3030
"verbose": True,
3131
"headless": False

‎examples/azure/json_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
graph_config = {
2424
"llm": {
2525
"api_key": os.environ["AZURE_OPENAI_KEY"],
26-
"model": "azure/gpt-3.5-turbo",
26+
"model": "azure_openai/gpt-3.5-turbo",
2727
},
2828
"verbose": True,
2929
"headless": False

‎examples/azure/json_scraper_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
graph_config = {
1313
"llm": {
1414
"api_key": os.environ["AZURE_OPENAI_KEY"],
15-
"model": "azure/gpt-3.5-turbo",
15+
"model": "azure_openai/gpt-3.5-turbo",
1616
},
1717
"verbose": True,
1818
"headless": False

‎examples/azure/pdf_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
graph_config = {
1111
"llm": {
1212
"api_key": os.environ["AZURE_OPENAI_KEY"],
13-
"model": "azure/gpt-3.5-turbo",
13+
"model": "azure_openai/gpt-3.5-turbo",
1414
},
1515
"verbose": True,
1616
"headless": False

‎examples/azure/scrape_plain_text_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
graph_config = {
2929
"llm": {
3030
"api_key": os.environ["AZURE_OPENAI_KEY"],
31-
"model": "azure/gpt-3.5-turbo",
31+
"model": "azure_openai/gpt-3.5-turbo",
3232
},
3333
"verbose": True,
3434
"headless": False

‎examples/azure/script_generator_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
graph_config = {
1616
"llm": {
1717
"api_key": os.environ["AZURE_OPENAI_KEY"],
18-
"model": "azure/gpt-3.5-turbo",
18+
"model": "azure_openai/gpt-3.5-turbo",
1919
},
2020
"verbose": True,
2121
"headless": False

‎examples/azure/script_multi_generator_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
graph_config = {
1717
"llm": {
1818
"api_key": os.environ["AZURE_OPENAI_KEY"],
19-
"model": "azure/gpt-3.5-turbo",
19+
"model": "azure_openai/gpt-3.5-turbo",
2020
},
2121
"verbose": True,
2222
"headless": False

‎examples/azure/search_graph_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
graph_config = {
2323
"llm": {
2424
"api_key": os.environ["AZURE_OPENAI_KEY"],
25-
"model": "azure/gpt-3.5-turbo",
25+
"model": "azure_openai/gpt-3.5-turbo",
2626
},
2727
"verbose": True,
2828
"headless": False

‎examples/azure/search_graph_schema_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ class Dishes(BaseModel):
3030
graph_config = {
3131
"llm": {
3232
"api_key": os.environ["AZURE_OPENAI_KEY"],
33-
"model": "azure/gpt-3.5-turbo",
33+
"model": "azure_openai/gpt-3.5-turbo",
3434
},
3535
"verbose": True,
3636
"headless": False

‎examples/azure/search_link_graph_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
graph_config = {
1616
"llm": {
1717
"api_key": os.environ["AZURE_OPENAI_KEY"],
18-
"model": "azure/gpt-3.5-turbo",
18+
"model": "azure_openai/gpt-3.5-turbo",
1919
},
2020
"verbose": True,
2121
"headless": False

‎examples/azure/smart_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
graph_config = {
2727
"llm": {
2828
"api_key": os.environ["AZURE_OPENAI_KEY"],
29-
"model": "azure/gpt-3.5-turbo",
29+
"model": "azure_openai/gpt-3.5-turbo",
3030
},
3131
"verbose": True,
3232
"headless": False

‎examples/azure/smart_scraper_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
graph_config = {
1515
"llm": {
1616
"api_key": os.environ["AZURE_OPENAI_KEY"],
17-
"model": "azure/gpt-3.5-turbo",
17+
"model": "azure_openai/gpt-3.5-turbo",
1818
},
1919
"verbose": True,
2020
"headless": False

‎examples/azure/smart_scraper_schema_azure.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
Basic example of scraping pipeline using SmartScraper with schema
33
"""
44

5-
import os, json
5+
import os
6+
import json
67
from typing import List
78
from pydantic import BaseModel, Field
89
from dotenv import load_dotenv
@@ -28,7 +29,7 @@ class Projects(BaseModel):
2829
graph_config = {
2930
"llm": {
3031
"api_key": os.environ["AZURE_OPENAI_KEY"],
31-
"model": "azure/gpt-3.5-turbo",
32+
"model": "azure_openai/gpt-3.5-turbo",
3233
},
3334
"verbose": True,
3435
"headless": False

‎examples/azure/xml_scraper_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
graph_config = {
2525
"llm": {
2626
"api_key": os.environ["AZURE_OPENAI_KEY"],
27-
"model": "azure/gpt-3.5-turbo",
27+
"model": "azure_openai/gpt-3.5-turbo",
2828
},
2929
"verbose": True,
3030
"headless": False

‎examples/azure/xml_scraper_graph_multi_azure.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
graph_config = {
2626
"llm": {
2727
"api_key": os.environ["AZURE_OPENAI_KEY"],
28-
"model": "azure/gpt-3.5-turbo",
28+
"model": "azure_openai/gpt-3.5-turbo",
2929
},
3030
"verbose": True,
3131
"headless": False

‎examples/ernie/custom_graph_ernie.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
from dotenv import load_dotenv
77

88
from langchain_openai import OpenAIEmbeddings
9-
from scrapegraphai.models import OpenAI
9+
from langchain_openai import ChatOpenAI
1010
from scrapegraphai.graphs import BaseGraph
1111
from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
1212

‎examples/extras/.env.example

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
OPENAI_API_KEY="OPENAI_API_KEY"
2+
BROWSER_BASE_PROJECT_ID="BROWSER_BASE_PROJECT_ID"
3+
BROWSER_BASE_API_KEY="BROWSERBASE_API_KEY"

‎examples/extras/browser_base_integration.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@
1818
graph_config = {
1919
"llm": {
2020
"api_key": os.getenv("OPENAI_API_KEY"),
21-
"model": "gpt-3.5-turbo",
21+
"model": "gpt-4o",
2222
},
2323
"browser_base": {
2424
"api_key": os.getenv("BROWSER_BASE_API_KEY"),
25-
"project_id": os.getenv("BROWSER_BASE_API_KEY"),
25+
"project_id": os.getenv("BROWSER_BASE_PROJECT_ID"),
2626
},
2727
"verbose": True,
2828
"headless": False,

‎examples/gemini/.env.example renamed to ‎examples/google_genai/.env.example

File renamed without changes.

‎examples/gemini/csv_scraper_gemini.py renamed to ‎examples/google_genai/csv_scraper_gemini.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
graph_config = {
2525
"llm": {
2626
"api_key": gemini_key,
27-
"model": "gemini-pro",
27+
"model": "google_genai/gemini-pro",
2828
},
2929
}
3030

‎examples/gemini/csv_scraper_graph_multi_gemini.py renamed to ‎examples/google_genai/csv_scraper_graph_multi_gemini.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
graph_config = {
2929
"llm": {
3030
"api_key": gemini_key,
31-
"model": "gemini-pro",
31+
"model": "google_genai/gemini-pro",
3232
},
3333
}
3434

‎examples/gemini/custom_graph_gemini.py renamed to ‎examples/google_genai/custom_graph_gemini.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
graph_config = {
1919
"llm": {
2020
"api_key": gemini_key,
21-
"model": "gemini-pro",
21+
"model": "google_genai/gemini-pro",
2222
"temperature": 0,
2323
"streaming": True
2424
},

‎examples/gemini/inputs/books.xml renamed to ‎examples/google_genai/inputs/books.xml

File renamed without changes.

‎examples/gemini/inputs/example.json renamed to ‎examples/google_genai/inputs/example.json

File renamed without changes.

‎examples/gemini/inputs/plain_html_example.txt renamed to ‎examples/google_genai/inputs/plain_html_example.txt

File renamed without changes.

‎examples/gemini/inputs/username.csv renamed to ‎examples/google_genai/inputs/username.csv

File renamed without changes.

‎examples/gemini/json_scraper_gemini.py renamed to ‎examples/google_genai/json_scraper_gemini.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
graph_config = {
2929
"llm": {
3030
"api_key": gemini_key,
31-
"model": "gemini-pro",
31+
"model": "google_genai/gemini-pro",
3232
},
3333
}
3434

0 commit comments

Comments
 (0)