Skip to content

Commit 4b2581d

Browse files
PeriniMPeriniM
authored
Merge pull request #478 from ScrapeGraphAI/pre/beta
big update
2 parents 99dc849 + 7a3c21b commit 4b2581d

File tree

60 files changed

+930
-654
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+930
-654
lines changed

‎CHANGELOG.md

Lines changed: 40 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,22 @@
1+
## [1.10.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.0-beta.4...v1.10.0-beta.5) (2024-07-20)
2+
3+
4+
### Bug Fixes
5+
6+
* parse_node ([07f1e23](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/07f1e23d235db1a0db2cb155f10b73b0bf882269))
7+
8+
## [1.10.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.0-beta.3...v1.10.0-beta.4) (2024-07-20)
9+
10+
11+
### Bug Fixes
12+
13+
* azure models ([03f4a3a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/03f4a3aa29c42a9a312c4afb6818de3450e7cedf))
14+
15+
16+
### CI
17+
18+
* **release:** 1.9.2 [skip ci] ([b4b90b3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b4b90b3c121911de68a860640419907ca7674953))
19+
120
## [1.9.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.1...v1.9.2) (2024-07-20)
221

322

@@ -13,46 +32,47 @@
1332
## [1.9.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0...v1.9.1) (2024-07-12)
1433

1534

35+
1636
### Bug Fixes
1737

18-
* solve a burr integration ([881290b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/881290b5066b39c505532656671fbf65f8fc312c))
38+
* add gpt o mini for azure ([77777c8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/77777c898d1fad40f340b06c5b36d35b65409ea6))
1939

20-
## [1.9.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.8.0...v1.9.0) (2024-07-09)
40+
## [1.10.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.0-beta.1...v1.10.0-beta.2) (2024-07-19)
2141

2242

2343
### Features
2444

25-
* add fireworks integration ([df0e310](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/df0e3108299071b849d7e055bd11d72764d24f08))
26-
* add integration for infos ([3bf5f57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3bf5f570a8f8e1b037a7ad3c9f583261a1536421))
27-
* add integrations for markdown files ([2804434](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2804434a9ee12c52ae8956a88b1778a4dd3ec32f))
28-
* add vertexai integration ([119514b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/119514bdfc2a16dfb8918b0c34ae7cc43a01384c))
29-
* improve md prompt recognition ([5fe694b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5fe694b6b4545a5091d16110318b992acfca4f58))
45+
* add gpt4o omni ([431edb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/431edb7bb2504f4c1335c3ae3ce2f91867fa7222))
3046

47+
## [1.10.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.1...v1.10.0-beta.1) (2024-07-19)
3148

32-
### Bug Fixes
3349

34-
* add test ([3a537ee](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3a537eec6fef1743924a9aa5cef0ba2f8d44bf11))
35-
* fix pyproject.toml ([7570bf8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7570bf8294e49bc54ec9e296aaadb763873390ca))
50+
### Features
3651

52+
* add searchngx integration ([5c92186](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c9218608140bf694fbfd96aa90276bc438bb475))
53+
* refactoring_to_md function ([602dd00](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/602dd00209ee1d72a1223fc4793759450921fcf9))
3754

38-
### chore
3955

40-
* **Docker:** fix port number ([afeb81f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/afeb81f77a884799192d79dcac85666190fb1c9d))
41-
* **CI:** fix pylint workflow ([583c321](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/583c32106e827f50235d8fc69511652fd4b07a35))
42-
* **rye:** rebuild lockfiles ([27c2dd2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/27c2dd23517a7e4b14fafd00320a8b81f73145dc))
56+
### Bug Fixes
4357

58+
* search link node ([cf3ab55](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf3ab5564ae5c415c63d1771b32ea68f5169ca82))
4459

45-
### Docs
4660

47-
* **roadmap:** fix urls ([14faba4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/14faba4f00dd9f947f8dc5e0b51be49ea684179f))
48-
* **roadmap:** next steps ([3e644f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e644f498f05eb505fbd4e94b144c81567569aaa))
61+
### chore
62+
63+
* correct search engine name ([7ba2f6a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7ba2f6ae0b9d2e9336e973e1f57ab8355c739e27))
64+
* remove unused import ([fd1b7cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/fd1b7cb24a7c252277607abde35826e3c58e34ef))
65+
* remove unused workflow ([5c6dd8d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c6dd8de4da08f09b5dd93c525d14b44778c9659))
66+
* **ci:** upgrade lockfiles ([c7b05a4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c7b05a4993df14d6ed4848121a3cd209571232f7))
67+
* upgrade tiktoken ([7314bc3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7314bc383068db590662bf7e512f799529308991))
4968

5069

5170
### CI
5271

53-
* **release:** 1.8.1-beta.1 [skip ci] ([8f9f96f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f9f96f7e7ff41d2fff5bbbf18bf4fc85d4f98b3))
54-
* **release:** 1.9.0-beta.1 [skip ci] ([146432d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/146432d476f775510441b062935adc47190141e2))
55-
* **release:** 1.9.0-beta.2 [skip ci] ([5cb5fbf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5cb5fbf5503eec9b34a6691eb993716cc9a821d6))
72+
* **release:** 1.9.0-beta.3 [skip ci] ([d3e63d9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d3e63d91be79f74e8a3fdb00e692d546c24cead5))
73+
* **release:** 1.9.0-beta.4 [skip ci] ([2fa04b5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2fa04b58159abf7af890ebc0768fe23d51bf177f))
74+
* **release:** 1.9.0-beta.5 [skip ci] ([bb62439](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bb624399cfc3924825892dd48697fc298ad3b002))
75+
* **release:** 1.9.0-beta.6 [skip ci] ([54a69de](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/54a69de69e8077e02fd5584783ca62cc2e0ec5bb))
5676

5777
## [1.9.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0-beta.1...v1.9.0-beta.2) (2024-07-05)
5878

‎examples/anthropic/search_link_graph_haiku.py

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
from langchain_openai import AzureChatOpenAI
9+
from langchain_openai import AzureOpenAIEmbeddings
10+
11+
# ************************************************
12+
# Define the configuration for the graph
13+
# ************************************************
14+
15+
load_dotenv()
16+
17+
llm_model_instance = AzureChatOpenAI(
18+
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
19+
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
20+
)
21+
22+
embedder_model_instance = AzureOpenAIEmbeddings(
23+
azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
24+
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
25+
)
26+
27+
# ************************************************
28+
# Create the SmartScraperGraph instance and run it
29+
# ************************************************
30+
31+
graph_config = {
32+
"llm": {"model_instance": llm_model_instance},
33+
"embeddings": {"model_instance": embedder_model_instance}
34+
}
35+
36+
# ************************************************
37+
# Create the SearchGraph instance and run it
38+
# ************************************************
39+
40+
search_graph = SearchGraph(
41+
prompt="List me the best escursions near Trento",
42+
config=graph_config
43+
)
44+
45+
result = search_graph.run()
46+
print(result)
47+
48+
# ************************************************
49+
# Get graph execution info
50+
# ************************************************
51+
52+
graph_exec_info = search_graph.get_execution_info()
53+
print(prettify_exec_info(graph_exec_info))
54+
55+
# Save to json and csv
56+
convert_to_csv(result, "result")
57+
convert_to_json(result, "result")

‎examples/azure/csv_scraper_azure.py

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,6 @@
55
import os
66
from dotenv import load_dotenv
77
import pandas as pd
8-
from langchain_openai import AzureChatOpenAI
9-
from langchain_openai import AzureOpenAIEmbeddings
108
from scrapegraphai.graphs import CSVScraperGraph
119
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
1210
load_dotenv()
@@ -24,25 +22,15 @@
2422
# ************************************************
2523
# Define the configuration for the graph
2624
# ************************************************
27-
28-
llm_model_instance = AzureChatOpenAI(
29-
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
30-
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
31-
)
32-
33-
embedder_model_instance = AzureOpenAIEmbeddings(
34-
azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
35-
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
36-
)
37-
38-
# ************************************************
39-
# Create the JSONScraperGraph instance and run it
40-
# ************************************************
41-
4225
graph_config = {
43-
"llm": {"model_instance": llm_model_instance},
44-
"embeddings": {"model_instance": embedder_model_instance}
26+
"llm": {
27+
"api_key": os.environ["AZURE_OPENAI_KEY"],
28+
"model": "azure/gpt-3.5-turbo",
29+
},
30+
"verbose": True,
31+
"headless": False
4532
}
33+
4634
# ************************************************
4735
# Create the CSVScraperGraph instance and run it
4836
# ************************************************

‎examples/azure/csv_scraper_graph_multi_azure.py

Lines changed: 7 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
from dotenv import load_dotenv
77
import pandas as pd
88
from scrapegraphai.graphs import CSVScraperMultiGraph
9-
from langchain_openai import AzureChatOpenAI
10-
from langchain_openai import AzureOpenAIEmbeddings
119
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
1210

1311
load_dotenv()
@@ -24,19 +22,15 @@
2422
# ************************************************
2523
# Define the configuration for the graph
2624
# ************************************************
27-
llm_model_instance = AzureChatOpenAI(
28-
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
29-
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
30-
)
31-
32-
embedder_model_instance = AzureOpenAIEmbeddings(
33-
azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
34-
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
35-
)
3625
graph_config = {
37-
"llm": {"model_instance": llm_model_instance},
38-
"embeddings": {"model_instance": embedder_model_instance}
26+
"llm": {
27+
"api_key": os.environ["AZURE_OPENAI_KEY"],
28+
"model": "azure/gpt-3.5-turbo",
29+
},
30+
"verbose": True,
31+
"headless": False
3932
}
33+
4034
# ************************************************
4135
# Create the CSVScraperMultiGraph instance and run it
4236
# ************************************************

‎examples/azure/custom_graph_azure.py

Lines changed: 0 additions & 117 deletions
This file was deleted.

‎examples/azure/json_scraper_azure.py

Lines changed: 6 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,11 @@
44

55
import os
66
from dotenv import load_dotenv
7-
from langchain_openai import AzureChatOpenAI
8-
from langchain_openai import AzureOpenAIEmbeddings
97
from scrapegraphai.graphs import JSONScraperGraph
108
from scrapegraphai.utils import prettify_exec_info
119

12-
13-
# required environment variable in .env
14-
# AZURE_OPENAI_ENDPOINT
15-
# AZURE_OPENAI_CHAT_DEPLOYMENT_NAME
16-
# MODEL_NAME
17-
# AZURE_OPENAI_API_KEY
18-
# OPENAI_API_TYPE
19-
# AZURE_OPENAI_API_VERSION
20-
# AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME
2110
load_dotenv()
2211

23-
2412
FILE_NAME = "inputs/example.json"
2513
curr_dir = os.path.dirname(os.path.realpath(__file__))
2614
file_path = os.path.join(curr_dir, FILE_NAME)
@@ -32,23 +20,13 @@
3220
# Initialize the model instances
3321
# ************************************************
3422

35-
llm_model_instance = AzureChatOpenAI(
36-
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
37-
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
38-
)
39-
40-
embedder_model_instance = AzureOpenAIEmbeddings(
41-
azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
42-
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
43-
)
44-
45-
# ************************************************
46-
# Create the JSONScraperGraph instance and run it
47-
# ************************************************
48-
4923
graph_config = {
50-
"llm": {"model_instance": llm_model_instance},
51-
"embeddings": {"model_instance": embedder_model_instance}
24+
"llm": {
25+
"api_key": os.environ["AZURE_OPENAI_KEY"],
26+
"model": "azure/gpt-3.5-turbo",
27+
},
28+
"verbose": True,
29+
"headless": False
5230
}
5331

5432
smart_scraper_graph = JSONScraperGraph(

0 commit comments

Comments
 (0)