Skip to content

Commit 3a9d556

Browse files
VinciGit00VinciGit00
authored
Merge branch 'pre/beta' into add-openai-supported-model-gpt-4o-mini
2 parents 4f079cf + a3d0aac commit 3a9d556

38 files changed

+778
-277
lines changed

‎CHANGELOG.md

Lines changed: 0 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,5 @@
1-
## [1.9.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0...v1.9.1) (2024-07-12)
21

32

4-
### Bug Fixes
5-
6-
* solve a burr integration ([881290b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/881290b5066b39c505532656671fbf65f8fc312c))
7-
8-
## [1.9.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.8.0...v1.9.0) (2024-07-09)
9-
10-
11-
### Features
12-
13-
* add fireworks integration ([df0e310](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/df0e3108299071b849d7e055bd11d72764d24f08))
14-
* add integration for infos ([3bf5f57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3bf5f570a8f8e1b037a7ad3c9f583261a1536421))
15-
* add integrations for markdown files ([2804434](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2804434a9ee12c52ae8956a88b1778a4dd3ec32f))
16-
* add vertexai integration ([119514b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/119514bdfc2a16dfb8918b0c34ae7cc43a01384c))
17-
* improve md prompt recognition ([5fe694b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5fe694b6b4545a5091d16110318b992acfca4f58))
18-
19-
20-
### Bug Fixes
21-
22-
* add test ([3a537ee](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3a537eec6fef1743924a9aa5cef0ba2f8d44bf11))
23-
* fix pyproject.toml ([7570bf8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7570bf8294e49bc54ec9e296aaadb763873390ca))
24-
25-
26-
### chore
27-
28-
* **Docker:** fix port number ([afeb81f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/afeb81f77a884799192d79dcac85666190fb1c9d))
29-
* **CI:** fix pylint workflow ([583c321](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/583c32106e827f50235d8fc69511652fd4b07a35))
30-
* **rye:** rebuild lockfiles ([27c2dd2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/27c2dd23517a7e4b14fafd00320a8b81f73145dc))
31-
32-
33-
### Docs
34-
35-
* **roadmap:** fix urls ([14faba4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/14faba4f00dd9f947f8dc5e0b51be49ea684179f))
36-
* **roadmap:** next steps ([3e644f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e644f498f05eb505fbd4e94b144c81567569aaa))
37-
38-
39-
### CI
40-
41-
* **release:** 1.8.1-beta.1 [skip ci] ([8f9f96f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f9f96f7e7ff41d2fff5bbbf18bf4fc85d4f98b3))
42-
* **release:** 1.9.0-beta.1 [skip ci] ([146432d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/146432d476f775510441b062935adc47190141e2))
43-
* **release:** 1.9.0-beta.2 [skip ci] ([5cb5fbf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5cb5fbf5503eec9b34a6691eb993716cc9a821d6))
44-
453
## [1.9.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0-beta.1...v1.9.0-beta.2) (2024-07-05)
464

475

‎examples/anthropic/search_link_graph_haiku.py

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
from langchain_openai import AzureChatOpenAI
9+
from langchain_openai import AzureOpenAIEmbeddings
10+
11+
# ************************************************
12+
# Define the configuration for the graph
13+
# ************************************************
14+
15+
load_dotenv()
16+
17+
llm_model_instance = AzureChatOpenAI(
18+
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
19+
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
20+
)
21+
22+
embedder_model_instance = AzureOpenAIEmbeddings(
23+
azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
24+
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
25+
)
26+
27+
# ************************************************
28+
# Create the SmartScraperGraph instance and run it
29+
# ************************************************
30+
31+
graph_config = {
32+
"llm": {"model_instance": llm_model_instance},
33+
"embeddings": {"model_instance": embedder_model_instance}
34+
}
35+
36+
# ************************************************
37+
# Create the SearchGraph instance and run it
38+
# ************************************************
39+
40+
search_graph = SearchGraph(
41+
prompt="List me the best escursions near Trento",
42+
config=graph_config
43+
)
44+
45+
result = search_graph.run()
46+
print(result)
47+
48+
# ************************************************
49+
# Get graph execution info
50+
# ************************************************
51+
52+
graph_exec_info = search_graph.get_execution_info()
53+
print(prettify_exec_info(graph_exec_info))
54+
55+
# Save to json and csv
56+
convert_to_csv(result, "result")
57+
convert_to_json(result, "result")

‎examples/azure/search_link_graph_azure.py

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
load_dotenv()
14+
15+
groq_key = os.getenv("GROQ_APIKEY")
16+
17+
graph_config = {
18+
"llm": {
19+
"model": "groq/gemma-7b-it",
20+
"api_key": groq_key,
21+
"temperature": 0
22+
},
23+
"embeddings": {
24+
"model": "ollama/nomic-embed-text",
25+
"temperature": 0,
26+
# "base_url": "http://localhost:11434", # set ollama URL arbitrarily
27+
},
28+
"headless": False
29+
}
30+
31+
# ************************************************
32+
# Create the SearchGraph instance and run it
33+
# ************************************************
34+
35+
search_graph = SearchGraph(
36+
prompt="List me the best escursions near Trento",
37+
config=graph_config
38+
)
39+
40+
result = search_graph.run()
41+
print(result)
42+
43+
# ************************************************
44+
# Get graph execution info
45+
# ************************************************
46+
47+
graph_exec_info = search_graph.get_execution_info()
48+
print(prettify_exec_info(graph_exec_info))
49+
50+
# Save to json and csv
51+
convert_to_csv(result, "result")
52+
convert_to_json(result, "result")

‎examples/bedrock/search_link_graph_bedrock.py

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
graph_config = {
14+
"llm": {
15+
"client": "client_name",
16+
"model": "bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
17+
"temperature": 0.0
18+
},
19+
"embeddings": {
20+
"model": "bedrock/cohere.embed-multilingual-v3"
21+
}
22+
}
23+
24+
# ************************************************
25+
# Create the SearchGraph instance and run it
26+
# ************************************************
27+
28+
search_graph = SearchGraph(
29+
prompt="List me the best escursions near Trento",
30+
config=graph_config
31+
)
32+
33+
result = search_graph.run()
34+
print(result)
35+
36+
# ************************************************
37+
# Get graph execution info
38+
# ************************************************
39+
40+
graph_exec_info = search_graph.get_execution_info()
41+
print(prettify_exec_info(graph_exec_info))
42+
43+
# Save to json and csv
44+
convert_to_csv(result, "result")
45+
convert_to_json(result, "result")

‎examples/deepseek/search_link_graph_deepseek.py

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
load_dotenv()
14+
15+
deepseek_key = os.getenv("DEEPSEEK_APIKEY")
16+
17+
graph_config = {
18+
"llm": {
19+
"model": "deepseek-chat",
20+
"openai_api_key": deepseek_key,
21+
"openai_api_base": 'https://api.deepseek.com/v1',
22+
},
23+
"embeddings": {
24+
"model": "ollama/nomic-embed-text",
25+
"temperature": 0,
26+
# "base_url": "http://localhost:11434", # set ollama URL arbitrarily
27+
},
28+
"verbose": True,
29+
}
30+
31+
# ************************************************
32+
# Create the SearchGraph instance and run it
33+
# ************************************************
34+
35+
search_graph = SearchGraph(
36+
prompt="List me the best escursions near Trento",
37+
config=graph_config
38+
)
39+
40+
result = search_graph.run()
41+
print(result)
42+
43+
# ************************************************
44+
# Get graph execution info
45+
# ************************************************
46+
47+
graph_exec_info = search_graph.get_execution_info()
48+
print(prettify_exec_info(graph_exec_info))
49+
50+
# Save to json and csv
51+
convert_to_csv(result, "result")
52+
convert_to_json(result, "result")

‎examples/ernie/search_graph_ernie.py

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,18 @@
1212
# Define the configuration for the graph
1313
# ************************************************
1414

15-
openai_key = os.getenv("OPENAI_APIKEY")
16-
1715
graph_config = {
1816
"llm": {
19-
"api_key": openai_key,
20-
"model": "gpt-3.5-turbo",
21-
},
22-
"max_results": 2,
23-
"verbose": True,
17+
"model": "ernie-bot-turbo",
18+
"ernie_client_id": "<ernie_client_id>",
19+
"ernie_client_secret": "<ernie_client_secret>",
20+
"temperature": 0.1
21+
},
22+
"embeddings": {
23+
"model": "ollama/nomic-embed-text",
24+
"temperature": 0,
25+
"base_url": "http://localhost:11434"},
26+
"library": "beautifulsoup"
2427
}
2528

2629
# ************************************************

‎examples/ernie/search_link_graph_ernie.py

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
from scrapegraphai.graphs import SearchGraph
5+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
6+
7+
# ************************************************
8+
# Define the configuration for the graph
9+
# ************************************************
10+
11+
graph_config = {
12+
"llm": {
13+
"model": "ernie-bot-turbo",
14+
"ernie_client_id": "<ernie_client_id>",
15+
"ernie_client_secret": "<ernie_client_secret>",
16+
"temperature": 0.1
17+
},
18+
"embeddings": {
19+
"model": "ollama/nomic-embed-text",
20+
"temperature": 0,
21+
"base_url": "http://localhost:11434"},
22+
"library": "beautifulsoup"
23+
}
24+
25+
# ************************************************
26+
# Create the SearchGraph instance and run it
27+
# ************************************************
28+
29+
search_graph = SearchGraph(
30+
prompt="List me the best escursions near Trento",
31+
config=graph_config
32+
)
33+
34+
result = search_graph.run()
35+
print(result)
36+
37+
# ************************************************
38+
# Get graph execution info
39+
# ************************************************
40+
41+
graph_exec_info = search_graph.get_execution_info()
42+
print(prettify_exec_info(graph_exec_info))
43+
44+
# Save to json and csv
45+
convert_to_csv(result, "result")
46+
convert_to_json(result, "result")

‎examples/fireworks/search_link_graph_fireworks.py

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
load_dotenv()
14+
15+
fireworks_api_key = os.getenv("FIREWORKS_APIKEY")
16+
17+
graph_config = {
18+
"llm": {
19+
"api_key": fireworks_api_key,
20+
"model": "fireworks/accounts/fireworks/models/mixtral-8x7b-instruct"
21+
},
22+
"embeddings": {
23+
"model": "ollama/nomic-embed-text",
24+
"temperature": 0,
25+
# "base_url": "http://localhost:11434", # set ollama URL arbitrarily
26+
},
27+
"max_results": 2,
28+
"verbose": True,
29+
"headless": False,
30+
}
31+
# ************************************************
32+
# Create the SearchGraph instance and run it
33+
# ************************************************
34+
35+
search_graph = SearchGraph(
36+
prompt="List me the best escursions near Trento",
37+
config=graph_config
38+
)
39+
40+
result = search_graph.run()
41+
print(result)
42+
43+
# ************************************************
44+
# Get graph execution info
45+
# ************************************************
46+
47+
graph_exec_info = search_graph.get_execution_info()
48+
print(prettify_exec_info(graph_exec_info))
49+
50+
# Save to json and csv
51+
convert_to_csv(result, "result")
52+
convert_to_json(result, "result")

0 commit comments

Comments
 (0)