Skip to content

Commit c9ade90

Browse files
VinciGit00VinciGit00
authored
Merge pull request #700 from ScrapeGraphAI/temp-1
Pre/beta
2 parents d116b77 + d0976dd commit c9ade90

File tree

60 files changed

+2566
-202
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+2566
-202
lines changed

‎CHANGELOG.md

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,41 @@
1-
## [1.23.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.23.0...v1.23.1) (2024-09-24)
1+
## [1.22.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.22.0-beta.2...v1.22.0-beta.3) (2024-09-25)
2+
23

34

45
### Bug Fixes
56

6-
* parse_node ([ceede46](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ceede4667312e7e295c7dfaf8a9e6570b45bd143))
7+
* update to pydantic documentation ([76ce257](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/76ce257efb9d9f46c0693472a1fe54b39e4eb1ef))
78

8-
## [1.23.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.22.0...v1.23.0) (2024-09-23)
9+
## [1.22.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.22.0-beta.1...v1.22.0-beta.2) (2024-09-25)
910

1011

11-
### Features
12+
### Bug Fixes
13+
14+
* node refiner + examples ([d55f6be](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d55f6bee4766f174abb2fdcd598542a9ca108a25))
1215

13-
* update search_link_graph ([de10b28](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/de10b281bab7385e250f4284ff3922dba38882f7))
16+
## [1.22.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.21.2-beta.2...v1.22.0-beta.1) (2024-09-24)
1417

15-
## [1.22.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.21.1...v1.22.0) (2024-09-22)
1618

1719

1820
### Features
1921

20-
* update search_link graph ([e724ae4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e724ae449282608507f7c28a39e655dc86a13aef))
22+
* add info to the dictionary for toghtherai ([3b5ee76](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3b5ee767cbb91cb0ca8e4691195d16c3b57140bb))
23+
* update exception ([3876cb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3876cb7be86e081065ca18c443647261a4b205d1))
24+
25+
## [1.21.2-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.21.2-beta.1...v1.21.2-beta.2) (2024-09-23)
26+
27+
28+
### Bug Fixes
29+
30+
* graph Iterator node ([8ce08ba](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8ce08baf01d7757c6fdcab0333405787c67d2dbc))
31+
* issue about parser ([7eda6bc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7eda6bc06bc4c32850029f54b9b4c22f3124296e))
32+
33+
## [1.21.2-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.21.1...v1.21.2-beta.1) (2024-09-22)
34+
35+
36+
### Bug Fixes
37+
38+
* chat for bedrock ([f9b121f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f9b121f7657e9eaf0b1b0e4a8574b8f1cbbd7c36))
2139

2240
## [1.21.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.21.0...v1.21.1) (2024-09-21)
2341

‎examples/anthropic/code_generator_graph_anthropic.py

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"""
2+
Basic example of scraping pipeline using Code Generator with schema
3+
"""
4+
5+
import os, json
6+
from typing import List
7+
from dotenv import load_dotenv
8+
from pydantic import BaseModel, Field
9+
from scrapegraphai.graphs import CodeGeneratorGraph
10+
11+
load_dotenv()
12+
13+
# ************************************************
14+
# Define the output schema for the graph
15+
# ************************************************
16+
17+
class Project(BaseModel):
18+
title: str = Field(description="The title of the project")
19+
description: str = Field(description="The description of the project")
20+
21+
class Projects(BaseModel):
22+
projects: List[Project]
23+
24+
# ************************************************
25+
# Define the configuration for the graph
26+
# ************************************************
27+
28+
anthropic_key = os.getenv("ANTHROPIC_API_KEY")
29+
30+
graph_config = {
31+
"llm": {
32+
"api_key":anthropic_key,
33+
"model": "anthropic/claude-3-haiku-20240307",
34+
},
35+
"verbose": True,
36+
"headless": False,
37+
"reduction": 2,
38+
"max_iterations": {
39+
"overall": 10,
40+
"syntax": 3,
41+
"execution": 3,
42+
"validation": 3,
43+
"semantic": 3
44+
},
45+
"output_file_name": "extracted_data.py"
46+
}
47+
48+
# ************************************************
49+
# Create the SmartScraperGraph instance and run it
50+
# ************************************************
51+
52+
code_generator_graph = CodeGeneratorGraph(
53+
prompt="List me all the projects with their description",
54+
source="https://perinim.github.io/projects/",
55+
schema=Projects,
56+
config=graph_config
57+
)
58+
59+
result = code_generator_graph.run()
60+
print(result)

‎examples/azure/code_generator_graph_azure.py

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
"""
2+
Basic example of scraping pipeline using Code Generator with schema
3+
"""
4+
5+
import os, json
6+
from typing import List
7+
from dotenv import load_dotenv
8+
from pydantic import BaseModel, Field
9+
from scrapegraphai.graphs import CodeGeneratorGraph
10+
11+
load_dotenv()
12+
13+
# ************************************************
14+
# Define the output schema for the graph
15+
# ************************************************
16+
17+
class Project(BaseModel):
18+
title: str = Field(description="The title of the project")
19+
description: str = Field(description="The description of the project")
20+
21+
class Projects(BaseModel):
22+
projects: List[Project]
23+
24+
# ************************************************
25+
# Define the configuration for the graph
26+
# ************************************************
27+
28+
graph_config = {
29+
"llm": {
30+
"api_key": os.environ["AZURE_OPENAI_KEY"],
31+
"model": "azure_openai/gpt-3.5-turbo",
32+
},
33+
"verbose": True,
34+
"headless": False,
35+
"reduction": 2,
36+
"max_iterations": {
37+
"overall": 10,
38+
"syntax": 3,
39+
"execution": 3,
40+
"validation": 3,
41+
"semantic": 3
42+
},
43+
"output_file_name": "extracted_data.py"
44+
}
45+
46+
# ************************************************
47+
# Create the SmartScraperGraph instance and run it
48+
# ************************************************
49+
50+
code_generator_graph = CodeGeneratorGraph(
51+
prompt="List me all the projects with their description",
52+
source="https://perinim.github.io/projects/",
53+
schema=Projects,
54+
config=graph_config
55+
)
56+
57+
result = code_generator_graph.run()
58+
print(result)

‎examples/bedrock/code_generator_graph_bedrock.py

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"""
2+
Basic example of scraping pipeline using Code Generator with schema
3+
"""
4+
5+
import os, json
6+
from typing import List
7+
from dotenv import load_dotenv
8+
from pydantic import BaseModel, Field
9+
from scrapegraphai.graphs import CodeGeneratorGraph
10+
11+
load_dotenv()
12+
13+
# ************************************************
14+
# Define the output schema for the graph
15+
# ************************************************
16+
17+
class Project(BaseModel):
18+
title: str = Field(description="The title of the project")
19+
description: str = Field(description="The description of the project")
20+
21+
class Projects(BaseModel):
22+
projects: List[Project]
23+
24+
# ************************************************
25+
# Define the configuration for the graph
26+
# ************************************************
27+
28+
29+
graph_config = {
30+
"llm": {
31+
"client": "client_name",
32+
"model": "bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
33+
"temperature": 0.0
34+
},
35+
"verbose": True,
36+
"headless": False,
37+
"reduction": 2,
38+
"max_iterations": {
39+
"overall": 10,
40+
"syntax": 3,
41+
"execution": 3,
42+
"validation": 3,
43+
"semantic": 3
44+
},
45+
"output_file_name": "extracted_data.py"
46+
}
47+
48+
# ************************************************
49+
# Create the SmartScraperGraph instance and run it
50+
# ************************************************
51+
52+
code_generator_graph = CodeGeneratorGraph(
53+
prompt="List me all the projects with their description",
54+
source="https://perinim.github.io/projects/",
55+
schema=Projects,
56+
config=graph_config
57+
)
58+
59+
result = code_generator_graph.run()
60+
print(result)

‎examples/deepseek/code_generator_graph_deepseek.py

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"""
2+
Basic example of scraping pipeline using Code Generator with schema
3+
"""
4+
5+
import os, json
6+
from typing import List
7+
from dotenv import load_dotenv
8+
from pydantic import BaseModel, Field
9+
from scrapegraphai.graphs import CodeGeneratorGraph
10+
11+
load_dotenv()
12+
13+
# ************************************************
14+
# Define the output schema for the graph
15+
# ************************************************
16+
17+
class Project(BaseModel):
18+
title: str = Field(description="The title of the project")
19+
description: str = Field(description="The description of the project")
20+
21+
class Projects(BaseModel):
22+
projects: List[Project]
23+
24+
# ************************************************
25+
# Define the configuration for the graph
26+
# ************************************************
27+
28+
deepseek_key = os.getenv("DEEPSEEK_APIKEY")
29+
30+
graph_config = {
31+
"llm": {
32+
"model": "deepseek/deepseek-chat",
33+
"api_key": deepseek_key,
34+
},
35+
"verbose": True,
36+
"headless": False,
37+
"reduction": 2,
38+
"max_iterations": {
39+
"overall": 10,
40+
"syntax": 3,
41+
"execution": 3,
42+
"validation": 3,
43+
"semantic": 3
44+
},
45+
"output_file_name": "extracted_data.py"
46+
}
47+
48+
# ************************************************
49+
# Create the SmartScraperGraph instance and run it
50+
# ************************************************
51+
52+
code_generator_graph = CodeGeneratorGraph(
53+
prompt="List me all the projects with their description",
54+
source="https://perinim.github.io/projects/",
55+
schema=Projects,
56+
config=graph_config
57+
)
58+
59+
result = code_generator_graph.run()
60+
print(result)

‎examples/ernie/code_generator_graph_ernie.py

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
"""
2+
Basic example of scraping pipeline using Code Generator with schema
3+
"""
4+
5+
import os, json
6+
from typing import List
7+
from dotenv import load_dotenv
8+
from pydantic import BaseModel, Field
9+
from scrapegraphai.graphs import CodeGeneratorGraph
10+
11+
load_dotenv()
12+
13+
# ************************************************
14+
# Define the output schema for the graph
15+
# ************************************************
16+
17+
class Project(BaseModel):
18+
title: str = Field(description="The title of the project")
19+
description: str = Field(description="The description of the project")
20+
21+
class Projects(BaseModel):
22+
projects: List[Project]
23+
24+
# ************************************************
25+
# Define the configuration for the graph
26+
# ************************************************
27+
28+
openai_key = os.getenv("OPENAI_APIKEY")
29+
30+
graph_config = {
31+
"llm": {
32+
"model": "ernie/ernie-bot-turbo",
33+
"ernie_client_id": "<ernie_client_id>",
34+
"ernie_client_secret": "<ernie_client_secret>",
35+
"temperature": 0.1
36+
},
37+
"verbose": True,
38+
"headless": False,
39+
"reduction": 2,
40+
"max_iterations": {
41+
"overall": 10,
42+
"syntax": 3,
43+
"execution": 3,
44+
"validation": 3,
45+
"semantic": 3
46+
},
47+
"output_file_name": "extracted_data.py"
48+
}
49+
50+
# ************************************************
51+
# Create the SmartScraperGraph instance and run it
52+
# ************************************************
53+
54+
code_generator_graph = CodeGeneratorGraph(
55+
prompt="List me all the projects with their description",
56+
source="https://perinim.github.io/projects/",
57+
schema=Projects,
58+
config=graph_config
59+
)
60+
61+
result = code_generator_graph.run()
62+
print(result)

0 commit comments

Comments
 (0)