Skip to content

Commit 1aa9c6e

Browse files
f-aguzzif-aguzzi
committed
chore(examples): add vertex examples, rename genai examples
1 parent ee078cb commit 1aa9c6e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+1437
-0
lines changed

‎examples/gemini/.env.example renamed to ‎examples/google_genai/.env.example

File renamed without changes.

‎examples/gemini/csv_scraper_gemini.py renamed to ‎examples/google_genai/csv_scraper_gemini.py

File renamed without changes.

‎examples/gemini/csv_scraper_graph_multi_gemini.py renamed to ‎examples/google_genai/csv_scraper_graph_multi_gemini.py

File renamed without changes.

‎examples/gemini/custom_graph_gemini.py renamed to ‎examples/google_genai/custom_graph_gemini.py

File renamed without changes.

‎examples/gemini/inputs/books.xml renamed to ‎examples/google_genai/inputs/books.xml

File renamed without changes.

‎examples/gemini/inputs/example.json renamed to ‎examples/google_genai/inputs/example.json

File renamed without changes.

‎examples/gemini/inputs/plain_html_example.txt renamed to ‎examples/google_genai/inputs/plain_html_example.txt

File renamed without changes.

‎examples/gemini/inputs/username.csv renamed to ‎examples/google_genai/inputs/username.csv

File renamed without changes.

‎examples/gemini/json_scraper_gemini.py renamed to ‎examples/google_genai/json_scraper_gemini.py

File renamed without changes.

‎examples/gemini/json_scraper_multi_gemini.py renamed to ‎examples/google_genai/json_scraper_multi_gemini.py

File renamed without changes.

‎examples/gemini/pdf_scraper_graph_gemini.py renamed to ‎examples/google_genai/pdf_scraper_graph_gemini.py

File renamed without changes.

‎examples/gemini/pdf_scraper_multi_gemini.py renamed to ‎examples/google_genai/pdf_scraper_multi_gemini.py

File renamed without changes.

‎examples/gemini/readme.md renamed to ‎examples/google_genai/readme.md

File renamed without changes.

‎examples/gemini/scrape_plain_text_gemini.py renamed to ‎examples/google_genai/scrape_plain_text_gemini.py

File renamed without changes.

‎examples/gemini/scrape_xml_gemini.py renamed to ‎examples/google_genai/scrape_xml_gemini.py

File renamed without changes.

‎examples/gemini/script_generator_gemini.py renamed to ‎examples/google_genai/script_generator_gemini.py

File renamed without changes.

‎examples/gemini/script_multi_generator_gemini.py renamed to ‎examples/google_genai/script_multi_generator_gemini.py

File renamed without changes.

‎examples/gemini/search_graph_gemini.py renamed to ‎examples/google_genai/search_graph_gemini.py

File renamed without changes.

‎examples/gemini/search_graph_schema_gemini.py renamed to ‎examples/google_genai/search_graph_schema_gemini.py

File renamed without changes.

‎examples/gemini/search_link_graph_gemini.py renamed to ‎examples/google_genai/search_link_graph_gemini.py

File renamed without changes.

‎examples/gemini/smart_scraper_gemini.py renamed to ‎examples/google_genai/smart_scraper_gemini.py

File renamed without changes.

‎examples/gemini/smart_scraper_multi_gemini.py renamed to ‎examples/google_genai/smart_scraper_multi_gemini.py

File renamed without changes.

‎examples/gemini/smart_scraper_schema_gemini.py renamed to ‎examples/google_genai/smart_scraper_schema_gemini.py

File renamed without changes.

‎examples/gemini/xml_scraper_gemini.py renamed to ‎examples/google_genai/xml_scraper_gemini.py

File renamed without changes.

‎examples/gemini/xml_scraper_graph_multi_gemini.py renamed to ‎examples/google_genai/xml_scraper_graph_multi_gemini.py

File renamed without changes.

‎examples/google_vertexai/.env.example

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
GOOGLE_APIKEY="your google api key"

‎examples/google_vertexai/csv_scraper_gemini.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
"""
2+
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
3+
"""
4+
5+
import os
6+
from dotenv import load_dotenv
7+
import pandas as pd
8+
from scrapegraphai.graphs import CSVScraperGraph
9+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
10+
11+
load_dotenv()
12+
13+
# ************************************************
14+
# Read the csv file
15+
# ************************************************
16+
17+
text = pd.read_csv("inputs/username.csv")
18+
19+
# ************************************************
20+
# Define the configuration for the graph
21+
# ************************************************
22+
gemini_key = os.getenv("GOOGLE_APIKEY")
23+
24+
graph_config = {
25+
"llm": {
26+
"api_key": gemini_key,
27+
"model": "google_vertexai/gemini-1.5-pro",
28+
},
29+
}
30+
31+
# ************************************************
32+
# Create the CSVScraperGraph instance and run it
33+
# ************************************************
34+
35+
csv_scraper_graph = CSVScraperGraph(
36+
prompt="List me all the last names",
37+
source=str(text), # Pass the content of the file, not the file object
38+
config=graph_config
39+
)
40+
41+
result = csv_scraper_graph.run()
42+
print(result)
43+
44+
# ************************************************
45+
# Get graph execution info
46+
# ************************************************
47+
48+
graph_exec_info = csv_scraper_graph.get_execution_info()
49+
print(prettify_exec_info(graph_exec_info))
50+
51+
# Save to json or csv
52+
convert_to_csv(result, "result")
53+
convert_to_json(result, "result")

‎examples/google_vertexai/csv_scraper_graph_multi_gemini.py

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
"""
2+
Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
3+
"""
4+
5+
import os
6+
from dotenv import load_dotenv
7+
import pandas as pd
8+
from scrapegraphai.graphs import CSVScraperMultiGraph
9+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
10+
11+
load_dotenv()
12+
# ************************************************
13+
# Read the CSV file
14+
# ************************************************
15+
16+
FILE_NAME = "inputs/username.csv"
17+
curr_dir = os.path.dirname(os.path.realpath(__file__))
18+
file_path = os.path.join(curr_dir, FILE_NAME)
19+
20+
text = pd.read_csv(file_path)
21+
22+
# ************************************************
23+
# Define the configuration for the graph
24+
# ************************************************
25+
26+
gemini_key = os.getenv("GOOGLE_APIKEY")
27+
28+
graph_config = {
29+
"llm": {
30+
"api_key": gemini_key,
31+
"model": "google_vertexai/gemini-1.5-pro",
32+
},
33+
}
34+
35+
# ************************************************
36+
# Create the CSVScraperMultiGraph instance and run it
37+
# ************************************************
38+
39+
csv_scraper_graph = CSVScraperMultiGraph(
40+
prompt="List me all the last names",
41+
source=[str(text), str(text)],
42+
config=graph_config
43+
)
44+
45+
result = csv_scraper_graph.run()
46+
print(result)
47+
48+
# ************************************************
49+
# Get graph execution info
50+
# ************************************************
51+
52+
graph_exec_info = csv_scraper_graph.get_execution_info()
53+
print(prettify_exec_info(graph_exec_info))
54+
55+
# Save to json or csv
56+
convert_to_csv(result, "result")
57+
convert_to_json(result, "result")

‎examples/google_vertexai/custom_graph_gemini.py

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
"""
2+
Example of custom graph using Gemini Google model
3+
"""
4+
5+
import os
6+
from dotenv import load_dotenv
7+
from scrapegraphai.models import Gemini
8+
from scrapegraphai.graphs import BaseGraph
9+
from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode
10+
load_dotenv()
11+
12+
# ************************************************
13+
# Define the configuration for the graph
14+
# ************************************************
15+
16+
gemini_key = os.getenv("GOOGLE_APIKEY")
17+
18+
graph_config = {
19+
"llm": {
20+
"api_key": gemini_key,
21+
"model": "google_vertexai/gemini-1.5-pro",
22+
"temperature": 0,
23+
"streaming": True
24+
},
25+
}
26+
27+
# ************************************************
28+
# Define the graph nodes
29+
# ************************************************
30+
31+
llm_model = Gemini(graph_config["llm"])
32+
33+
# define the nodes for the graph
34+
fetch_node = FetchNode(
35+
input="url | local_dir",
36+
output=["doc"],
37+
)
38+
parse_node = ParseNode(
39+
input="doc",
40+
output=["parsed_doc"],
41+
node_config={"chunk_size": 4096}
42+
)
43+
rag_node = RAGNode(
44+
input="user_prompt & (parsed_doc | doc)",
45+
output=["relevant_chunks"],
46+
node_config={"llm": llm_model},
47+
)
48+
generate_answer_node = GenerateAnswerNode(
49+
input="user_prompt & (relevant_chunks | parsed_doc | doc)",
50+
output=["answer"],
51+
node_config={"llm": llm_model},
52+
)
53+
54+
# ************************************************
55+
# Create the graph by defining the connections
56+
# ************************************************
57+
58+
graph = BaseGraph(
59+
nodes={
60+
fetch_node,
61+
parse_node,
62+
rag_node,
63+
generate_answer_node,
64+
},
65+
edges={
66+
(fetch_node, parse_node),
67+
(parse_node, rag_node),
68+
(rag_node, generate_answer_node)
69+
},
70+
entry_point=fetch_node
71+
)
72+
73+
# ************************************************
74+
# Execute the graph
75+
# ************************************************
76+
77+
result, execution_info = graph.execute({
78+
"user_prompt": "List me the projects with their description",
79+
"url": "https://perinim.github.io/projects/"
80+
})
81+
82+
# get the answer from the result
83+
result = result.get("answer", "No answer found.")
84+
print(result)

‎examples/google_vertexai/inputs/books.xml

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
<?xml version="1.0"?>
2+
<catalog>
3+
<book id="bk101">
4+
<author>Gambardella, Matthew</author>
5+
<title>XML Developer's Guide</title>
6+
<genre>Computer</genre>
7+
<price>44.95</price>
8+
<publish_date>2000-10-01</publish_date>
9+
<description>An in-depth look at creating applications
10+
with XML.</description>
11+
</book>
12+
<book id="bk102">
13+
<author>Ralls, Kim</author>
14+
<title>Midnight Rain</title>
15+
<genre>Fantasy</genre>
16+
<price>5.95</price>
17+
<publish_date>2000-12-16</publish_date>
18+
<description>A former architect battles corporate zombies,
19+
an evil sorceress, and her own childhood to become queen
20+
of the world.</description>
21+
</book>
22+
<book id="bk103">
23+
<author>Corets, Eva</author>
24+
<title>Maeve Ascendant</title>
25+
<genre>Fantasy</genre>
26+
<price>5.95</price>
27+
<publish_date>2000-11-17</publish_date>
28+
<description>After the collapse of a nanotechnology
29+
society in England, the young survivors lay the
30+
foundation for a new society.</description>
31+
</book>
32+
<book id="bk104">
33+
<author>Corets, Eva</author>
34+
<title>Oberon's Legacy</title>
35+
<genre>Fantasy</genre>
36+
<price>5.95</price>
37+
<publish_date>2001-03-10</publish_date>
38+
<description>In post-apocalypse England, the mysterious
39+
agent known only as Oberon helps to create a new life
40+
for the inhabitants of London. Sequel to Maeve
41+
Ascendant.</description>
42+
</book>
43+
<book id="bk105">
44+
<author>Corets, Eva</author>
45+
<title>The Sundered Grail</title>
46+
<genre>Fantasy</genre>
47+
<price>5.95</price>
48+
<publish_date>2001-09-10</publish_date>
49+
<description>The two daughters of Maeve, half-sisters,
50+
battle one another for control of England. Sequel to
51+
Oberon's Legacy.</description>
52+
</book>
53+
<book id="bk106">
54+
<author>Randall, Cynthia</author>
55+
<title>Lover Birds</title>
56+
<genre>Romance</genre>
57+
<price>4.95</price>
58+
<publish_date>2000-09-02</publish_date>
59+
<description>When Carla meets Paul at an ornithology
60+
conference, tempers fly as feathers get ruffled.</description>
61+
</book>
62+
<book id="bk107">
63+
<author>Thurman, Paula</author>
64+
<title>Splish Splash</title>
65+
<genre>Romance</genre>
66+
<price>4.95</price>
67+
<publish_date>2000-11-02</publish_date>
68+
<description>A deep sea diver finds true love twenty
69+
thousand leagues beneath the sea.</description>
70+
</book>
71+
<book id="bk108">
72+
<author>Knorr, Stefan</author>
73+
<title>Creepy Crawlies</title>
74+
<genre>Horror</genre>
75+
<price>4.95</price>
76+
<publish_date>2000-12-06</publish_date>
77+
<description>An anthology of horror stories about roaches,
78+
centipedes, scorpions and other insects.</description>
79+
</book>
80+
<book id="bk109">
81+
<author>Kress, Peter</author>
82+
<title>Paradox Lost</title>
83+
<genre>Science Fiction</genre>
84+
<price>6.95</price>
85+
<publish_date>2000-11-02</publish_date>
86+
<description>After an inadvertant trip through a Heisenberg
87+
Uncertainty Device, James Salway discovers the problems
88+
of being quantum.</description>
89+
</book>
90+
<book id="bk110">
91+
<author>O'Brien, Tim</author>
92+
<title>Microsoft .NET: The Programming Bible</title>
93+
<genre>Computer</genre>
94+
<price>36.95</price>
95+
<publish_date>2000-12-09</publish_date>
96+
<description>Microsoft's .NET initiative is explored in
97+
detail in this deep programmer's reference.</description>
98+
</book>
99+
<book id="bk111">
100+
<author>O'Brien, Tim</author>
101+
<title>MSXML3: A Comprehensive Guide</title>
102+
<genre>Computer</genre>
103+
<price>36.95</price>
104+
<publish_date>2000-12-01</publish_date>
105+
<description>The Microsoft MSXML3 parser is covered in
106+
detail, with attention to XML DOM interfaces, XSLT processing,
107+
SAX and more.</description>
108+
</book>
109+
<book id="bk112">
110+
<author>Galos, Mike</author>
111+
<title>Visual Studio 7: A Comprehensive Guide</title>
112+
<genre>Computer</genre>
113+
<price>49.95</price>
114+
<publish_date>2001-04-16</publish_date>
115+
<description>Microsoft Visual Studio 7 is explored in depth,
116+
looking at how Visual Basic, Visual C++, C#, and ASP+ are
117+
integrated into a comprehensive development
118+
environment.</description>
119+
</book>
120+
</catalog>

0 commit comments

Comments
 (0)