Skip to content

fix: refactoring of merge_answer_node #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 7, 2024
Merged

fix: refactoring of merge_answer_node #515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
898e5a7
fix: refactoring of merge_answer_node
vedovati-matteo Aug 7, 2024
108cdd3
Merge branch 'pre/beta' of https://github.com/vedovati-matteo/Scrapeg…
vedovati-matteo Aug 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions scrapegraphai/helpers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@
from .generate_answer_node_csv_prompts import template_chunks_csv, template_no_chunks_csv, template_merge_csv
from .generate_answer_node_pdf_prompts import template_chunks_pdf, template_no_chunks_pdf, template_merge_pdf
from .generate_answer_node_omni_prompts import template_chunks_omni, template_no_chunk_omni, template_merge_omni
from .merge_answer_node_prompts import template_combined
13 changes: 13 additions & 0 deletions scrapegraphai/helpers/merge_answer_node_prompts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
"""
Merge answer node prompts
"""

template_combined = """
You are a website scraper and you have just scraped some content from multiple websites.\n
You are now asked to provide an answer to a USER PROMPT based on the content you have scraped.\n
You need to merge the content from the different websites into a single answer without repetitions (if there are any). \n
The scraped contents are in a JSON format and you need to merge them based on the context and providing a correct JSON structure.\n
OUTPUT INSTRUCTIONS: {format_instructions}\n
USER PROMPT: {user_prompt}\n
WEBSITE CONTENT: {website_content}
"""
13 changes: 2 additions & 11 deletions scrapegraphai/nodes/merge_answers_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from langchain_core.output_parsers import JsonOutputParser
from ..utils.logging import get_logger
from .base_node import BaseNode
from ..helpers import template_combined


class MergeAnswersNode(BaseNode):
Expand Down Expand Up @@ -79,18 +80,8 @@ def execute(self, state: dict) -> dict:

format_instructions = output_parser.get_format_instructions()

template_merge = """
You are a website scraper and you have just scraped some content from multiple websites.\n
You are now asked to provide an answer to a USER PROMPT based on the content you have scraped.\n
You need to merge the content from the different websites into a single answer without repetitions (if there are any). \n
The scraped contents are in a JSON format and you need to merge them based on the context and providing a correct JSON structure.\n
OUTPUT INSTRUCTIONS: {format_instructions}\n
USER PROMPT: {user_prompt}\n
WEBSITE CONTENT: {website_content}
"""

prompt_template = PromptTemplate(
template=template_merge,
template=template_combined,
input_variables=["user_prompt"],
partial_variables={
"format_instructions": format_instructions,
Expand Down