Skip to content

Pre/beta #747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 72 commits into from
Oct 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
ac31d7f
ci(release): 1.26.0-beta.1 [skip ci]
semantic-release-bot Sep 29, 2024
26f89d8
feat: refactoring of research web
VinciGit00 Oct 1, 2024
99ad654
Merge branch 'pre/beta' into 713-pdf-scrapping
VinciGit00 Oct 1, 2024
a84176f
Merge pull request #715 from ScrapeGraphAI/713-pdf-scrapping
VinciGit00 Oct 1, 2024
5cedeb8
ci(release): 1.26.0-beta.2 [skip ci]
semantic-release-bot Oct 1, 2024
6664da0
Merge pull request #720 from ScrapeGraphAI/deep_scraper_integration
VinciGit00 Oct 4, 2024
4f65be4
ci(release): 1.26.0-beta.3 [skip ci]
semantic-release-bot Oct 4, 2024
f837dc1
feat: conditional_node
ekinsenler Oct 4, 2024
99aac5b
Merge branch 'pre/beta' into tem
VinciGit00 Oct 5, 2024
8bacd53
Merge pull request #724 from ScrapeGraphAI/tem
VinciGit00 Oct 5, 2024
84d7937
ci(release): 1.26.0-beta.4 [skip ci]
semantic-release-bot Oct 5, 2024
c06cafc
Merge branch 'pre/beta' into cond_node
VinciGit00 Oct 5, 2024
ae5d2ef
Merge pull request #723 from ekinsenler/cond_node
VinciGit00 Oct 5, 2024
ea9ed1a
ci(release): 1.26.0-beta.5 [skip ci]
semantic-release-bot Oct 5, 2024
420c71b
feat: refactoring of the conditional node
VinciGit00 Oct 6, 2024
80ece21
feat: undected_chromedriver support
ekinsenler Oct 6, 2024
d94959c
pass headless state to the browser
ekinsenler Oct 6, 2024
3d1523d
Merge pull request #728 from ScrapeGraphAI/conditional_node_refactoring
VinciGit00 Oct 7, 2024
4cd21f5
ci(release): 1.26.0-beta.6 [skip ci]
semantic-release-bot Oct 7, 2024
37b6ba0
fix: import error
VinciGit00 Oct 7, 2024
482f060
ci(release): 1.26.0-beta.7 [skip ci]
semantic-release-bot Oct 7, 2024
e2d1816
Merge pull request #729 from ekinsenler/selenium
VinciGit00 Oct 7, 2024
eb42c44
refactoring of undected playwrigth
VinciGit00 Oct 7, 2024
95d00e9
Update search_graph_openai.py
VinciGit00 Oct 7, 2024
402e5c4
Merge pull request #730 from ScrapeGraphAI/refactoring-of-pr-#729
VinciGit00 Oct 8, 2024
38b795e
ci(release): 1.26.0-beta.8 [skip ci]
semantic-release-bot Oct 8, 2024
a200f59
refactoring of examples
VinciGit00 Oct 8, 2024
a447aea
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
VinciGit00 Oct 8, 2024
a57da96
fix: removed pdf_scraper graph and created document scraper
VinciGit00 Oct 8, 2024
3b27c5e
fix: pyproject.toml
VinciGit00 Oct 8, 2024
4dc0699
ci(release): 1.26.0-beta.9 [skip ci]
semantic-release-bot Oct 8, 2024
bdafce4
Merge pull request #732 from ScrapeGraphAI/removing-pdf-scraper
VinciGit00 Oct 9, 2024
0c7ebe2
ci(release): 1.26.0-beta.10 [skip ci]
semantic-release-bot Oct 9, 2024
d0e5671
removed unused example
VinciGit00 Oct 9, 2024
7797631
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
VinciGit00 Oct 9, 2024
4f816f3
feat: update chromium loader
VinciGit00 Oct 9, 2024
88ba231
feat:add dynamic rendering
VinciGit00 Oct 9, 2024
7dba46a
add tests
VinciGit00 Oct 9, 2024
c655642
fix: refactoring prompts
VinciGit00 Oct 9, 2024
6d8828a
ci(release): 1.26.0-beta.11 [skip ci]
semantic-release-bot Oct 9, 2024
8753537
fix: nodes prompt
VinciGit00 Oct 9, 2024
7c01d8a
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
VinciGit00 Oct 9, 2024
44d10aa
ci(release): 1.26.0-beta.12 [skip ci]
semantic-release-bot Oct 9, 2024
72f16ab
Merge pull request #736 from ScrapeGraphAI/refractoring-chromiumloader
VinciGit00 Oct 10, 2024
12f2b99
ci(release): 1.26.0-beta.13 [skip ci]
semantic-release-bot Oct 10, 2024
5a2f6d9
feat: prompt refactoring
VinciGit00 Oct 10, 2024
39a029e
feat: refactoring fetch_node
VinciGit00 Oct 10, 2024
e9c8402
Update fetch_node.py
VinciGit00 Oct 10, 2024
f56448a
Merge pull request #740 from ScrapeGraphAI/738-smartscrapergraph-smar…
VinciGit00 Oct 10, 2024
eb25725
ci(release): 1.26.0-beta.14 [skip ci]
semantic-release-bot Oct 10, 2024
91ede93
refactoring of the code
VinciGit00 Oct 10, 2024
46195d6
Update: proxy integration in googlesearch
aziz-ullah-khan Oct 11, 2024
36875b6
Merge remote-tracking branch 'refs/remotes/origin/pre/beta' into pre/…
aziz-ullah-khan Oct 11, 2024
6fd9f14
Merge remote-tracking branch 'origin' into pre/beta
aziz-ullah-khan Oct 11, 2024
cf0cfbd
Merge pull request #737 from ScrapeGraphAI/prompt-refactoring
VinciGit00 Oct 11, 2024
528a974
ci(release): 1.26.0-beta.15 [skip ci]
semantic-release-bot Oct 11, 2024
3b7b701
feat: refactoring of mdscraper
VinciGit00 Oct 11, 2024
e828c70
perf: Proxy integration in googlesearch
aziz-ullah-khan Oct 11, 2024
257f393
feat: async invocation
VinciGit00 Oct 11, 2024
0a275d5
Merge pull request #743 from aziz-ullah-khan/pre/beta
VinciGit00 Oct 11, 2024
a986523
feat: add google proxy support
VinciGit00 Oct 11, 2024
630e640
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
VinciGit00 Oct 11, 2024
e285127
fix: typo
VinciGit00 Oct 11, 2024
04bd2a8
ci(release): 1.26.0-beta.16 [skip ci]
semantic-release-bot Oct 11, 2024
1fb618c
removed unused node
VinciGit00 Oct 11, 2024
6940322
Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegra…
VinciGit00 Oct 11, 2024
7f03ec1
Fix: search_on_web paremter
aziz-ullah-khan Oct 11, 2024
5211cbe
Merge pull request #744 from aziz-ullah-khan/pre/beta
VinciGit00 Oct 11, 2024
47645d9
removed unused files
VinciGit00 Oct 12, 2024
026a70b
fix: bugs
VinciGit00 Oct 12, 2024
e4421ff
Merge branch '731-fetchnode-object-has-no-attribute-update_state' int…
VinciGit00 Oct 12, 2024
f17089c
ci(release): 1.26.0-beta.17 [skip ci]
semantic-release-bot Oct 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
157 changes: 157 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,123 @@
## [1.26.0-beta.17](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.16...v1.26.0-beta.17) (2024-10-12)


### Features

* async invocation ([257f393](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/257f393761e8ff823e37c72659c8b55925c4aecb))
* refactoring of mdscraper ([3b7b701](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3b7b701a89aad503dea771db3f043167f7203d46))


### Bug Fixes

* bugs ([026a70b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/026a70bd3a01b0ebab4d175ae4005e7f3ba3a833))
* search_on_web paremter ([7f03ec1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7f03ec15de20fc2d6c2aad2655cc5348cced1951))

## [1.26.0-beta.16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.15...v1.26.0-beta.16) (2024-10-11)


### Features

* add google proxy support ([a986523](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a9865238847e2edccde579ace7ba226f7012e95d))


### Bug Fixes

* typo ([e285127](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e28512720c3d47917814cf388912aef0e2230188))


### Perf

* Proxy integration in googlesearch ([e828c70](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e828c7010acb1bd04498e027da69f35d53a37890))

## [1.26.0-beta.15](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.14...v1.26.0-beta.15) (2024-10-11)


### Features

* prompt refactoring ([5a2f6d9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5a2f6d9a77a814d5c3756e85cabde8af978f4c06))

## [1.26.0-beta.14](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.13...v1.26.0-beta.14) (2024-10-10)


### Features

* refactoring fetch_node ([39a029e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/39a029ed9a8cd7c2277ba1386b976738e99d231b))

## [1.26.0-beta.13](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.12...v1.26.0-beta.13) (2024-10-10)


### Features

* update chromium loader ([4f816f3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f816f3b04974e90ca4208158f05724cfe68ffb8))

## [1.26.0-beta.12](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.11...v1.26.0-beta.12) (2024-10-09)


### Bug Fixes

* nodes prompt ([8753537](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8753537ecd2a0ba480cda482b6dc50c090b418d6))

## [1.26.0-beta.11](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.10...v1.26.0-beta.11) (2024-10-09)


### Bug Fixes

* refactoring prompts ([c655642](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c65564257798a5ccdc2bdf92487cd9b069e6d951))

## [1.26.0-beta.10](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.9...v1.26.0-beta.10) (2024-10-09)


### Bug Fixes

* removed pdf_scraper graph and created document scraper ([a57da96](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a57da96175a09a16d990eeee679988d10832ce13))

## [1.26.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.8...v1.26.0-beta.9) (2024-10-08)


### Bug Fixes

* pyproject.toml ([3b27c5e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3b27c5e88c0b0744438e8b604f40929e22d722bc))

## [1.26.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.7...v1.26.0-beta.8) (2024-10-08)


### Features

* undected_chromedriver support ([80ece21](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/80ece2179ac47a7ea42fbae4b61504a49ca18daa))

## [1.26.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.6...v1.26.0-beta.7) (2024-10-07)


### Bug Fixes

* import error ([37b6ba0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/37b6ba08ae9972240fc00a15efe43233fd093f3b))

## [1.26.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.5...v1.26.0-beta.6) (2024-10-07)


### Features

* refactoring of the conditional node ([420c71b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/420c71ba2ca0fc77465dd533a807b887c6a87f52))

## [1.26.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.4...v1.26.0-beta.5) (2024-10-05)


### Features

* conditional_node ([f837dc1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f837dc16ce6db0f38fd181822748ca413b7ab4b0))

## [1.26.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.3...v1.26.0-beta.4) (2024-10-05)


### Bug Fixes

* update dependencies ([7579d0e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7579d0e2599d63c0003b1b7a0918132511a9c8f1))


### CI

* **release:** 1.25.2 [skip ci] ([5db4c51](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5db4c518056e9946c00f2fdab612786e0db9ce95))

## [1.25.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.25.1...v1.25.2) (2024-10-03)


Expand All @@ -6,12 +126,49 @@
* update dependencies ([7579d0e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7579d0e2599d63c0003b1b7a0918132511a9c8f1))

## [1.25.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.25.0...v1.25.1) (2024-09-29)
## [1.26.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.2...v1.26.0-beta.3) (2024-10-04)


### Features

* add deep scraper implementation ([4b371f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4b371f4d94dae47986aad751508813d89ce87b93))
* finished basic version of deep scraper ([85cb957](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/85cb9572971719f9f7c66171f5e2246376b6aed2))

## [1.26.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.1...v1.26.0-beta.2) (2024-10-01)


### Features

* refactoring of research web ([26f89d8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/26f89d895d547ef2463492f82da7ac21b57b9d1b))


### CI

* **release:** 1.25.1 [skip ci] ([a98328c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a98328c7f2f39bdd609615247cb71ecf912a3bd8))

## [1.26.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.25.0...v1.26.0-beta.1) (2024-09-29)



* add html_mode to smart_scraper ([bdcffd6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bdcffd6360237b27797546a198ceece55ce4bc81))
* add reasoning integration ([b2822f6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b2822f620a610e61d295cbf4b670aa08fde9de24))



### Bug Fixes

* removed deep scraper ([9aa8c88](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9aa8c889fb32f2eb2005a2fb04f05dc188092279))

* integration with html_mode ([f87ffa1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f87ffa1d8db32b38c47d9f5aa2ae88f1d7978a04))
* removed deep scraper ([9aa8c88](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9aa8c889fb32f2eb2005a2fb04f05dc188092279))


### CI

* **release:** 1.22.0-beta.4 [skip ci] ([4330179](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4330179cb65674d65423c1763f90182e85c15a74))
* **release:** 1.22.0-beta.5 [skip ci] ([6d8f543](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6d8f5435d1ecd2d90b06aade50abc064f75c9d78))
* **release:** 1.22.0-beta.6 [skip ci] ([39f7815](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/39f78154a6f1123fa8aca5e169c803111c175473))

## [1.25.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.24.1...v1.25.0) (2024-09-27)


Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/code_generator_graph_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using Code Generator with schema
"""

import os, json
from typing import List
from dotenv import load_dotenv
Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/csv_scraper_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
"""

import os
from dotenv import load_dotenv
import pandas as pd
Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/csv_scraper_graph_multi_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
"""

import os
from dotenv import load_dotenv
import pandas as pd
Expand Down
2 changes: 0 additions & 2 deletions examples/anthropic/custom_graph_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
"""
Example of custom graph using existing nodes
"""

import os
from dotenv import load_dotenv

from langchain_anthropic import ChatAnthropic
from scrapegraphai.graphs import BaseGraph
from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode, RobotsNode
Expand Down
2 changes: 1 addition & 1 deletion examples/anthropic/json_scraper_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using JSONScraperGraph from JSON documents
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info

load_dotenv()

# ************************************************
Expand Down
39 changes: 0 additions & 39 deletions examples/anthropic/pdf_scraper_graph_anthropic.py

This file was deleted.

71 changes: 0 additions & 71 deletions examples/anthropic/pdf_scraper_multi_anthropic.py

This file was deleted.

1 change: 0 additions & 1 deletion examples/anthropic/rate_limit_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper while setting an API rate limit.
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/scrape_plain_text_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/script_generator_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph
Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/script_multi_generator_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorMultiGraph
Expand Down
2 changes: 1 addition & 1 deletion examples/anthropic/search_graph_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
"""
Example of Search Graph
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info

load_dotenv()

# ************************************************
Expand Down
1 change: 0 additions & 1 deletion examples/anthropic/search_graph_schema_anthropic.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""
Example of Search Graph
"""

import os
from typing import List
from dotenv import load_dotenv
Expand Down
Loading