Merge branch 'md_scraper_integration' into integration_markdown

Author: VinciGit00

CHANGELOG.md

@@ -1,18 +1,162 @@
-## [1.7.0-beta.14](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.13...v1.7.0-beta.14) (2024-06-19)
+## [1.8.0-beta.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.4...v1.8.0-beta.1) (2024-06-25)
 
 
 ### Features
 
 * add new search engine avaiability and new tests ([073d226](https://github.com/VinciGit00/Scrapegraph-ai/commit/073d226723f5f03b960865d07408905b7a506180))
 * add research with bing + test function ([aa2160c](https://github.com/VinciGit00/Scrapegraph-ai/commit/aa2160c108764745a696ffc16038f370e9702c14))
 
-## [1.7.0-beta.13](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.12...v1.7.0-beta.13) (2024-06-18)
-
 
 ### Bug Fixes
 
 * updated for schema changes ([aedda44](https://github.com/VinciGit00/Scrapegraph-ai/commit/aedda448682ce5a921a62e661bffb02478bab75f))
 
+
+### CI
+
+* **release:** 1.7.0-beta.13 [skip ci] ([ce0a47a](https://github.com/VinciGit00/Scrapegraph-ai/commit/ce0a47aee5edbb26fd82e41f6688a4bc48a10822))
+* **release:** 1.7.0-beta.14 [skip ci] ([ec77ff7](https://github.com/VinciGit00/Scrapegraph-ai/commit/ec77ff7ea4eb071469c2fb53e5959d4ea1f73ad6))
+
+## [1.7.4](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.3...v1.7.4) (2024-06-21)
+
+
+### Bug Fixes
+
+* add new model for claude ([599512d](https://github.com/VinciGit00/Scrapegraph-ai/commit/599512d2e561540396ca3b6762acd5b8ed3c3e59))
+
+## [1.7.3](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.2...v1.7.3) (2024-06-19)
+
+
+### Bug Fixes
+
+* reduced model tokens ([88f9def](https://github.com/VinciGit00/Scrapegraph-ai/commit/88f9def69d80c2f5b1a81878fcd0e385b25ed65f))
+
+
+### Docs
+
+* **version:** fixed compatible versions ([ecb7601](https://github.com/VinciGit00/Scrapegraph-ai/commit/ecb7601be79137f4c520614c53d52aa07bb18f6a))
+
+## [1.7.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.1...v1.7.2) (2024-06-18)
+
+
+### Bug Fixes
+
+* total tokens and docs ([c787090](https://github.com/VinciGit00/Scrapegraph-ai/commit/c7870905e10da85b81761ab2c3f71220bafe9f22))
+
+
+### Docs
+
+* fixed readme по русский ([2373073](https://github.com/VinciGit00/Scrapegraph-ai/commit/23730735bac7e87ddaf6cdbc1edd1598a315413b))
+
+## [1.7.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0...v1.7.1) (2024-06-18)
+
+
+### Bug Fixes
+
+* add new embedding models ([1d0cbbc](https://github.com/VinciGit00/Scrapegraph-ai/commit/1d0cbbc6d6e8c50299bb38b3bfa5e241488ff6f4))
+
+## [1.7.0](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.1...v1.7.0) (2024-06-17)
+
+
+
+### Features
+
+* add caching ([d790361](https://github.com/VinciGit00/Scrapegraph-ai/commit/d79036149a3197a385b73553f29df66d36480c38))
+* add csv scraper and xml scraper multi ([b408655](https://github.com/VinciGit00/Scrapegraph-ai/commit/b4086550cc9dc42b2fd91ee7ef60c6a2c2ac3fd2))
+* add dynamic caching ([7ed2fe8](https://github.com/VinciGit00/Scrapegraph-ai/commit/7ed2fe8ef0d16fd93cb2ff88840bcaa643349e33))
+* **indexify-node:** add example ([5d1fbf8](https://github.com/VinciGit00/Scrapegraph-ai/commit/5d1fbf806a20746931ebb7fcb32c383d9d549d93))
+* add forcing format as json ([5cfc101](https://github.com/VinciGit00/Scrapegraph-ai/commit/5cfc10178abf0b7a3e0b2229512396e243305438))
+* add json as output ([5d20186](https://github.com/VinciGit00/Scrapegraph-ai/commit/5d20186bf20fb2384f2a9e7e81c2e875ff50a4f3))
+* add json multiscraper ([5bda918](https://github.com/VinciGit00/Scrapegraph-ai/commit/5bda918a39e4b50d86d784b4c592cc2ea1a68986))
+* add new chunking function ([e1f045b](https://github.com/VinciGit00/Scrapegraph-ai/commit/e1f045b2809fc7db0c252f4c6f2f9a435c66ba91))
+* add Parse_Node ([e6c7940](https://github.com/VinciGit00/Scrapegraph-ai/commit/e6c7940a57929c2ed8c9fda1a6e375cc87a2b7f4))
+* add pdf scraper multi graph ([f5cbd80](https://github.com/VinciGit00/Scrapegraph-ai/commit/f5cbd80c977f51233ac1978d8450fcf0ec2ff461))
+* **merge:** add scriptcreatormulti, rag cache and semchunk ([15421ef](https://github.com/VinciGit00/Scrapegraph-ai/commit/15421eff7009b80293f7d84df5086d22944dfb99))
+* **telemetry:** add telemetry module ([080a318](https://github.com/VinciGit00/Scrapegraph-ai/commit/080a318ff68652a3c81a6890cd40fd20c48ac6d0))
+* Add tests for RobotsNode and update test setup ([b0511ae](https://github.com/VinciGit00/Scrapegraph-ai/commit/b0511aeaaac55570c8dad25b7cac7237bd20ef4c))
+* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([c927145](https://github.com/VinciGit00/Scrapegraph-ai/commit/c927145bd06693d0fad02b2285b426276b7d61a8))
+* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([9e7038c](https://github.com/VinciGit00/Scrapegraph-ai/commit/9e7038c5962563f53e0d44943d5c604cb1a2b035))
+* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([c286b16](https://github.com/VinciGit00/Scrapegraph-ai/commit/c286b1649e75d6c655698f38d695b58e3efa6270))
+* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([08f1be6](https://github.com/VinciGit00/Scrapegraph-ai/commit/08f1be682b0509f1e06148269fec1fa2897c394e))
+* **pydantic:** added pydantic output schema ([376f758](https://github.com/VinciGit00/Scrapegraph-ai/commit/376f758a76e3e111dc34416dedf8e294dc190963))
+* **append_node:** append node to existing graph ([f8b08e0](https://github.com/VinciGit00/Scrapegraph-ai/commit/f8b08e0b33ca31124c2773f47a624eeb0a4f302f))
+* fix an if ([c8d556d](https://github.com/VinciGit00/Scrapegraph-ai/commit/c8d556da4e4b8730c6c35f1d448270b8e26923f2))
+* **schema:** merge scripts to follow pydantic schema ([5d692bf](https://github.com/VinciGit00/Scrapegraph-ai/commit/5d692bff9e4f124146dd37e573f7c3c0aa8d9a23))
+* refactoring of abstract graph ([fff89f4](https://github.com/VinciGit00/Scrapegraph-ai/commit/fff89f431f60b5caa4dd87643a1bb8895bf96d48))
+* refactoring of an in if ([244aada](https://github.com/VinciGit00/Scrapegraph-ai/commit/244aada2de1f3bc88782fa90e604e8b936b79aa4))
+* refactoring of rag node ([7a13a68](https://github.com/VinciGit00/Scrapegraph-ai/commit/7a13a6819ff35a6f6197ee837d0eb8ea65e31776))
+* removed a bug ([8de720d](https://github.com/VinciGit00/Scrapegraph-ai/commit/8de720d37958e31b73c5c89bc21f474f3303b42b))
+* removed rag node ([930f673](https://github.com/VinciGit00/Scrapegraph-ai/commit/930f67374752561903462a25728c739946f9449b))
+* **version:** update burr version ([cfa1336](https://github.com/VinciGit00/Scrapegraph-ai/commit/cfa13368f4d5c7dd8be27aabe19c7602d24686da))
+* update fetch node ([1e7f334](https://github.com/VinciGit00/Scrapegraph-ai/commit/1e7f3349f3192ca1b9c54b110619171c5248816c))
+
+
+### Bug Fixes
+
+* add chinese embedding model ([03ffebc](https://github.com/VinciGit00/Scrapegraph-ai/commit/03ffebc52de3fc6f80a968880e8ade3e3cdf95ec))
+* common params ([6b4cdf9](https://github.com/VinciGit00/Scrapegraph-ai/commit/6b4cdf92b82fa143e4217a2e5da46d04f2585de8))
+* **cache:** correctly pass the node arguments and logging ([c881f64](https://github.com/VinciGit00/Scrapegraph-ai/commit/c881f64209a86a69ddd3105f5d0360d9ed183490))
+* **pdf:** correctly read .pdf files ([203de83](https://github.com/VinciGit00/Scrapegraph-ai/commit/203de834051ea1d6443841921f3aa3e6adbd9174))
+* fix robot node ([2419003](https://github.com/VinciGit00/Scrapegraph-ai/commit/24190039996b9cbe04952f6734d996e0cdb15296))
+* **node:** fixed generate answer node pydantic schema ([ab00f23](https://github.com/VinciGit00/Scrapegraph-ai/commit/ab00f23d859c64995ccfe329b24379cf3c14d73c))
+* **schema:** fixed json output ([5c9843f](https://github.com/VinciGit00/Scrapegraph-ai/commit/5c9843f1410a78568892635e53872793d5ba0d6f))
+* oneapi model ([4fcb990](https://github.com/VinciGit00/Scrapegraph-ai/commit/4fcb9902fe4c147c61a1622a919ade338c03b8d8))
+* shallow copy config of create_embedder ([62b372b](https://github.com/VinciGit00/Scrapegraph-ai/commit/62b372b675a45ca4d031f337b6f8728151689442))
+* test for fetch node ([49c7e0e](https://github.com/VinciGit00/Scrapegraph-ai/commit/49c7e0eaab6fc7a9242054b7d3f375369af9bcdc))
+* typo in prompt ([4639f0c](https://github.com/VinciGit00/Scrapegraph-ai/commit/4639f0cac5029c6802a6caded7103d247f4f06dd))
+* **multi:** updated multi pdf scraper with schema ([91c5b5a](https://github.com/VinciGit00/Scrapegraph-ai/commit/91c5b5af43134671f4d5c801ee315f935b4fed4f))
+
+
+### Docs
+
+* **cache:** added cache_path param ([edddb68](https://github.com/VinciGit00/Scrapegraph-ai/commit/edddb682d06262088885e340b7b73cc70adf9583))
+* better logging ([283b61f](https://github.com/VinciGit00/Scrapegraph-ai/commit/283b61fafcc805e7f866e1acf68ffd6581ace1a9))
+* **scriptcreator:** enhance documentation ([650c3aa](https://github.com/VinciGit00/Scrapegraph-ai/commit/650c3aaa60dab169358c2c04bfca9dee8d1a5d68))
+* fix label&logo for github action badges ([071f3d1](https://github.com/VinciGit00/Scrapegraph-ai/commit/071f3d19066eee6deb62a671132acf8a5b8ac927))
+* refactor graph section and added telemetry ([39bf4c9](https://github.com/VinciGit00/Scrapegraph-ai/commit/39bf4c960d703a321af64e3b1b41ca9a1a15794e))
+* stylize badges in readme ([8696ade](https://github.com/VinciGit00/Scrapegraph-ai/commit/8696adede79cf9557c49a8b30a095b76ec3d02f6))
+
+
+### Refactor
+
+* add missing schemas and renamed files ([09cb6e9](https://github.com/VinciGit00/Scrapegraph-ai/commit/09cb6e964eaa41587237c622a1ea8894722d87cb))
+
+
+### Test
+
+* fix tests for fetch node with proper mock&refactor ([17dd936](https://github.com/VinciGit00/Scrapegraph-ai/commit/17dd936af7cfd1d0822202d908e50ab11893bddd))
+
+
+### CI
+
+* **release:** 1.5.3-beta.1 [skip ci] ([6ea1d2c](https://github.com/VinciGit00/Scrapegraph-ai/commit/6ea1d2c4d0aaf7a341a2ea6ea7070438a7610fe4))
+* **release:** 1.5.3-beta.2 [skip ci] ([b57bcef](https://github.com/VinciGit00/Scrapegraph-ai/commit/b57bcef5c18530ce03ff6ec65e9e33d00d9f6515))
+* **release:** 1.5.5-beta.1 [skip ci] ([38d138e](https://github.com/VinciGit00/Scrapegraph-ai/commit/38d138e36faa718632b7560fab197c25e24da9de))
+* **release:** 1.6.0-beta.1 [skip ci] ([1d217e4](https://github.com/VinciGit00/Scrapegraph-ai/commit/1d217e4ae682ddf16d911b6db6973dc05445660c))
+* **release:** 1.6.0-beta.10 [skip ci] ([4d0d8fa](https://github.com/VinciGit00/Scrapegraph-ai/commit/4d0d8fa453f411927f49d75b9f67fb08ab168759))
+* **release:** 1.6.0-beta.11 [skip ci] ([3453ac0](https://github.com/VinciGit00/Scrapegraph-ai/commit/3453ac01f5da9148c8d10f29724b4a1c20d0a6e8))
+* **release:** 1.6.0-beta.2 [skip ci] ([ed1dc0b](https://github.com/VinciGit00/Scrapegraph-ai/commit/ed1dc0be08faf7e050f627c175897ae9c0eccbcf))
+* **release:** 1.6.0-beta.3 [skip ci] ([b70cb37](https://github.com/VinciGit00/Scrapegraph-ai/commit/b70cb37c623d56f5508650937bc314724ceec0e9))
+* **release:** 1.6.0-beta.4 [skip ci] ([08a14ef](https://github.com/VinciGit00/Scrapegraph-ai/commit/08a14efdd334ae645cb5cfe0dec04332659b99d5))
+* **release:** 1.6.0-beta.5 [skip ci] ([dde0c7e](https://github.com/VinciGit00/Scrapegraph-ai/commit/dde0c7e27deb55a0005691d402406a13ee507420))
+* **release:** 1.6.0-beta.6 [skip ci] ([ac8e7c1](https://github.com/VinciGit00/Scrapegraph-ai/commit/ac8e7c12fe677a357b8b1b8d42a1aca8503de727))
+* **release:** 1.6.0-beta.7 [skip ci] ([cab5f68](https://github.com/VinciGit00/Scrapegraph-ai/commit/cab5f6828cac926a82d9ecfe7a97596aaabfa385))
+* **release:** 1.6.0-beta.8 [skip ci] ([7a6f016](https://github.com/VinciGit00/Scrapegraph-ai/commit/7a6f016f9231f92e1bb99059e08b431ce99b14cf))
+* **release:** 1.6.0-beta.9 [skip ci] ([ca8aff8](https://github.com/VinciGit00/Scrapegraph-ai/commit/ca8aff8d8849552159ff1b86fd175fa5e9fe7c1f))
+* **release:** 1.7.0-beta.1 [skip ci] ([84a74b2](https://github.com/VinciGit00/Scrapegraph-ai/commit/84a74b2f79a3f53e7112b6c7054c5764842bafd1))
+* **release:** 1.7.0-beta.10 [skip ci] ([7f3b907](https://github.com/VinciGit00/Scrapegraph-ai/commit/7f3b90741055cea074be12b4bd0fe68d4e2e01d8))
+* **release:** 1.7.0-beta.11 [skip ci] ([c016efd](https://github.com/VinciGit00/Scrapegraph-ai/commit/c016efd021b58930ca8f08881b0bb1d00064768c))
+* **release:** 1.7.0-beta.12 [skip ci] ([a794405](https://github.com/VinciGit00/Scrapegraph-ai/commit/a794405471f6cae4de161f2327e11f2883a4ed08))
+* **release:** 1.7.0-beta.2 [skip ci] ([e5bb5ae](https://github.com/VinciGit00/Scrapegraph-ai/commit/e5bb5ae473f1b5f68741126559d5033191f31c72))
+* **release:** 1.7.0-beta.3 [skip ci] ([85a75c8](https://github.com/VinciGit00/Scrapegraph-ai/commit/85a75c893a6b9b5d07f8f561f65bb562007c0a3e))
+* **release:** 1.7.0-beta.4 [skip ci] ([b4d7532](https://github.com/VinciGit00/Scrapegraph-ai/commit/b4d7532c6ce8e989403b94651af4b77738ab674d))
+* **release:** 1.7.0-beta.5 [skip ci] ([79b8326](https://github.com/VinciGit00/Scrapegraph-ai/commit/79b8326b5becce7ee22ff7323c00457f6dff7519))
+* **release:** 1.7.0-beta.6 [skip ci] ([dae3158](https://github.com/VinciGit00/Scrapegraph-ai/commit/dae3158519666af1747e5e9bc1263d6d4235997d))
+* **release:** 1.7.0-beta.7 [skip ci] ([7da6cd2](https://github.com/VinciGit00/Scrapegraph-ai/commit/7da6cd2ab2c3581599cd7516aaa56e2c2664f100))
+* **release:** 1.7.0-beta.8 [skip ci] ([a87702f](https://github.com/VinciGit00/Scrapegraph-ai/commit/a87702f107f3fd16ee73e1af1585cd763788bf46))
+* **release:** 1.7.0-beta.9 [skip ci] ([0c5d6e2](https://github.com/VinciGit00/Scrapegraph-ai/commit/0c5d6e2c82b9ee81c91cd2325948bb5a4eddcb31))
+
+
 ## [1.7.0-beta.12](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.11...v1.7.0-beta.12) (2024-06-17)
 
 

README.md

@@ -2,7 +2,7 @@
 # 🕷️ ScrapeGraphAI: You Only Scrape Once
 [English](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/README.md) | [中文](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/chinese.md) | [日本語](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/japanese.md)
 | [코리아노](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/korean.md)
-| [русский](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/russian.md)
+| [Русский](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/russian.md)
 
 
 [![Downloads](https://img.shields.io/pepy/dt/scrapegraphai?style=for-the-badge)](https://pepy.tech/project/scrapegraphai)

docker-compose.yml

@@ -4,7 +4,7 @@ services:
     image: ollama/ollama
     container_name: ollama
     ports:
-      - "11434:11434"
+      - "5000:5000"
     volumes:
       - ollama_volume:/root/.ollama
     restart: unless-stopped