Merge pull request #25 from ScrapeGraphAI/async-change

Async change

Author: VinciGit00

.gitignore

@@ -0,0 +1,5 @@
+.env
+# Ignore .DS_Store files anywhere in the repository
+.DS_Store
+**/.DS_Store
+*.csv

README.md

@@ -3,17 +3,21 @@
 [![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 [![Python SDK](https://img.shields.io/badge/Python_SDK-Latest-blue)](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/scrapegraph-py)
 [![JavaScript SDK](https://img.shields.io/badge/JavaScript_SDK-Latest-yellow)](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/scrapegraph-js)
-[![Documentation](https://img.shields.io/badge/Documentation-Latest-green)](https://scrapegraphai.com/docs)
+[![Documentation](https://img.shields.io/badge/Documentation-Latest-green)](https://docs.scrapegraphai.com)
+
+<p align="left">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png" alt="ScrapeGraph API Banner" style="width: 70%;">
+</p>
 
 Official SDKs for the ScrapeGraph AI API - Intelligent web scraping powered by AI. Extract structured data from any webpage with natural language prompts.
 
-The credits can be bougth [here](https://scrapegraphai.com)!
+Get your [API key](https://scrapegraphai.com)!
 
 ## 🚀 Quick Links
 
 - [Python SDK Documentation](scrapegraph-py/README.md)
 - [JavaScript SDK Documentation](scrapegraph-js/README.md)
-- [API Documentation](https://scrapegraphai.com/docs)
+- [API Documentation](https://docs.scrapegraphai.com) 
 - [Website](https://scrapegraphai.com)
 
 ## 📦 Installation
@@ -69,7 +73,7 @@ Extract information from a local HTML file using AI.
 For detailed documentation and examples, visit:
 - [Python SDK Guide](scrapegraph-py/README.md)
 - [JavaScript SDK Guide](scrapegraph-js/README.md)
-- [API Documentation](https://scrapegraphai.com/docs)
+- [API Documentation](https://docs.scrapegraphai.com) 
 
 ## 💬 Support & Feedback
 

cookbook/chat-webpage-simple-rag/scrapegraph_burr_lancedb.ipynb

@@ -1,9 +1,10 @@
 # 🌐 ScrapeGraph JavaScript SDK
 
-[![npm version](https://badge.fury.io/js/scrapegraph-js.svg)](https://badge.fury.io/js/scrapegraph-js)  
-[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)  
-[![Build Status](https://github.com/ScrapeGraphAI/scrapegraph-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/ScrapeGraphAI/scrapegraph-sdk/actions)  
-[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://docs.scrapegraphai.com)
+[![npm version](https://badge.fury.io/js/scrapegraph-js.svg)](https://badge.fury.io/js/scrapegraph-js) [![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://docs.scrapegraphai.com)
+
+<p align="left">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png" alt="ScrapeGraph API Banner" style="width: 70%;">
+</p>
 
 Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.
 
@@ -246,7 +247,7 @@ Contributions are welcome! Please feel free to submit a Pull Request. For major
 ## 🔗 Links
 
 - [Website](https://scrapegraphai.com)
-- [Documentation](https://scrapegraphai.com/documentation)
+- [Documentation](https://docs.scrapegraphai.com)
 - [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
 
 ## 💬 Support

cookbook/company-info/scrapegraph_langchain.ipynb

@@ -0,0 +1,9 @@
+## 📚 Official Cookbook
+
+Looking for examples and guides? Then head over to the official ScrapeGraph SDK [Cookbook](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/cookbook)!
+
+The cookbook provides step-by-step instructions, practical examples, and tips to help you get started and make the most out of ScrapeGraph SDK.
+
+You will find some colab notebooks with our partners as well, including Langchain 🦜 and LlamaIndex 🦙
+
+Happy scraping! 🚀

cookbook/company-info/scrapegraph_llama_index.ipynb

@@ -4,9 +4,13 @@
 [![Python Support](https://img.shields.io/pypi/pyversions/scrapegraph-py.svg)](https://pypi.org/project/scrapegraph-py/)
 [![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![Documentation Status](https://readthedocs.org/projects/scrapegraph-py/badge/?version=latest)](https://scrapegraph-py.readthedocs.io/en/latest/?badge=latest)
+[![Documentation Status](https://readthedocs.org/projects/scrapegraph-py/badge/?version=latest)](https://docs.scrapegraphai.com) 
 
-Official Python SDK for the ScrapeGraph API - Smart web scraping powered by AI.
+<p align="left">
+  <img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png" alt="ScrapeGraph API Banner" style="width: 70%;">
+</p>
+
+Official [Python SDK ](https://scrapegraphai.com) for the ScrapeGraph API - Smart web scraping powered by AI.
 
 ## 📦 Installation
 
@@ -142,7 +146,7 @@ asyncio.run(main())
 
 ## 📖 Documentation
 
-For detailed documentation, visit [scrapegraphai.com/docs](https://scrapegraphai.com/docs)
+For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegraphai.com)
 
 ## 🛠️ Development
 
@@ -173,7 +177,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 ## 🔗 Links
 
 - [Website](https://scrapegraphai.com)
-- [Documentation](https://scrapegraphai.com/docs)
+- [Documentation](https://docs.scrapegraphai.com) 
 - [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
 
 ---

cookbook/company-info/scrapegraph_sdk.ipynb

@@ -0,0 +1,9 @@
+## 📚 Official Cookbook
+
+Looking for examples and guides? Then head over to the official ScrapeGraph SDK [Cookbook](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/cookbook)!
+
+The cookbook provides step-by-step instructions, practical examples, and tips to help you get started and make the most out of ScrapeGraph SDK.
+
+You will find some colab notebooks with our partners as well, such as Langchain 🦜 and LlamaIndex 🦙
+
+Happy scraping! 🚀

cookbook/github-trending/scrapegraph_langchain.ipynb

@@ -1,6 +1,6 @@
 [project]
 name = "scrapegraph_py"
-version = "0.0.3"
+version = "1.8.0"
 description = "ScrapeGraph Python SDK for API"
 authors = [
     { name = "Marco Vinciguerra", email = "[email protected]" },

cookbook/github-trending/scrapegraph_llama_index.ipynb

@@ -1,7 +1,6 @@
 import asyncio
 from typing import Any, Optional
 
-import aiohttp
 from aiohttp import ClientSession, ClientTimeout, TCPConnector
 from aiohttp.client_exceptions import ClientError
 from pydantic import BaseModel
@@ -27,15 +26,15 @@ class AsyncClient:
     def from_env(
         cls,
         verify_ssl: bool = True,
-        timeout: float = 120,
+        timeout: Optional[float] = None,
         max_retries: int = 3,
         retry_delay: float = 1.0,
     ):
         """Initialize AsyncClient using API key from environment variable.
 
         Args:
             verify_ssl: Whether to verify SSL certificates
-            timeout: Request timeout in seconds
+            timeout: Request timeout in seconds. None means no timeout (infinite)
             max_retries: Maximum number of retry attempts
             retry_delay: Delay between retries in seconds
         """
@@ -56,7 +55,7 @@ def __init__(
         self,
         api_key: str = None,
         verify_ssl: bool = True,
-        timeout: float = 120,
+        timeout: Optional[float] = None,
         max_retries: int = 3,
         retry_delay: float = 1.0,
     ):
@@ -65,7 +64,7 @@ def __init__(
         Args:
             api_key: API key for authentication. If None, will try to load from environment
             verify_ssl: Whether to verify SSL certificates
-            timeout: Request timeout in seconds
+            timeout: Request timeout in seconds. None means no timeout (infinite)
             max_retries: Maximum number of retry attempts
             retry_delay: Delay between retries in seconds
         """
@@ -91,7 +90,7 @@ def __init__(
         self.retry_delay = retry_delay
 
         ssl = None if verify_ssl else False
-        self.timeout = ClientTimeout(total=timeout)
+        self.timeout = ClientTimeout(total=timeout) if timeout is not None else None
 
         self.session = ClientSession(
             headers=self.headers, connector=TCPConnector(ssl=ssl), timeout=self.timeout
@@ -137,6 +136,33 @@ async def _make_request(self, method: str, url: str, **kwargs) -> Any:
                 logger.info(f"⏳ Waiting {retry_delay}s before retry {attempt + 2}")
                 await asyncio.sleep(retry_delay)
 
+    async def markdownify(self, website_url: str):
+        """Send a markdownify request"""
+        logger.info(f"🔍 Starting markdownify request for {website_url}")
+
+        request = MarkdownifyRequest(website_url=website_url)
+        logger.debug("✅ Request validation passed")
+
+        result = await self._make_request(
+            "POST", f"{API_BASE_URL}/markdownify", json=request.model_dump()
+        )
+        logger.info("✨ Markdownify request completed successfully")
+        return result
+
+    async def get_markdownify(self, request_id: str):
+        """Get the result of a previous markdownify request"""
+        logger.info(f"🔍 Fetching markdownify result for request {request_id}")
+
+        # Validate input using Pydantic model
+        GetMarkdownifyRequest(request_id=request_id)
+        logger.debug("✅ Request ID validation passed")
+
+        result = await self._make_request(
+            "GET", f"{API_BASE_URL}/markdownify/{request_id}"
+        )
+        logger.info(f"✨ Successfully retrieved result for request {request_id}")
+        return result
+
     async def smartscraper(
         self,
         website_url: str,
@@ -154,17 +180,11 @@ async def smartscraper(
         )
         logger.debug("✅ Request validation passed")
 
-        try:
-            async with self.session.post(
-                f"{API_BASE_URL}/smartscraper", json=request.model_dump()
-            ) as response:
-                response.raise_for_status()
-                result = await handle_async_response(response)
-                logger.info("✨ Smartscraper request completed successfully")
-                return result
-        except aiohttp.ClientError as e:
-            logger.error(f"❌ Smartscraper request failed: {str(e)}")
-            raise ConnectionError(f"Failed to connect to API: {str(e)}")
+        result = await self._make_request(
+            "POST", f"{API_BASE_URL}/smartscraper", json=request.model_dump()
+        )
+        logger.info("✨ Smartscraper request completed successfully")
+        return result
 
     async def get_smartscraper(self, request_id: str):
         """Get the result of a previous smartscraper request"""
@@ -174,80 +194,8 @@ async def get_smartscraper(self, request_id: str):
         GetSmartScraperRequest(request_id=request_id)
         logger.debug("✅ Request ID validation passed")
 
-        async with self.session.get(
-            f"{API_BASE_URL}/smartscraper/{request_id}"
-        ) as response:
-            result = await handle_async_response(response)
-            logger.info(f"✨ Successfully retrieved result for request {request_id}")
-            return result
-
-    async def get_credits(self):
-        """Get credits information"""
-        logger.info("💳 Fetching credits information")
-
-        async with self.session.get(
-            f"{API_BASE_URL}/credits",
-        ) as response:
-            result = await handle_async_response(response)
-            logger.info(
-                f"✨ Credits info retrieved: {result.get('remaining_credits')} credits remaining"
-            )
-            return result
-
-    async def submit_feedback(
-        self, request_id: str, rating: int, feedback_text: Optional[str] = None
-    ):
-        """Submit feedback for a request"""
-        logger.info(f"📝 Submitting feedback for request {request_id}")
-        logger.debug(f"⭐ Rating: {rating}, Feedback: {feedback_text}")
-
-        feedback = FeedbackRequest(
-            request_id=request_id, rating=rating, feedback_text=feedback_text
-        )
-        logger.debug("✅ Feedback validation passed")
-
-        async with self.session.post(
-            f"{API_BASE_URL}/feedback", json=feedback.model_dump()
-        ) as response:
-            result = await handle_async_response(response)
-            logger.info("✨ Feedback submitted successfully")
-            return result
-
-    async def close(self):
-        """Close the session to free up resources"""
-        logger.info("🔒 Closing AsyncClient session")
-        await self.session.close()
-        logger.debug("✅ Session closed successfully")
-
-    async def __aenter__(self):
-        return self
-
-    async def __aexit__(self, exc_type, exc_val, exc_tb):
-        await self.close()
-
-    async def markdownify(self, website_url: str):
-        """Send a markdownify request"""
-        logger.info(f"🔍 Starting markdownify request for {website_url}")
-
-        request = MarkdownifyRequest(website_url=website_url)
-        logger.debug("✅ Request validation passed")
-
-        result = await self._make_request(
-            "POST", f"{API_BASE_URL}/markdownify", json=request.model_dump()
-        )
-        logger.info("✨ Markdownify request completed successfully")
-        return result
-
-    async def get_markdownify(self, request_id: str):
-        """Get the result of a previous markdownify request"""
-        logger.info(f"🔍 Fetching markdownify result for request {request_id}")
-
-        # Validate input using Pydantic model
-        GetMarkdownifyRequest(request_id=request_id)
-        logger.debug("✅ Request ID validation passed")
-
         result = await self._make_request(
-            "GET", f"{API_BASE_URL}/markdownify/{request_id}"
+            "GET", f"{API_BASE_URL}/smartscraper/{request_id}"
         )
         logger.info(f"✨ Successfully retrieved result for request {request_id}")
         return result
@@ -288,3 +236,46 @@ async def get_localscraper(self, request_id: str):
         )
         logger.info(f"✨ Successfully retrieved result for request {request_id}")
         return result
+
+    async def submit_feedback(
+        self, request_id: str, rating: int, feedback_text: Optional[str] = None
+    ):
+        """Submit feedback for a request"""
+        logger.info(f"📝 Submitting feedback for request {request_id}")
+        logger.debug(f"⭐ Rating: {rating}, Feedback: {feedback_text}")
+
+        feedback = FeedbackRequest(
+            request_id=request_id, rating=rating, feedback_text=feedback_text
+        )
+        logger.debug("✅ Feedback validation passed")
+
+        result = await self._make_request(
+            "POST", f"{API_BASE_URL}/feedback", json=feedback.model_dump()
+        )
+        logger.info("✨ Feedback submitted successfully")
+        return result
+
+    async def get_credits(self):
+        """Get credits information"""
+        logger.info("💳 Fetching credits information")
+
+        result = await self._make_request(
+            "GET",
+            f"{API_BASE_URL}/credits",
+        )
+        logger.info(
+            f"✨ Credits info retrieved: {result.get('remaining_credits')} credits remaining"
+        )
+        return result
+
+    async def close(self):
+        """Close the session to free up resources"""
+        logger.info("🔒 Closing AsyncClient session")
+        await self.session.close()
+        logger.debug("✅ Session closed successfully")
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        await self.close()