Merge branch 'main' of https://github.com/ScrapeGraphAI/scrapegraph-sdk

Author: VinciGit00

scrapegraph-js/CODE_OF_CONDUCT.md

@@ -17,23 +17,23 @@ diverse, inclusive, and healthy community.
 Examples of behavior that contributes to a positive environment for our
 community include:
 
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes,
   and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the
+- Focusing on what is best not just for us as individuals, but for the
   overall community
 
 Examples of unacceptable behavior include:
 
-* The use of sexualized language or imagery, and sexual attention or
+- The use of sexualized language or imagery, and sexual attention or
   advances of any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email
   address, without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
+- Other conduct which could reasonably be considered inappropriate in a
   professional setting
 
 ## Enforcement Responsibilities
@@ -106,7 +106,7 @@ Violating these terms may lead to a permanent ban.
 ### 4. Permanent Ban
 
 **Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior,  harassment of an
+standards, including sustained inappropriate behavior, harassment of an
 individual, or aggression toward or disparagement of classes of individuals.
 
 **Consequence**: A permanent ban from any sort of public interaction within

scrapegraph-js/README.md

@@ -9,11 +9,11 @@ Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scrapi
 
 ## 🚀 Features
 
-- ✨ Smart web scraping with AI  
-- 🔄 Fully asynchronous design 
-- 🔍 Detailed error handling  
-- ⚡ Automatic retries and logging  
-- 🔐 Secure API authentication  
+- ✨ Smart web scraping with AI
+- 🔄 Fully asynchronous design
+- 🔍 Detailed error handling
+- ⚡ Automatic retries and logging
+- 🔐 Secure API authentication
 
 ## 📦 Installation
 
@@ -23,11 +23,10 @@ Install the package using npm or yarn:
 # Using npm
 npm i scrapegraph-js
 
-# Using yarn 
+# Using yarn
 yarn add scrapegraph-js
 ```
 
-
 ## 🔧 Quick Start
 
 > **Note**: Store your API keys securely in environment variables. Use `.env` files and libraries like `dotenv` to load them into your app.
@@ -36,7 +35,6 @@ yarn add scrapegraph-js
 
 ```javascript
 import { smartScraper } from 'scrapegraph-js';
-import 'dotenv/config';
 
 // Initialize variables
 const apiKey = process.env.SGAI_APIKEY; // Set your API key as an environment variable
@@ -77,6 +75,7 @@ const prompt = 'Extract the main heading and description.';
 ```
 
 #### Scraping with Custom Output Schema
+
 > [!NOTE]  
 > To use this feature, it is necessary to employ the [Zod](https://www.npmjs.com/package/zod) package for schema creation.
 
@@ -85,7 +84,6 @@ Here is a real-world example:
 ```javascript
 import { smartScraper } from 'scrapegraph-js';
 import { z } from 'zod';
-import 'dotenv/config';
 
 const apiKey = 'your-api-key';
 const url = 'https://scrapegraphai.com/';
@@ -94,7 +92,7 @@ const prompt = 'What does the company do? and ';
 const schema = z.object({
   title: z.string().describe('The title of the webpage'),
   description: z.string().describe('The description of the webpage'),
-  summary: z.string().describe('A brief summary of the webpage')
+  summary: z.string().describe('A brief summary of the webpage'),
 });
 
 (async () => {
@@ -107,6 +105,25 @@ const schema = z.object({
 })();
 ```
 
+### Markdownify
+Converts a webpage into clean, well-structured markdown format.
+```javascript
+import { smartScraper } from 'scrapegraph-js';
+
+const apiKey = "your_api_key";
+const url = 'https://scrapegraphai.com/';
+
+(async () => {
+  try {
+    const response = await markdownify(apiKey, url);
+    console.log(response);
+  } catch (error) {
+    console.error(error);
+  }
+})();
+```
+
+
 ### Checking API Credits
 
 ```javascript
@@ -153,12 +170,14 @@ For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegr
 ### Setup
 
 1. Clone the repository:
+
    ```bash
    git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
    cd scrapegraph-sdk/scrapegraph-js
    ```
 
 2. Install dependencies:
+
    ```bash
    npm install
    ```
@@ -187,22 +206,22 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 
 Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
 
-1. Fork the repository  
-2. Create your feature branch (`git checkout -b feature/AmazingFeature`)  
-3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)  
-4. Push to the branch (`git push origin feature/AmazingFeature`)  
-5. Open a Pull Request  
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
 
 ## 🔗 Links
 
-- [Website](https://scrapegraphai.com)  
-- [Documentation](https://scrapegraphai.com/documentation)  
-- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)  
+- [Website](https://scrapegraphai.com)
+- [Documentation](https://scrapegraphai.com/documentation)
+- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
 
 ## 💬 Support
 
-- 📧 Email: [email protected]  
-- 💻 GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)  
+- 📧 Email: [email protected]
+- 💻 GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)
 - 🌟 Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new)
 
 ---

scrapegraph-js/eslint.config.js

@@ -7,5 +7,5 @@ export default [
   { languageOptions: { globals: { ...globals.browser, ...globals.node } } },
   pluginJs.configs.recommended,
   eslintPluginPrettierRecommended,
-  { ignorePatterns: ['node_modules/'] },
+  { ignores: ['node_modules/'] },
 ];

scrapegraph-js/examples/getCredits_example.js

@@ -4,8 +4,8 @@ import 'dotenv/config';
 const apiKey = process.env.SGAI_APIKEY;
 
 try {
-	const myCredit = await getCredits(apiKey);
-	console.log(myCredit)
+  const myCredit = await getCredits(apiKey);
+  console.log(myCredit);
 } catch (error) {
-	console.error(error)
-}
+  console.error(error);
+}

scrapegraph-js/examples/getSmartScraperRequest_example.js

@@ -2,11 +2,11 @@ import { getSmartScraperRequest } from 'scrapegraph-js';
 import 'dotenv/config';
 
 const apiKey = process.env.SGAI_APIKEY;
-const requestId = '3fa85f64-5717-4562-b3fc-2c963f66afa6'
+const requestId = '3fa85f64-5717-4562-b3fc-2c963f66afa6';
 
 try {
   const requestInfo = await getSmartScraperRequest(apiKey, requestId);
   console.log(requestInfo);
 } catch (error) {
   console.error(error);
-}
+}

scrapegraph-js/examples/markdownify_example.js

@@ -0,0 +1,35 @@
+import { getMarkdownifyRequest, markdownify } from 'scrapegraph-js';
+import fs from 'fs';
+import 'dotenv/config';
+
+// markdownify function example
+const apiKey = process.env.SGAI_APIKEY;
+const url = 'https://scrapegraphai.com/';
+
+try {
+  const response = await markdownify(apiKey, url);
+  console.log(response);
+  saveFile(response.result);
+} catch (error) {
+  console.error(error);
+}
+
+// helper function for save the file locally
+function saveFile(output) {
+  try {
+    fs.writeFileSync('result.md', output);
+    console.log('Success!');
+  } catch (err) {
+    console.error('Error during the file writing: ', err);
+  }
+}
+
+// getMarkdownifyRequest function example
+const requestId = '2563b972-cb6f-400b-be76-edb235458560';
+
+try {
+  const response = await getMarkdownifyRequest(apiKey, requestId);
+  console.log(response);
+} catch (error) {
+  console.log(error);
+}

scrapegraph-js/examples/schema_smartScraper_example.js

@@ -6,11 +6,15 @@ const apiKey = process.env.SGAI_APIKEY;
 const url = 'https://scrapegraphai.com/';
 const prompt = 'What does the company do? and ';
 
-const schema = 2;
+const schema = z.object({
+  title: z.string().describe('The title of the webpage'),
+  description: z.string().describe('The description of the webpage'),
+  summary: z.string().describe('A brief summary of the webpage'),
+});
 
 try {
   const response = await smartScraper(apiKey, url, prompt, schema);
   console.log(response.result);
 } catch (error) {
   console.error(error);
-}
+}

scrapegraph-js/examples/sendFeedback_example.js

@@ -10,5 +10,5 @@ try {
   const feedback_response = await sendFeedback(apiKey, requestId, rating, feedbackMessage);
   console.log(feedback_response);
 } catch (error) {
-  console.error(error)
-}
+  console.error(error);
+}

scrapegraph-js/index.js

@@ -1,3 +1,4 @@
 export { smartScraper, getSmartScraperRequest } from './src/smartScraper.js';
+export { markdownify, getMarkdownifyRequest } from './src/markdownify.js';
 export { getCredits } from './src/credits.js';
-export { sendFeedback } from './src/feedback.js'; 
+export { sendFeedback } from './src/feedback.js';

scrapegraph-js/src/credits.js

@@ -3,21 +3,21 @@ import handleError from './utils/handleError.js';
 
 /**
  * Retrieve credits from the API.
- * 
+ *
  * @param {string} apiKey - Your ScrapeGraph AI API key
  * @returns {Promise<string>} Response from the API in JSON format
  */
 export async function getCredits(apiKey) {
   const endpoint = 'https://api.scrapegraphai.com/v1/credits';
   const headers = {
     'accept': 'application/json',
-    'SGAI-APIKEY': apiKey
+    'SGAI-APIKEY': apiKey,
   };
 
   try {
     const response = await axios.get(endpoint, { headers });
     return response.data;
   } catch (error) {
-    handleError(error)
+    handleError(error);
   }
-} 
+}

scrapegraph-js/src/feedback.js

@@ -3,7 +3,7 @@ import handleError from './utils/handleError.js';
 
 /**
  * Send feedback to the API.
- * 
+ *
  * @param {string} apiKey - Your ScrapeGraph AI API key
  * @param {string} requestId - The request ID associated with the feedback
  * @param {number} rating - The rating score
@@ -15,13 +15,13 @@ export async function sendFeedback(apiKey, requestId, rating, feedbackText = nul
   const headers = {
     'accept': 'application/json',
     'SGAI-APIKEY': apiKey,
-    'Content-Type': 'application/json'
+    'Content-Type': 'application/json',
   };
 
   const feedbackData = {
     request_id: requestId,
     rating: rating,
-    feedback_text: feedbackText
+    feedback_text: feedbackText,
   };
 
   try {
@@ -30,4 +30,4 @@ export async function sendFeedback(apiKey, requestId, rating, feedbackText = nul
   } catch (error) {
     handleError(error);
   }
-} 
+}

scrapegraph-js/src/markdownify.js

@@ -0,0 +1,52 @@
+import axios from 'axios';
+import handleError from './utils/handleError.js';
+
+/**
+ * Converts a webpage into clean, well-structured markdown format.
+ *
+ * @param {string} apiKey - Your ScrapeGraph AI API key.
+ * @param {string} url - The URL of the webpage to be converted.
+ * @returns {Promise<string>} A promise that resolves to the markdown representation of the webpage.
+ * @throws {Error} Throws an error if the HTTP request fails.
+ */
+export async function markdownify(apiKey, url){
+  const endpoint = 'https://api.scrapegraphai.com/v1/markdownify';
+  const headers = {
+    'accept': 'application/json',
+    'SGAI-APIKEY': apiKey,
+  };
+
+  const payload = {
+    website_url: url,
+  };
+
+  try {
+    const response = await axios.post(endpoint, payload, { headers });
+    return response.data;
+  } catch (error) {
+    handleError(error)
+  }
+}
+
+/**
+ * Retrieves the status or result of a markdownify request, with the option to review results from previous requests.
+ *
+ * @param {string} apiKey - Your ScrapeGraph AI API key.
+ * @param {string} requestId - The unique identifier for the markdownify request whose result you want to retrieve.
+ * @returns {Promise<string>} A promise that resolves with details about the status or outcome of the specified request.
+ * @throws {Error} Throws an error if the HTTP request fails.
+ */
+export async function getMarkdownifyRequest(apiKey, requestId){
+  const endpoint = 'https://api.scrapegraphai.com/v1/markdownify/' + requestId;
+  const headers = {
+    'accept': 'application/json',
+    'SGAI-APIKEY': apiKey,
+  };
+
+  try {
+    const response = await axios.get(endpoint, { headers });
+    return response.data;
+  } catch (error) {
+    handleError(error)
+  }
+}