Skip to content

Fixed some bugs and improved code style #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions scrapegraph-js/CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,23 @@ diverse, inclusive, and healthy community.
Examples of behavior that contributes to a positive environment for our
community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
- Focusing on what is best not just for us as individuals, but for the
overall community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or
- The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
- Other conduct which could reasonably be considered inappropriate in a
professional setting

## Enforcement Responsibilities
Expand Down Expand Up @@ -106,7 +106,7 @@ Violating these terms may lead to a permanent ban.
### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within
Expand Down
38 changes: 20 additions & 18 deletions scrapegraph-js/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scrapi

## 🚀 Features

- ✨ Smart web scraping with AI
- 🔄 Fully asynchronous design
- 🔍 Detailed error handling
- ⚡ Automatic retries and logging
- 🔐 Secure API authentication
- ✨ Smart web scraping with AI
- 🔄 Fully asynchronous design
- 🔍 Detailed error handling
- ⚡ Automatic retries and logging
- 🔐 Secure API authentication

## 📦 Installation

Expand All @@ -23,11 +23,10 @@ Install the package using npm or yarn:
# Using npm
npm i scrapegraph-js

# Using yarn
# Using yarn
yarn add scrapegraph-js
```


## 🔧 Quick Start

> **Note**: Store your API keys securely in environment variables. Use `.env` files and libraries like `dotenv` to load them into your app.
Expand Down Expand Up @@ -77,6 +76,7 @@ const prompt = 'Extract the main heading and description.';
```

#### Scraping with Custom Output Schema

> [!NOTE]
> To use this feature, it is necessary to employ the [Zod](https://www.npmjs.com/package/zod) package for schema creation.

Expand All @@ -94,7 +94,7 @@ const prompt = 'What does the company do? and ';
const schema = z.object({
title: z.string().describe('The title of the webpage'),
description: z.string().describe('The description of the webpage'),
summary: z.string().describe('A brief summary of the webpage')
summary: z.string().describe('A brief summary of the webpage'),
});

(async () => {
Expand Down Expand Up @@ -153,12 +153,14 @@ For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegr
### Setup

1. Clone the repository:

```bash
git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-js
```

2. Install dependencies:

```bash
npm install
```
Expand Down Expand Up @@ -187,22 +189,22 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## 🔗 Links

- [Website](https://scrapegraphai.com)
- [Documentation](https://scrapegraphai.com/documentation)
- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
- [Website](https://scrapegraphai.com)
- [Documentation](https://scrapegraphai.com/documentation)
- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)

## 💬 Support

- 📧 Email: [email protected]
- 💻 GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)
- 📧 Email: [email protected]
- 💻 GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)
- 🌟 Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new)

---
Expand Down
2 changes: 1 addition & 1 deletion scrapegraph-js/eslint.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@ export default [
{ languageOptions: { globals: { ...globals.browser, ...globals.node } } },
pluginJs.configs.recommended,
eslintPluginPrettierRecommended,
{ ignorePatterns: ['node_modules/'] },
{ ignores: ['node_modules/'] },
];
8 changes: 4 additions & 4 deletions scrapegraph-js/examples/getCredits_example.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ import 'dotenv/config';
const apiKey = process.env.SGAI_APIKEY;

try {
const myCredit = await getCredits(apiKey);
console.log(myCredit)
const myCredit = await getCredits(apiKey);
console.log(myCredit);
} catch (error) {
console.error(error)
}
console.error(error);
}
4 changes: 2 additions & 2 deletions scrapegraph-js/examples/getSmartScraperRequest_example.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ import { getSmartScraperRequest } from 'scrapegraph-js';
import 'dotenv/config';

const apiKey = process.env.SGAI_APIKEY;
const requestId = '3fa85f64-5717-4562-b3fc-2c963f66afa6'
const requestId = '3fa85f64-5717-4562-b3fc-2c963f66afa6';

try {
const requestInfo = await getSmartScraperRequest(apiKey, requestId);
console.log(requestInfo);
} catch (error) {
console.error(error);
}
}
8 changes: 6 additions & 2 deletions scrapegraph-js/examples/schema_smartScraper_example.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,15 @@ const apiKey = process.env.SGAI_APIKEY;
const url = 'https://scrapegraphai.com/';
const prompt = 'What does the company do? and ';

const schema = 2;
const schema = z.object({
title: z.string().describe('The title of the webpage'),
description: z.string().describe('The description of the webpage'),
summary: z.string().describe('A brief summary of the webpage'),
});

try {
const response = await smartScraper(apiKey, url, prompt, schema);
console.log(response.result);
} catch (error) {
console.error(error);
}
}
4 changes: 2 additions & 2 deletions scrapegraph-js/examples/sendFeedback_example.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ try {
const feedback_response = await sendFeedback(apiKey, requestId, rating, feedbackMessage);
console.log(feedback_response);
} catch (error) {
console.error(error)
}
console.error(error);
}
2 changes: 1 addition & 1 deletion scrapegraph-js/index.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
export { smartScraper, getSmartScraperRequest } from './src/smartScraper.js';
export { getCredits } from './src/credits.js';
export { sendFeedback } from './src/feedback.js';
export { sendFeedback } from './src/feedback.js';
8 changes: 4 additions & 4 deletions scrapegraph-js/src/credits.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,21 @@ import handleError from './utils/handleError.js';

/**
* Retrieve credits from the API.
*
*
* @param {string} apiKey - Your ScrapeGraph AI API key
* @returns {Promise<string>} Response from the API in JSON format
*/
export async function getCredits(apiKey) {
const endpoint = 'https://api.scrapegraphai.com/v1/credits';
const headers = {
'accept': 'application/json',
'SGAI-APIKEY': apiKey
'SGAI-APIKEY': apiKey,
};

try {
const response = await axios.get(endpoint, { headers });
return response.data;
} catch (error) {
handleError(error)
handleError(error);
}
}
}
8 changes: 4 additions & 4 deletions scrapegraph-js/src/feedback.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import handleError from './utils/handleError.js';

/**
* Send feedback to the API.
*
*
* @param {string} apiKey - Your ScrapeGraph AI API key
* @param {string} requestId - The request ID associated with the feedback
* @param {number} rating - The rating score
Expand All @@ -15,13 +15,13 @@ export async function sendFeedback(apiKey, requestId, rating, feedbackText = nul
const headers = {
'accept': 'application/json',
'SGAI-APIKEY': apiKey,
'Content-Type': 'application/json'
'Content-Type': 'application/json',
};

const feedbackData = {
request_id: requestId,
rating: rating,
feedback_text: feedbackText
feedback_text: feedbackText,
};

try {
Expand All @@ -30,4 +30,4 @@ export async function sendFeedback(apiKey, requestId, rating, feedbackText = nul
} catch (error) {
handleError(error);
}
}
}
14 changes: 7 additions & 7 deletions scrapegraph-js/src/smartScraper.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { zodToJsonSchema } from 'zod-to-json-schema';

/**
* Scrape and extract structured data from a webpage using ScrapeGraph AI.
*
*
* @param {string} apiKey - Your ScrapeGraph AI API key
* @param {string} url - The URL of the webpage to scrape
* @param {string} prompt - Natural language prompt describing what data to extract
Expand All @@ -18,12 +18,12 @@ export async function smartScraper(apiKey, url, prompt, schema = null) {
const headers = {
'accept': 'application/json',
'SGAI-APIKEY': apiKey,
'Content-Type': 'application/json'
'Content-Type': 'application/json',
};

const payload = {
website_url: url,
user_prompt: prompt
user_prompt: prompt,
};

if (schema) {
Expand All @@ -38,13 +38,13 @@ export async function smartScraper(apiKey, url, prompt, schema = null) {
const response = await axios.post(endpoint, payload, { headers });
return response.data;
} catch (error) {
handleError(error)
handleError(error);
}
}

/**
* Retrieve the status or the result of a smartScraper request. It also allows you to see the result of old requests.
*
*
* @param {string} apiKey - Your ScrapeGraph AI API key
* @param {string} requestId - The request ID associated with the output of a smartScraper request.
* @returns {Promise<string>} Information related to the status or result of a scraping request.
Expand All @@ -60,6 +60,6 @@ export async function getSmartScraperRequest(apiKey, requestId) {
const response = await axios.get(endpoint, { headers });
return response.data;
} catch (error) {
handleError(error)
handleError(error);
}
}
}
4 changes: 2 additions & 2 deletions scrapegraph-js/src/utils/handleError.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,10 @@ class UnexpectedError extends Error {

export default function handleError(error) {
if (error.response) {
throw new HttpError(error.response.status, error.response.statusText, error.response.data.detail)
throw new HttpError(error.response.status, error.response.statusText, error.response.data.detail);
} else if (error.request) {
throw new NetworkError('Impossible to contact the server. Check your internet connection.');
} else {
throw new UnexpectedError(`${error.message}`);
}
}
}
Loading