|
1 |
| -# ScrapeGraph JS SDK |
| 1 | +# π ScrapeGraph JavaScript SDK |
2 | 2 |
|
3 |
| -A JavaScript SDK for interacting with the ScrapeGraph AI API. This SDK provides easy-to-use functions for web scraping, managing credits, and submitting feedback. |
| 3 | +[](https://badge.fury.io/js/scrapegraph-js) |
| 4 | +[](https://opensource.org/licenses/MIT) |
| 5 | +[](https://github.com/ScrapeGraphAI/scrapegraph-sdk/actions) |
| 6 | +[](https://docs.scrapegraphai.com) |
4 | 7 |
|
5 |
| -## Installation |
| 8 | +Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scraping powered by AI. |
6 | 9 |
|
7 |
| -Install the package using npm: |
| 10 | +## π Features |
8 | 11 |
|
9 |
| -```bash |
10 |
| -npm install scrapegraph-js |
11 |
| -``` |
| 12 | +- β¨ Smart web scraping with AI |
| 13 | +- π Fully asynchronous design |
| 14 | +- π Detailed error handling |
| 15 | +- β‘ Automatic retries and logging |
| 16 | +- π Secure API authentication |
12 | 17 |
|
13 |
| -## Usage |
| 18 | +## π¦ Installation |
14 | 19 |
|
15 |
| -> [!WARNING] |
16 |
| -> Remember not to write API keys directly in the code; instead, store them securely in `.env` files. |
| 20 | +Install the package using npm or yarn: |
17 | 21 |
|
18 |
| -First, import the required functions: |
| 22 | +```bash |
| 23 | +# Using npm |
| 24 | +npm i scrapegraph-js |
19 | 25 |
|
20 |
| -```javascript |
21 |
| -import { smartScraper, getSmartScraperRequest, getCredits, sendFeedback } from 'scrapegraph-sdk'; |
| 26 | +# Using yarn |
| 27 | +yarn add scrapegraph-js |
22 | 28 | ```
|
23 | 29 |
|
24 |
| -### Scraping Websites |
25 | 30 |
|
26 |
| -#### Basic scraping |
| 31 | +## π§ Quick Start |
| 32 | + |
| 33 | +> **Note**: Store your API keys securely in environment variables. Use `.env` files and libraries like `dotenv` to load them into your app. |
| 34 | +
|
| 35 | +### Basic Example |
27 | 36 |
|
28 | 37 | ```javascript
|
29 |
| -import { smartScraper } from 'scrapegraph-sdk'; |
| 38 | +import { smartScraper } from 'scrapegraph-js'; |
| 39 | +import 'dotenv/config'; |
30 | 40 |
|
31 |
| -const apiKey = process.env.SGAI_APIKEY; |
32 |
| -const url = 'https://scrapegraphai.com'; |
| 41 | +// Initialize variables |
| 42 | +const apiKey = process.env.SGAI_APIKEY; // Set your API key as an environment variable |
| 43 | +const websiteUrl = 'https://example.com'; |
33 | 44 | const prompt = 'What does the company do?';
|
34 | 45 |
|
35 |
| -try { |
36 |
| - const response = await smartScraper(apiKey, url, prompt); |
37 |
| - console.log(response); |
38 |
| -} catch (error) { |
39 |
| - console.error(error); |
40 |
| -} |
| 46 | +(async () => { |
| 47 | + try { |
| 48 | + const response = await smartScraper(apiKey, websiteUrl, prompt); |
| 49 | + console.log(response.result); |
| 50 | + } catch (error) { |
| 51 | + console.error('Error:', error); |
| 52 | + } |
| 53 | +})(); |
41 | 54 | ```
|
42 | 55 |
|
43 |
| -#### Scraping with custom output schema |
| 56 | +## π― Examples |
44 | 57 |
|
45 |
| -```javascript |
46 |
| -import { smartScraper } from 'scrapegraph-sdk'; |
| 58 | +### Scraping Websites |
47 | 59 |
|
48 |
| -const apiKey = 'your_api_key'; |
49 |
| -const url = 'https://scrapegraphai.com'; |
50 |
| -const prompt = 'What does the company do?'; |
51 |
| -const schema = //TODO |
52 |
| - |
53 |
| -try { |
54 |
| - const response = await smartScraper(apiKey, url, prompt, schema); |
55 |
| - console.log(response); |
56 |
| -} catch (error) { |
57 |
| - console.error(error); |
58 |
| -} |
| 60 | +#### Basic Scraping |
| 61 | + |
| 62 | +```javascript |
| 63 | +import { smartScraper } from 'scrapegraph-js'; |
| 64 | + |
| 65 | +const apiKey = 'your-api-key'; |
| 66 | +const url = 'https://example.com'; |
| 67 | +const prompt = 'Extract the main heading and description.'; |
| 68 | + |
| 69 | +(async () => { |
| 70 | + try { |
| 71 | + const response = await smartScraper(apiKey, url, prompt); |
| 72 | + console.log(response.result); |
| 73 | + } catch (error) { |
| 74 | + console.error('Error:', error); |
| 75 | + } |
| 76 | +})(); |
59 | 77 | ```
|
60 | 78 |
|
61 |
| -### Checking Credits |
| 79 | +#### Scraping with Custom Output Schema |
62 | 80 |
|
63 | 81 | ```javascript
|
64 |
| -import { getCredist } from 'scrapegraph-sdk'; |
| 82 | +//TODO |
| 83 | +``` |
65 | 84 |
|
66 |
| -const apiKey = 'your_api_key'; |
| 85 | +### Checking API Credits |
67 | 86 |
|
68 |
| -try { |
69 |
| - const myCredit = await getCredits(apiKey); |
70 |
| - console.log(myCredit) |
71 |
| -} catch (error) { |
72 |
| - console.error(error) |
73 |
| -} |
| 87 | +```javascript |
| 88 | +import { getCredits } from 'scrapegraph-js'; |
| 89 | + |
| 90 | +const apiKey = 'your-api-key'; |
| 91 | + |
| 92 | +(async () => { |
| 93 | + try { |
| 94 | + const credits = await getCredits(apiKey); |
| 95 | + console.log('Available credits:', credits); |
| 96 | + } catch (error) { |
| 97 | + console.error('Error fetching credits:', error); |
| 98 | + } |
| 99 | +})(); |
74 | 100 | ```
|
75 | 101 |
|
76 | 102 | ### Submitting Feedback
|
77 | 103 |
|
78 | 104 | ```javascript
|
79 |
| -import { sendFeedback } from 'scrapegraph-sdk'; |
| 105 | +import { sendFeedback } from 'scrapegraph-js'; |
80 | 106 |
|
81 |
| -const apiKey = 'your_api_key'; |
| 107 | +const apiKey = 'your-api-key'; |
82 | 108 | const requestId = '16a63a80-c87f-4cde-b005-e6c3ecda278b';
|
83 | 109 | const rating = 5;
|
84 |
| -const feedbackMessage = 'This is a test feedback message.'; |
85 |
| - |
86 |
| -try { |
87 |
| - const feedback_response = await sendFeedback(apiKey, requestId, rating, feedbackMessage); |
88 |
| - console.log(feedback_response); |
89 |
| -} catch (error) { |
90 |
| - console.error(error) |
91 |
| -} |
| 110 | +const feedbackText = 'This is a test feedback message.'; |
| 111 | + |
| 112 | +(async () => { |
| 113 | + try { |
| 114 | + const response = await sendFeedback(apiKey, requestId, rating, feedbackText); |
| 115 | + console.log('Feedback response:', response); |
| 116 | + } catch (error) { |
| 117 | + console.error('Error sending feedback:', error); |
| 118 | + } |
| 119 | +})(); |
92 | 120 | ```
|
93 | 121 |
|
94 |
| -## API Reference |
95 |
| - |
96 |
| -### scrape(apiKey, url[, options]) |
| 122 | +## π Documentation |
97 | 123 |
|
98 |
| -Scrapes a website and returns the extracted data. |
| 124 | +For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegraphai.com) |
99 | 125 |
|
100 |
| -Parameters: |
101 |
| -- `apiKey` (string): Your ScrapeGraph AI API key |
102 |
| -- `url` (string): The URL to scrape |
103 |
| -- `options` (object, optional): |
104 |
| - - `elements` (array): Specific elements to extract |
105 |
| - - `wait_for` (string): CSS selector to wait for before scraping |
106 |
| - - `javascript` (boolean): Enable JavaScript rendering |
| 126 | +## π οΈ Development |
107 | 127 |
|
108 |
| -### credits(apiKey) |
| 128 | +### Setup |
109 | 129 |
|
110 |
| -Retrieves your current credit balance. |
| 130 | +1. Clone the repository: |
| 131 | + ```bash |
| 132 | + git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git |
| 133 | + cd scrapegraph-sdk/scrapegraph-js |
| 134 | + ``` |
111 | 135 |
|
112 |
| -Parameters: |
113 |
| -- `apiKey` (string): Your ScrapeGraph AI API key |
| 136 | +2. Install dependencies: |
| 137 | + ```bash |
| 138 | + npm install |
| 139 | + ``` |
114 | 140 |
|
115 |
| -### feedback(apiKey, requestId, rating, feedbackText) |
| 141 | +3. Run linting and testing: |
| 142 | + ```bash |
| 143 | + npm run lint |
| 144 | + npm test |
| 145 | + ``` |
116 | 146 |
|
117 |
| -Submits feedback for a scraping request. |
| 147 | +### Running Tests |
118 | 148 |
|
119 |
| -Parameters: |
120 |
| -- `apiKey` (string): Your ScrapeGraph AI API key |
121 |
| -- `requestId` (string): The request ID from the scrape response |
122 |
| -- `rating` (number): Rating score |
123 |
| -- `feedbackText` (string) (optional): Feedback message |
| 149 | +```bash |
| 150 | +# Run all tests |
| 151 | +npm test |
124 | 152 |
|
125 |
| -## Error Handling |
| 153 | +# Run tests with coverage |
| 154 | +npm run test:coverage |
| 155 | +``` |
126 | 156 |
|
127 |
| -All functions return javascript `Error` object with imformation. In case of errors, the response will include error details: |
| 157 | +## π License |
128 | 158 |
|
129 |
| -// TODO error list |
| 159 | +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
130 | 160 |
|
131 |
| -```javascript |
132 |
| -{ |
133 |
| - "statusCode": 400, |
134 |
| - "title": "HTTP error occurred" |
135 |
| - "details": "Error details", |
136 |
| - |
137 |
| -} |
138 |
| -``` |
| 161 | +## π€ Contributing |
139 | 162 |
|
140 |
| -## License |
| 163 | +Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change. |
141 | 164 |
|
142 |
| -MIT |
| 165 | +1. Fork the repository |
| 166 | +2. Create your feature branch (`git checkout -b feature/AmazingFeature`) |
| 167 | +3. Commit your changes (`git commit -m 'Add some AmazingFeature'`) |
| 168 | +4. Push to the branch (`git push origin feature/AmazingFeature`) |
| 169 | +5. Open a Pull Request |
143 | 170 |
|
144 |
| -## Support |
| 171 | +## π Links |
145 | 172 |
|
146 |
| -For support, please visit [ScrapeGraph AI Documentation](https://sgai-api.onrender.com/docs). |
| 173 | +- [Website](https://scrapegraphai.com) |
| 174 | +- [Documentation](https://scrapegraphai.com/documentation) |
| 175 | +- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk) |
147 | 176 |
|
| 177 | +## π¬ Support |
148 | 178 |
|
| 179 | + |
| 180 | +- π» GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues) |
| 181 | +- π Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new) |
149 | 182 |
|
| 183 | +--- |
150 | 184 |
|
| 185 | +Made with β€οΈ by [ScrapeGraph AI](https://scrapegraphai.com) |
0 commit comments