|
1 |
| -# ScrapeGraph JS SDK |
| 1 | +# π ScrapeGraph JavaScript SDK |
2 | 2 |
|
3 |
| -A JavaScript SDK for interacting with the ScrapeGraph AI API. This SDK provides easy-to-use functions for web scraping, managing credits, and submitting feedback. |
| 3 | +[](https://badge.fury.io/js/scrapegraph-js) |
| 4 | +[](https://www.typescriptlang.org/) |
| 5 | +[](https://opensource.org/licenses/MIT) |
| 6 | +[](https://github.com/ScrapeGraphAI/scrapegraph-sdk/actions) |
| 7 | +[](https://docs.scrapegraphai.com) |
4 | 8 |
|
5 |
| -## Installation |
| 9 | +Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scraping powered by AI. |
6 | 10 |
|
7 |
| -Install the package using npm: |
| 11 | +## π Features |
8 | 12 |
|
9 |
| -```bash |
10 |
| -npm install scrapegraph-js |
11 |
| -``` |
| 13 | +- β¨ Smart web scraping with AI |
| 14 | +- π Fully asynchronous design |
| 15 | +- π TypeScript-ready with strongly typed responses |
| 16 | +- π Detailed error handling |
| 17 | +- β‘ Automatic retries and logging |
| 18 | +- π Secure API authentication |
12 | 19 |
|
13 |
| -## Usage |
| 20 | +## π¦ Installation |
14 | 21 |
|
15 |
| -> [!WARNING] |
16 |
| -> Remember not to write API keys directly in the code; instead, store them securely in `.env` files. |
| 22 | +Install the package using npm or yarn: |
17 | 23 |
|
18 |
| -First, import the required functions: |
| 24 | +```bash |
| 25 | +# Using npm |
| 26 | +npm install scrapegraph-js |
19 | 27 |
|
20 |
| -```javascript |
21 |
| -import { smartScraper, getSmartScraperRequest, getCredits, sendFeedback } from 'scrapegraph-sdk'; |
22 |
| -``` |
23 | 28 |
|
24 |
| -### Scraping Websites |
| 29 | +## π§ Quick Start |
| 30 | + |
| 31 | +> **Note**: Store your API keys securely in environment variables. Use `.env` files and libraries like `dotenv` to load them into your app. |
25 | 32 |
|
26 |
| -#### Basic scraping |
| 33 | +### Basic Example |
27 | 34 |
|
28 | 35 | ```javascript
|
29 |
| -import { smartScraper } from 'scrapegraph-sdk'; |
| 36 | +import { smartScraper } from 'scrapegraph-js'; |
30 | 37 |
|
31 |
| -const apiKey = process.env.SGAI_APIKEY; |
32 |
| -const url = 'https://scrapegraphai.com'; |
| 38 | +// Initialize variables |
| 39 | +const apiKey = process.env.SGAI_APIKEY; // Set your API key as an environment variable |
| 40 | +const websiteUrl = 'https://example.com'; |
33 | 41 | const prompt = 'What does the company do?';
|
34 | 42 |
|
35 |
| -try { |
36 |
| - const response = await smartScraper(apiKey, url, prompt); |
37 |
| - console.log(response); |
38 |
| -} catch (error) { |
39 |
| - console.error(error); |
40 |
| -} |
| 43 | +(async () => { |
| 44 | + try { |
| 45 | + const response = await smartScraper(apiKey, websiteUrl, prompt); |
| 46 | + console.log(response.result); |
| 47 | + } catch (error) { |
| 48 | + console.error('Error:', error); |
| 49 | + } |
| 50 | +})(); |
41 | 51 | ```
|
42 | 52 |
|
43 |
| -#### Scraping with custom output schema |
| 53 | +## π― Examples |
| 54 | +
|
| 55 | +### Scraping Websites |
| 56 | +
|
| 57 | +#### Basic Scraping |
44 | 58 |
|
45 | 59 | ```javascript
|
46 |
| -import { smartScraper } from 'scrapegraph-sdk'; |
| 60 | +import { smartScraper } from 'scrapegraph-js'; |
| 61 | +
|
| 62 | +const apiKey = 'your-api-key'; |
| 63 | +const url = 'https://example.com'; |
| 64 | +const prompt = 'Extract the main heading and description.'; |
| 65 | +
|
| 66 | +(async () => { |
| 67 | + try { |
| 68 | + const response = await smartScraper(apiKey, url, prompt); |
| 69 | + console.log(response.result); |
| 70 | + } catch (error) { |
| 71 | + console.error('Error:', error); |
| 72 | + } |
| 73 | +})(); |
| 74 | +``` |
47 | 75 |
|
48 |
| -const apiKey = 'your_api_key'; |
49 |
| -const url = 'https://scrapegraphai.com'; |
50 |
| -const prompt = 'What does the company do?'; |
51 |
| -const schema = //TODO |
| 76 | +#### Scraping with Custom Output Schema |
| 77 | +
|
| 78 | +```typescript |
| 79 | +import { smartScraper } from 'scrapegraph-js'; |
52 | 80 |
|
53 |
| -try { |
54 |
| - const response = await smartScraper(apiKey, url, prompt, schema); |
55 |
| - console.log(response); |
56 |
| -} catch (error) { |
57 |
| - console.error(error); |
| 81 | +interface WebsiteData { |
| 82 | + title: string; |
| 83 | + description: string; |
58 | 84 | }
|
| 85 | +
|
| 86 | +const apiKey = 'your-api-key'; |
| 87 | +const url = 'https://example.com'; |
| 88 | +const prompt = 'Extract the title and description.'; |
| 89 | +
|
| 90 | +(async () => { |
| 91 | + try { |
| 92 | + const response = await smartScraper<WebsiteData>(apiKey, url, prompt); |
| 93 | + console.log(response.result.title, response.result.description); |
| 94 | + } catch (error) { |
| 95 | + console.error('Error:', error); |
| 96 | + } |
| 97 | +})(); |
59 | 98 | ```
|
60 | 99 |
|
61 |
| -### Checking Credits |
| 100 | +### Checking API Credits |
62 | 101 |
|
63 | 102 | ```javascript
|
64 |
| -import { getCredist } from 'scrapegraph-sdk'; |
65 |
| - |
66 |
| -const apiKey = 'your_api_key'; |
67 |
| - |
68 |
| -try { |
69 |
| - const myCredit = await getCredits(apiKey); |
70 |
| - console.log(myCredit) |
71 |
| -} catch (error) { |
72 |
| - console.error(error) |
73 |
| -} |
| 103 | +import { getCredits } from 'scrapegraph-js'; |
| 104 | +
|
| 105 | +const apiKey = 'your-api-key'; |
| 106 | +
|
| 107 | +(async () => { |
| 108 | + try { |
| 109 | + const credits = await getCredits(apiKey); |
| 110 | + console.log('Available credits:', credits); |
| 111 | + } catch (error) { |
| 112 | + console.error('Error fetching credits:', error); |
| 113 | + } |
| 114 | +})(); |
74 | 115 | ```
|
75 | 116 |
|
76 | 117 | ### Submitting Feedback
|
77 | 118 |
|
78 | 119 | ```javascript
|
79 |
| -import { sendFeedback } from 'scrapegraph-sdk'; |
| 120 | +import { sendFeedback } from 'scrapegraph-js'; |
80 | 121 |
|
81 |
| -const apiKey = 'your_api_key'; |
| 122 | +const apiKey = 'your-api-key'; |
82 | 123 | const requestId = '16a63a80-c87f-4cde-b005-e6c3ecda278b';
|
83 | 124 | const rating = 5;
|
84 |
| -const feedbackMessage = 'This is a test feedback message.'; |
85 |
| - |
86 |
| -try { |
87 |
| - const feedback_response = await sendFeedback(apiKey, requestId, rating, feedbackMessage); |
88 |
| - console.log(feedback_response); |
89 |
| -} catch (error) { |
90 |
| - console.error(error) |
91 |
| -} |
| 125 | +const feedbackText = 'This is a test feedback message.'; |
| 126 | +
|
| 127 | +(async () => { |
| 128 | + try { |
| 129 | + const response = await sendFeedback(apiKey, requestId, rating, feedbackText); |
| 130 | + console.log('Feedback response:', response); |
| 131 | + } catch (error) { |
| 132 | + console.error('Error sending feedback:', error); |
| 133 | + } |
| 134 | +})(); |
92 | 135 | ```
|
93 | 136 |
|
94 |
| -## API Reference |
95 |
| - |
96 |
| -### scrape(apiKey, url[, options]) |
| 137 | +## π Documentation |
97 | 138 |
|
98 |
| -Scrapes a website and returns the extracted data. |
| 139 | +For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegraphai.com) |
99 | 140 |
|
100 |
| -Parameters: |
101 |
| -- `apiKey` (string): Your ScrapeGraph AI API key |
102 |
| -- `url` (string): The URL to scrape |
103 |
| -- `options` (object, optional): |
104 |
| - - `elements` (array): Specific elements to extract |
105 |
| - - `wait_for` (string): CSS selector to wait for before scraping |
106 |
| - - `javascript` (boolean): Enable JavaScript rendering |
| 141 | +## π οΈ Development |
107 | 142 |
|
108 |
| -### credits(apiKey) |
| 143 | +### Setup |
109 | 144 |
|
110 |
| -Retrieves your current credit balance. |
| 145 | +1. Clone the repository: |
| 146 | + ```bash |
| 147 | + git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git |
| 148 | + cd scrapegraph-sdk/scrapegraph-js |
| 149 | + ``` |
111 | 150 |
|
112 |
| -Parameters: |
113 |
| -- `apiKey` (string): Your ScrapeGraph AI API key |
| 151 | +2. Install dependencies: |
| 152 | + ```bash |
| 153 | + npm install |
| 154 | + ``` |
114 | 155 |
|
115 |
| -### feedback(apiKey, requestId, rating, feedbackText) |
| 156 | +3. Run linting and testing: |
| 157 | + ```bash |
| 158 | + npm run lint |
| 159 | + npm test |
| 160 | + ``` |
116 | 161 |
|
117 |
| -Submits feedback for a scraping request. |
| 162 | +### Running Tests |
118 | 163 |
|
119 |
| -Parameters: |
120 |
| -- `apiKey` (string): Your ScrapeGraph AI API key |
121 |
| -- `requestId` (string): The request ID from the scrape response |
122 |
| -- `rating` (number): Rating score |
123 |
| -- `feedbackText` (string) (optional): Feedback message |
| 164 | +```bash |
| 165 | +# Run all tests |
| 166 | +npm test |
124 | 167 |
|
125 |
| -## Error Handling |
| 168 | +# Run tests with coverage |
| 169 | +npm run test:coverage |
| 170 | +``` |
126 | 171 |
|
127 |
| -All functions return javascript `Error` object with imformation. In case of errors, the response will include error details: |
| 172 | +## π License |
128 | 173 |
|
129 |
| -// TODO error list |
| 174 | +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
130 | 175 |
|
131 |
| -```javascript |
132 |
| -{ |
133 |
| - "statusCode": 400, |
134 |
| - "title": "HTTP error occurred" |
135 |
| - "details": "Error details", |
136 |
| - |
137 |
| -} |
138 |
| -``` |
| 176 | +## π€ Contributing |
139 | 177 |
|
140 |
| -## License |
| 178 | +Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change. |
141 | 179 |
|
142 |
| -MIT |
| 180 | +1. Fork the repository |
| 181 | +2. Create your feature branch (`git checkout -b feature/AmazingFeature`) |
| 182 | +3. Commit your changes (`git commit -m 'Add some AmazingFeature'`) |
| 183 | +4. Push to the branch (`git push origin feature/AmazingFeature`) |
| 184 | +5. Open a Pull Request |
143 | 185 |
|
144 |
| -## Support |
| 186 | +## π Links |
145 | 187 |
|
146 |
| -For support, please visit [ScrapeGraph AI Documentation](https://sgai-api.onrender.com/docs). |
| 188 | +- [Website](https://scrapegraphai.com) |
| 189 | +- [Documentation](https://scrapegraphai.com/documentation) |
| 190 | +- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk) |
147 | 191 |
|
| 192 | +## π¬ Support |
148 | 193 |
|
| 194 | + |
| 195 | +- π» GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues) |
| 196 | +- π Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new) |
149 | 197 |
|
| 198 | +--- |
150 | 199 |
|
| 200 | +Made with β€οΈ by [ScrapeGraph AI](https://scrapegraphai.com) |
0 commit comments