Skip to content

feat: add infinite scrolling #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat: add infinite scrolling #38

wants to merge 4 commits into from

Conversation

VinciGit00
Copy link
Contributor

No description provided.

@gitguardian GitGuardian
Copy link

gitguardian bot commented Jun 10, 2025 •
edited
Loading

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 10, 2025 •
edited
Loading

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@VinciGit00
Copy link
Contributor Author

Screenshot 2025-06-10 alle 15 33 50

Vikrant-Khedkar
Vikrant-Khedkar reviewed Jun 10, 2025
View reviewed changes
scrapegraph-py/uv.lock
@@ -1525,7 +1525,7 @@ dev = [

[package.metadata]
requires-dist = [
{ name = "aiohttp", specifier = ">=3.11.8" },
{ name = "aiohttp", specifier = ">=3.10" },
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we encounter any problems with aiohttp 3.11.8 that we’re considering downgrading to 3.10?

@Vikrant-Khedkar
Copy link
Collaborator

Rest LGTM!

mdehsan873
mdehsan873 reviewed Jun 10, 2025
View reviewed changes
scrapegraph-py/examples/async/async_smartscraper_infinite_scroll_example.py
async def scrape_companies(client: AsyncClient, url: str, batch: str) -> None:
"""Scrape companies from a specific YC batch with infinite scroll."""
try:
response = await client.smartscraper(
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try block should be a linear only

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update it

mdehsan873
mdehsan873 reviewed Jun 10, 2025
View reviewed changes
scrapegraph-py/examples/sync/smartscraper_infinite_scroll_example.py Outdated
companies: List[Company]

# Initialize the client with explicit API key
sgai_client = Client(api_key="sgai-4cf4a4f5-87f7-457a-8c58-0790ecaf323e")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@VinciGit00 APi key?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Vikrant-Khedkar
Vikrant-Khedkar approved these changes Jun 10, 2025
View reviewed changes
scrapegraph-py/examples/sync/smartscraper_infinite_scroll_example.py Outdated
companies: List[Company]

# Initialize the client with explicit API key
sgai_client = Client(api_key="sgai-4cf4a4f5-87f7-457a-8c58-0790ecaf323e")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might be exposing a secret here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdehsan873 mdehsan873 mdehsan873 left review comments

@Vikrant-Khedkar Vikrant-Khedkar Vikrant-Khedkar approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@VinciGit00 @Vikrant-Khedkar @mdehsan873