Commit Graph

14 Commits

Author SHA1 Message Date
Alejandro Gutiérrez
bdffb5eaac Add API interception for hybrid scraping and update selectors
- Add new api_interceptor.py module for CDP network interception
- Capture Google Maps internal API responses during scrolling
- Parse protobuf-like JSON responses to extract review data
- Merge API-captured reviews with DOM-scraped data
- Update CSS selectors for January 2026 Google Maps structure
- Add cookie consent dismissal for multiple languages
- Add --api-intercept CLI flag and config option
- Fix review card and pane selectors (.jftiEf, .XiKgde)
- Improve review ID extraction from card elements

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-17 21:51:10 +00:00
George Khananaev
262f0c0be7 migrate to SeleniumBase UC Mode for automatic version management
- Replace undetected-chromedriver with seleniumbase for better Chrome/ChromeDriver compatibility
- Automatic version matching eliminates manual cache clearing and version conflicts
- Enhanced anti-detection with UC Mode and CDP stealth settings
- Simplified requirements.txt (SeleniumBase manages common dependencies)
- Fix sort selection bug (was selecting wrong menu items)
- Improve scrolling patience (max_idle: 3→15, max_attempts: 10→50)
- Add scroll position tracking to detect when stuck
- Add fallback pane selectors for better reliability
- Update documentation (README, ARCHITECTURE, TROUBLESHOOTING)
- Add comprehensive test suite for SeleniumBase integration
- Version bump to 1.0.1

Developed by George Khananaev
2025-12-07 19:40:13 +07:00
George Khananaev
6b60b02eec Test 2025-08-20 02:46:01 +07:00
George Khananaev
dddf388422 Added api support, now the scrapper can be triggered from 3rd party services 2025-08-20 02:42:01 +07:00
George Khananaev
0b561f7618 Merge pull request #2 from rrmn/master
Get original size images from Google
2025-08-20 00:18:00 +07:00
RomanAbashin
72fcc6f162 Get original size images from Google 2025-08-09 10:55:51 +03:00
George Khananaev
50aaa9ce26 Added pytest + some tests.
Added AWS S3 Support (optional, for cloud image storage)
2025-06-03 00:12:11 +07:00
George Khananaev
84399dfbe8 Merge branch 'detached'
# Conflicts:
#	modules/scraper.py
2025-06-02 23:33:29 +07:00
George Khananaev
c4fa7ecd93 fixed the english scraper 2025-06-02 23:22:19 +07:00
George Khananaev
54f98ae921 fixed the issue with english localization 2025-06-02 13:22:50 +07:00
George Khananaev
cbc4bfe72d added config file. 2025-05-12 01:30:17 +07:00
George Khananaev
06bbd18b6b Update README.md 2025-05-02 00:35:44 +07:00
George Khananaev
c6011b7c50 Added config example and sample output
Threw in some practical stuff:

- Detailed config.yaml with all the settings explained
- Sample JSON output showing what you actually get from this thing
- Comments in the sample so people know WTF each field means

Should help folks figure out how to set this up without having to read the whole damn README. I'll probably add more examples later when I get time.

Co-Authored-By: George K (MHG) <122952523+ttm-tech@users.noreply.github.com>
2025-04-24 23:19:36 +07:00
George Khananaev
5bbaf455d8 Release Google Reviews Scraper Pro v1.0.0 (2025)
Initial release with multi-language support, MongoDB integration, image handling, URL replacement, and robust error handling. Includes detailed documentation, usage examples, and recommended usage guidelines. Built to effectively handle Google's 2025 interface changes.
2025-04-24 22:12:07 +07:00