faa0704737
Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
42 lines
480 B
Plaintext
42 lines
480 B
Plaintext
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
|
|
|
|
# dependencies
|
|
/node_modules
|
|
/.pnp
|
|
.pnp.*
|
|
.yarn/*
|
|
!.yarn/patches
|
|
!.yarn/plugins
|
|
!.yarn/releases
|
|
!.yarn/versions
|
|
|
|
# testing
|
|
/coverage
|
|
|
|
# next.js
|
|
/.next/
|
|
/out/
|
|
|
|
# production
|
|
/build
|
|
|
|
# misc
|
|
.DS_Store
|
|
*.pem
|
|
|
|
# debug
|
|
npm-debug.log*
|
|
yarn-debug.log*
|
|
yarn-error.log*
|
|
.pnpm-debug.log*
|
|
|
|
# env files (can opt-in for committing if needed)
|
|
.env*
|
|
|
|
# vercel
|
|
.vercel
|
|
|
|
# typescript
|
|
*.tsbuildinfo
|
|
next-env.d.ts
|