Piranavan Selvanandan's Team
Unmute
Crawlee is a popular (~50k downloads/week!) web scraping and browser automation library that allows developers to programmatically navigate and interact with web pages.
You'll be addressing a bug that prevents Crawlee's PlaywrightCrawler from enqueueing links (i.e., adding URLs to a list of webpages to be scraped) if the URL of the webpage from which those links are sourced has a www redirect (e.g., https://reddit.com/r/legal redirects to https://www.reddit.com/r/legal).
Note: Since Crawlee is a library, once you've set up your development environment, you'll have to import and use it in your own code to try it out and make sure everything is working properly. Also, be sure to obtain the source code by forking and cloning the repository (and not by following the quick start / installation instructions for non-contributors) so that you'll be able to make a PR once you're done!
You'll be addressing a bug that prevents Crawlee's PlaywrightCrawler from enqueueing links (i.e., adding URLs to a list of webpages to be scraped) if the URL of the webpage from which those links are sourced has a www redirect (e.g., https://reddit.com/r/legal redirects to https://www.reddit.com/r/legal).
Note: Since Crawlee is a library, once you've set up your development environment, you'll have to import and use it in your own code to try it out and make sure everything is working properly. Also, be sure to obtain the source code by forking and cloning the repository (and not by following the quick start / installation instructions for non-contributors) so that you'll be able to make a PR once you're done!
What tools did you use to create your project?
How much experience does your group have? Does the project use anything (art, music, starter kits) you didn't create?
CodeDay Labs team in the advanced track.
Made with 💖 by robots running our open-source software and GraphQL APIs.