Subscribe
Sign in
Crawly — Never write another web scraper
Turn websites into data in seconds. Crawly spiders and extracts complete structured data from an entire website.
42
Replies
@callmeed Thanks Erik. If you'd like, I can set you up with a Diffbot trial account for crawling ecommerce pages. Interested?
I love this for its uniqueness ability to specify parameters. Hugely useful!
Adding this to my running list of Scraping and Crawling Technologies!
http://scrapinghub.com
http://www.outwit.com/products/hub/
http://webroots.io
http://kimonolabs.com
http://grabby.io
http://fullcontact.com
http://emailhunter.co
http://clearbit.com
http://toofr.com
http://import.io
http://kimonolabs.com
http://apifier.com (my number one favorite)
http://elink.club
http://www.eliteproxyswitcher.com/ - ;)
http://www.uipath.com/
http://diffbot.com
http://cloudscrape.com
http://community.screen-scraper....
https://commoncrawl.org/
http://www.fminer.com/
https://scraperwiki.com/
http://nutch.apache.org/
http://www.ubotstudio.com/index7
http://mozenda.com
http://fivefilters.org/
Awesome!!! Looking for a replacement for Kimono (https://www.kimonolabs.com/) since they got acquired by Palantir. Need something to power my Slack menu bot ;)
Is there a way to customize what data gets scraped? The ability to mouse over elements and select/de-select them would be ideal.
@drlynam Not at the moment within Crawly. That's something that we fully support using our Custom Api (http://www.diffbot.com/products/...)
@druwynings Any student access at a cheaper price? love this kinda stuff but don't have the money for it
Looks great. Congrats on the launch! I can see some nice applications for this. One suggestion - I understand that it might take a while to scrape the data, but an instant email to say it will be X minutes, or just a notification after email input would be good, to manage expectations. I used it 10 mins ago, and am on the verge of tears that I still have no email... ;-)
@neilcocker Hey Neil, things should be back to normal. Servers were crumbling under the PH load :)
@neilcocker I didn't want to inundate people with unnecessary emails, but I don't want people crying either...
@druwynings Tears are over. All working now. Very impressive. Good work - this will definitely be very useful.
Just tried not so great.
1. does not capture the actual URL, it captures root/ folder
2. We can save as json and csv? what's the use of that? We need to call the API each time right? is it only onetime getting?
@sarthakgrover To be honest, probably not. That being said, Crawly's big brother is Crawlbot (https://www.diffbot.com/products...) which has a fully-supported API (https://www.diffbot.com/dev/docs...)
Diffbot is such a useful service for manipulating published content - huge fan of this team.
Hey ProductHunters!
Crawly is a free tool I built that uses Diffbot's automatic article extraction api to turn web content into structured data. I've used it for creating a centralized database of all of our content, but you could also use it to do content audits / migrations or analyze your competitor's content.
It's currently limited to 200 pages and only articles at the moment, but I plan to add support for scraping products in the future. Any other features you'd like to see added?
@druwynings You should really update that page to clarify the "articles only" caveat. Especially when the tagline on your home page is "No rules required"
@callmeed Will do! Like I mentioned, support for products, discussions, images, and videos is in the works.
Seems, like a suitable solution for small data scraping tasks, I am going to make a little testing to it right now! However, for more extended tasks, it is reasonable to go for professional services on request, like this one https://data-ox.com/webscraping-..., for instance. This guys can also develop a custom tool for specific needs, by the way.
Baremetrics