Subscribe
Sign in
ScrapingBee
p/scrapingbeeEasiest Web Scraping API
Start new thread
Pierre de Wulf

ScrapingBee (formerly ScrapingNinja) - The easiest web scraping API on the web

by•

Web Scraping is hard, scraping at scale can be very challenging.
ScrapingBee is a simple API that does many things for you so can just focus on the data you want to extract and the product you wan to build.

Add a comment

Replies

Best
Pierre de Wulf
Maker
Hello Product Hunt! I'm super excited to show you ScrapingBee (formerly ScrapingNinja) 🕷! One year ago, with my partner @sahinkevin we decided to leave our full-time job to go the entrepreneurship way 😎. Being both huge fans of the indie-hacker/shipping/entrepreneurship culture promoted by people such a @levelsio or @marckohlbrugge we wanted to see if we could make it. Fast forward 1 year, and after many ups and downs, we have finally something that currently brings us enough 💵 to 🍜, and although the battle is far from being won, we are proud to show you ScrapingNinja today. Web Scraping is hard, scraping at scale can be very challenging. You have to handle: - Javascript rendering 💻 - Chrome headless 🛠 - Captcha 🤖 - Proxy 🕵️‍♀️ ScrapingBee is a simple API that does all the above for you. We quickly realized that there was a need for less technical users for ready-made APIs that returns formatted data from the most common websites. That’s why we just launched our API-store: - https://www.scrapingbee.com/api-... The idea is to be able to turn every website into a usable API, we currently only support Instagram, but more are coming soon!
Sandor Farkas
Hey, is this really legally safe to do?
Frank Ramos
Launching soon!
@pierre_dwf Pierre has built an awesome product. Pulling in data through a headless browser using a proxy helps get around issues that makes most scrapers completely useless. It’s a great product. Glad I joined the beta.
Pierre de Wulf
Maker
@francisco_ramos Thank you very much!
David Clifford
Can it download things like audio files and such from websites as well?
Kevin Sahin
Maker
@david_clifford9 Hey David thanks for asking. Unfortunately at the moment ScrapingNinja API only returns the raw HTML from websites, or JSON formatted data if you use the ready made APIs. What would be your use case ?
Francoo
Hello there, congrats on launching ! Why should i pay for ninja when spiders is free ? https://www.producthunt.com/post... I only want to scrapp data from ecommerce once a day and apply some logic to the result to ping me (or not) IF some rule. Is your product right for me ?
Pierre de Wulf
Maker
@francoolaami , hello, the product you linked is meant to be used by a real human with its browser open, it does not allow you to scrape all day long, multiples pages at the same time for example. And about the thing you asked, ScrapingNinja can definetely help you, but you’d need to setup the alerting thing yourself :)
Alexandru Porcescu
@francoolaami we built a tool that extract product data from both category and product pages automatically. No selectors, no rules. We started working on alerting features. Try it out https://quantati.com/#demo-page
Pierre de Wulf
Maker
@francoolaami @litechip Hey Alexandru, great job on quantati. Just a few questions, Is quantati supposed to work on every website or only on a certain list of websites? I tried to extract data from Zalando of this url: https://www.zalando.fr/converse-... on your demo page, and it did not work, it only returned me "Type, url, humanlangage, resolvedpageurl'
Musab
What differentiate you from scraperapi.com ?
Paul Gordon
@pierre_dwf .. I am using scraperapi along with a few other proxy services .. would love to know how it compares to scraperapi?
Pierre de Wulf
Maker
@musab better proxy, real premium proxies, render js available for all plan, Api store :)
Pierre de Wulf
Maker
@pgordon please let me know how it went ;)
Paul Gordon
@pierre_dwf thanks .. yeah i noticed they moved their JS Rendering to the $99 plan .. mine still works as I am on a a grandfathered plan .. will signup soon and take a look
Musab
@pierre_dwf What is "real premium" proxy?
Marie Vernières
Easiest and most reliable scrapping API I have used yet, good job!
Shun Yamada
Congrats on your indie way! It's really helpful.
Pierre de Wulf
Maker
Hey, yes it does, we still have little problems with LinkedIn but we are working on it. The websites work fines
Tobin Smith
Can it help me scrape twitter followers?
Tobin Smith
@pierre_dwf what would you charge me to scrape Twitter hashtags and facebook groups?
Pierre de Wulf
Maker
@tobin_smith Please send me your required volume, concurrency at hello@scrapingninja.co, I'll answer to you ASAP :)
James J Beuerlein
What can this product do? The only web scraping I'm familiar with is what is done with kodi add ons and thats illegal.
Supratik Mukherjee
Hey @pierre_dwf This is awesome! Does ScrapingNinja work on sites like LinkedIn, Facebook, Twitter, Glassdoor etc? As some websites block scraping tools, was wondering if this would be helpful for our use case.
Pierre de Wulf
Maker
@soup_up Hey, thank you very much. ScrapingNinja works for every website although we have some trouble with Linkedin. We are currently working on it :)
Nino
Scraping didn't work for the sites I needed to scrape. Also, I did the feedback call with them, but never received the credits award, even after reminding them about it. Avoid.
Pierre de Wulf
Maker
Hi Nino, during the call you mentioned that you did not plan on using our API so we did not think you wanted the free credits. When you tell you reminded us about this, what do you mean? I could not find an email from you in our inbox. But if you changed your mind and plan to use it, please tell us and we will give them to you. No questions asked. We have made around 100 calls since launch and have never failed to deliver those credits, I am sad you felt that way.