Subscribe
Sign in
Simple Scraper — Turn any website into an API in just a few seconds.
The easiest way to scrape the web — Simple scraper turns websites into structured data with just a few clicks.
62
Replies
@mrdobelina Hey, these are introductory credits — however there will always be a free plan. Once there's enough data to have a better understanding of usage / expenses we'll lock down a fair monthly figure. For now there's a soft cap - credits decrease but you can still scrape if they hit zero.
Hey @humanismusic_ 👋
Just trying it out and I have to say – really nice job :)
I have some feedback: (is this the best place to share it?)
• When trying to select the "next" element for pagination, the extension correctly highlights in on hover, but when I click it to select, I actually click the link and thus end up being redirected to the next page. (Tried on GitHub) https://take.ms/f9leZ
• It took me a while to realize that I can actually edit the selector, it's really not obvious https://take.ms/AUgAb
• When looking for "pagination" functionality, I actually glanced over it a few times as I was sure that the icon next to View results means Copy to clipboard https://take.ms/B7sPq
• When creating a scraper, it could tell me that I already have a scraper for this site / offer scrapers from Recipe store
• Personally, I would prefer a more condensed table preview for the results.
• Copy to Clipboard would be useful as an alternative to CSV and JSON
@straaajk Hey Pavel,
Thanks for taking the time to put together such great feedback. Rather than reply to each point individually, let's just say I agree with most of them and am busy working on the next version of Simplescraper which should improve much of what you've mentioned.
For pagination, try using the SHIFT key instead of clicking to select the 'next' element without navigating the page.
Cheers!
Does this give you the ability to extract additional details found in child/nested pages?
@kdexsta Hi Kevin, you could create one recipe to scrape urls (from a list of product results for example) and then create a second recipe for the product page itself. You then feed the url results from the first recipe into the second in order to crawl all the individual product pages.
The only thing missing is an easy way to import url results from one recipe into another - working on it right now so should be possible in a few days.
Update: this is now live.
Hands down Mike! Cool product. I tested with Amazon product listings and the only issue I see is that it's not able to scrape multiple pages in one recipe run.
@humanismusic_ feels like I was missing the party here! Awesome product btw! I'm curious about one thing. Do you handle IP blacklisting? From my xp when scrapping, it's quite easy to get your IP blocked (if using a backend with fixed ip)
@jpgbarbosa party's just getting started :) Thank you and yes IP-rotation is built in so you shouldn't be blocked while scraping
@anthemaker Hey, yes it's started. Once the Chrome version is stable I'll port over fully.
Hey PH,
Simple scraper is a tool I've created to make web scraping easy.
You select the data you want, create a recipe, and about 30 seconds later you've got a URL endpoint that you can query for results. (there's JSON and CSV download options too).
The idea is to turn any webpage into an API, making data extraction as simple as possible.
After a month of feedback, coding and coffee I've just hit the launch button with new features like:
⚡ Scheduling: run recipes daily or hourly, or in real-time via a URL endpoint
⚡ Crawler: scrape hundreds of links using any recipe you've created
Take it for a spin and let me know what you think. Hope it adds value.
Peace, Mike
Can the product go into pages behind a log-in - for example, can it scrape from data that I have in my Notion accounts that is a private page?
@mina_iskarous Hi, not with cloud scraping at the moment - but yes in the future. For now can use 'local' scraping for sites behind a login.
So when I pull data, it has to be from tables or some repeatable div/ card elements? I pick the of a table and it figures out the elements from body, pulls, cleans and gives it as an API?
Also love the choice of music.
@varun_agarwal1 Hi Varun, yes that's pretty much it. It will scrape whatever your select, in most cases this means data with a specific structure / pattern (title, price, description etc.) but that's up to you.
> Also love the choice of music.
Excellent taste ;)
Awesome product!
I have one question... Does it update the data (ie. in a price change) or it's only static data?
@thiago_sardim thanks Thiago! To get the latest data from a website automatically you have two options:
1. set a schedule (hourly and daily)
2. add the 'run_now=true' parameter if you're accessing the data via API
@serhat_iletir there's a quick intro guide here - https://simplescraper.io/guide, and for the API the parameters are explained on the API tab. (will create more video guides over the coming weeks)
Hey @humanismusic_ how do I scrap a link description which opens a new tab on clicking? Like this website https://www.1mg.com/manufacturer.... After clicking on a link, another page opens and the extension doesn't stay active.
@mukul23 Hey Mukul, think this may be a similar question to Kevin's so I'll link to that (https://www.producthunt.com/post...). Basically you create one recipe to scrape a list of urls and crawl each of those urls using another recipe.
Hey hunters,
first of all congratulations on simple scraper ,was an amazing it looks cool :) . We also happen to have developed a social network that is structured based on a tree. We would appreciate it if you could check it out and provides us with useful feedback , since it is in under development. https://subranch.com/?ref=produc...
Spacepod