How to scrape any website or service in Scade

[nik]

Howdy! We finally added an APIFY node and now you can extract data from LITERALLY anywhere.

Let’s take an example.

You want to make a summary of some creator on X. Like, what topics are covered by this fella and what is he up to.

You’ve already prepped some stunning prompt via prompt gen template. And finally you are ready to analyze profile with 100 tweets. How to parse em?
Ofc you can always send your $100 to Sir Elon

But instead you can use an APIFY and Scade.

1. Create an account in Apify
2. Open Profile Settings
   
   
3. Go to “Integrations”
4. Copy your API key
   
   

Now you need to create APIFY credentials in Scade.

1. Create new flow in Scade
2. Add Apify node (just type it in searchbar or scroll down in popular category)
   
   
3. Add credentials
   
   
4. Paste your API key from previous steps at APIFY platform

Now let’s add a scraper.

1. Open APIFY catalog
2. Select any suitable Actor (i suggest you find one with pay for result billing type)
   
   
3. Copy this id
   
   
4. Paste it in node input
   
   

Now we will configure it in apify and paste config in Scade

1. Make all your changes and switch to JSON format
   

   
   
   
   

2. Paste this config in Scade APIFY node
   

   
   
   
   

3. Let’s add some extra parameters to clean-up our output
   

   
   
   You can also filter your output right here if you want

4. Tadaa! You can see your scraped tweets. So you can pass it to ChatGPT or Claude node and analyze

[KarlieI]

Thanks for the tutorial — especially the bit about the Extract Dataset toggle, which was super helpful! But I gotta say, I was pretty frustrated to find out that using this particular Actor requires OMG a $50/month APIFY subscription. Like what? We are talking about $100 vs $50 cost difference! It’s really disappointing to dive into all those tips and tricks, only to hit a paywall at the very end. Not cool!

[george]

I totally agree w @KarlieI — @nik you should really find a better example for APIFY tutorial. Also, has anyone tried using GPT-4 or Perplexity Node to parse X profiles? Curious to hear if it works!

[nik]

Karliel, your point is valid. It is true that you must have a paid plan that costs $50 per month until Scade incorporates it into their own plans. However, for that price, you will not only have access to X but also numerous other actors. Additionally, these actors are not just basic APIs for specific services; they are a more user-friendly API that enables you to effortlessly obtain any necessary data

[nik]

Regarding the X - i tried GPT4o and Perplexity.
I know there should be some prompt allowing you to extract data from specific url with GPT4, but It doesn’t work with X so far.

As for perplexity - for me this prompt works:

what are the latest tweets on
https://x.com/elonmusk