I cant get any links from imdb.com to work with the Get URL as Text ( url ) function. I can make it work for other sites but I get a 403 error. any ideas?

The scriptmaster Get URL module is just a basic tool, and may have a problem on some sites. We may take a look at this at some point, but for now I recommend that you check out our Web Assistant Plugin at http://360works.com/url-plugin/ which should work for you.

I encountered the same 403 problem attempting to retrieve HTML from IMDb.com using ScriptMaster.

In the WebAssistant Example.fp7 demo, I modified the "Fetching HTML" example (#6) with an IMDb URL -- the full HTML appeared in the Result field.

That is the correct behavior. You should receive the HTML from the URL you are trying to retrieve. If you don't want all of the HTML tags I recommend using the WAStripTags to remove the HTML tags from the HTML you have retrieved.

The 403 error occurs because the user agent (browser) has not been set. Some sites will not respond unless the user agent is a browser; i.e. if it is cURL or Java, they won't return a page. When I used to use Troi's URL plugin, I used their HTTP-SETUSERAGENT command.

Setting it to Mozilla or IE will allow it to work, e.g. use this to fetch the IMDB url from SM:

url_to_fetch= new URL(url);

    		

HttpURLConnection httpcon = (HttpURLConnection) url_to_fetch.openConnection(); 

httpcon.addRequestProperty("User-Agent", "Mozilla/3.6"); 

BufferedReader in2 = new BufferedReader(new InputStreamReader(httpcon.getInputStream()));



String inputLine;

String Results="";



while ((inputLine = in2.readLine()) != null)

Results=Results+inputLine+"\n";



in2.close();

return Results;