Get URL as Text ( url ) and IMDB

[Eden Morris]

I cant get any links from imdb.com to work with the Get URL as Text ( url ) function. I can make it work for other sites but I get a 403 error. any ideas?

[Smef]

The scriptmaster Get URL module is just a basic tool, and may have a problem on some sites. We may take a look at this at some point, but for now I recommend that you check out our Web Assistant Plugin at http://360works.com/url-plugin/ which should work for you.

[Ward]

I encountered the same 403 problem attempting to retrieve HTML from IMDb.com using ScriptMaster.

In the WebAssistant Example.fp7 demo, I modified the "Fetching HTML" example (#6) with an IMDb URL -- the full HTML appeared in the Result field.

[Smef]

That is the correct behavior. You should receive the HTML from the URL you are trying to retrieve. If you don't want all of the HTML tags I recommend using the WAStripTags to remove the HTML tags from the HTML you have retrieved.

[fseipel]

The 403 error occurs because the user agent (browser) has not been set. Some sites will not respond unless the user agent is a browser; i.e. if it is cURL or Java, they won't return a page. When I used to use Troi's URL plugin, I used their HTTP-SETUSERAGENT command.

Setting it to Mozilla or IE will allow it to work, e.g. use this to fetch the IMDB url from SM:

url_to_fetch= new URL(url);

    		

HttpURLConnection httpcon = (HttpURLConnection) url_to_fetch.openConnection(); 

httpcon.addRequestProperty("User-Agent", "Mozilla/3.6"); 

BufferedReader in2 = new BufferedReader(new InputStreamReader(httpcon.getInputStream()));



String inputLine;

String Results="";



while ((inputLine = in2.readLine()) != null)

Results=Results+inputLine+"\n";



in2.close();

return Results;