Server returning 403 for url openStream()

Question

import java.net.URL;
import java.io.*;
import java.net.MalformedURLException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
    public static void main(String args[]) {
        try {
            processHTMLFromLink(new URL("http://fwallpapers.com"));
        } catch (MalformedURLException ex) {
            Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public static int processHTMLFromLink(URL url) {
        InputStream is = null;
        DataInputStream dis;
        String line;
        int count = 0;
        try {
            BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));

            while ((line = in.readLine()) != null) {
                System.out.println(line);
            }
        } catch (MalformedURLException mue) {
            System.out.println(mue.toString());
        } catch (IOException ioe) {
            System.out.println(ioe.toString());
        } finally {
            try {
                is.close();
            } catch (IOException ioe) {
                // nothing to see here
            }
        }
        return count;
    }
}

error:

java.io.IOException: Server returned HTTP response code: 403 for URL: http://fwallpapers.com
Exception in thread "main" java.lang.NullPointerException
    at Test.processHTMLFromLink(Test.java:38)
    at Test.main(Test.java:15)
Java Result: 1

It is working fine on browser. But I am getting null point exceptions. this code works fine with other links. can anyone help me out with this. How can I get content while i am getting 403 error.

Answer 1

This is an old post but if people wanted to know how this works.

a 403 means acces-denied. There is a work around for this. If you want to able to do this you have to set a user agant parameter to 'fool' the website

This is how my old method looked like:

private InputStream read() {
try {
    return url.openStream();
 } 
catch (IOException e) {
  String error = e.toString();
  throw new RuntimeException(e);
 }
}

Changed it to: (And it works for me!)

private InputStream read() {
try {
    HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
    httpcon.addRequestProperty("User-Agent", "Mozilla/4.0");

  return httpcon.getInputStream();
 } catch (IOException e) {
    String error = e.toString();
  throw new RuntimeException(e);
 }
}

Answer 2

Your mistake is swallowing the exception.

When I run my code, I get an HTTP 403 - "forbidden". The web server won't allow you to do this.

My code works perfectly for http://www.yahoo.com.

Here's how I do it:

package url;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;

/**
 * UrlReader
 * @author Michael
 * @since 3/20/11
 */
public class UrlReader {

    public static void main(String[] args) {
        UrlReader urlReader = new UrlReader();

        for (String url : args) {
            try {
                String contents = urlReader.readContents(url);
                System.out.printf("url: %s contents: %s\n", url, contents);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }


    public String readContents(String address) throws IOException {
        StringBuilder contents = new StringBuilder(2048);
        BufferedReader br = null;

        try {
            URL url = new URL(address);
            br = new BufferedReader(new InputStreamReader(url.openStream()));
            String line = "";
            while (line != null) {
                line = br.readLine();
                contents.append(line);
            }
        } finally {
            close(br);
        }

        return contents.toString();
    }

    private static void close(Reader br) {
        try {
            if (br != null) {
                br.close();
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}