Java URL java.net.ConnectException Error (Ping works, and other URLs work)

Question

I am writing a program that reads some IDs from a list, figures out different URLs from them, and saves the images to my C: drive.

The image URLs work if I navigate to them in my browser. Additionally, if I try URLs to images from a different server this program works entirely. The issue is when I try to connect to specific images at this URL it does not seem to work. The strange thing is when I ping the URL I get 0% packet loss. Additionally, my browser is not using special proxy settings that would cause it to work over the Java program.

What could be the cause of the below error/output?

Job started ...
1) AC_0A47_EXT1.jpg 2/0/47/307/398/HA47H01_H.jpg
Exception #3 java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.<init>(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at pullVFMImagesFromVFMURLs.writeImage(pullVFMImagesFromVFMURLs.java:99)
at pullVFMImagesFromVFMURLs.pullvfmimagesfromvfmurls(pullVFMImagesFromVFMURLs.java:48)
at pullVFMImagesFromVFMURLs.main(pullVFMImagesFromVFMURLs.java:21)

More info:

The ID list (MissingImagez.txt) is as follows:

AC_0A47_EXT1.jpg|2/0/47/307/398/xxx.jpg
AC_0C09_EXT1.jpg|3/0/44/130/589/xyz.jpg
AC_0C16_RM1.jpg|3/0/44/602/895/zzz.jpg
AC_0C17_BAR1.jpg|3/0/45/284/10/www.jpg

and the corresponding code is as follows:

  public static void main(String[] args)
  {
    pullImagesFromURLs processor = new pullImagesFromURLs();

    String passParam = null;

    for (int i = 0; i < args.length; i++)
       passParam = (args[i].toString().toUpperCase());

    processor.pullvfmimagesfromvfmurls(passParam);
  }

  public void pullvfmimagesfromvfmurls(String passParam)
  {
    System.out.println("Job started ...");

    try
    {     
       boolean bOK = false;
       String sOrigName = "";
       String sURL = ""; 
       int iCount = 0;
       int iInt = 0;

       try
       {
          BufferedReader in = new BufferedReader(new FileReader("/tmp/missingImagez.txt"));
          String str;

          while ((str = in.readLine()) != null)
          {       
             iInt = str.indexOf("|");     
             sOrigName = str.substring(0,iInt);
             sURL = str.substring(iInt+1);
             iCount ++;
             System.out.println(iCount + ") " + sOrigName + " " + sURL);
             bOK = writeImage(sOrigName, sURL);         
          }           

          try
          {
             if (in != null)
                in.close();             
          }      
          catch (Exception e)
          {
          }

       }
       catch (IOException e)
       {
         System.out.println("Exception #1 " + e);
       }
    }

    catch (Exception e)
    {
       System.out.println("Exception #2 " + e);
       System.exit(-666);
    }

    System.out.println("Job completed");

    System.exit(-666);

  } 
 private static boolean writeImage(String origName, String vfmURL) 
  {

    boolean bOK = true;
    String sPath = "C:/images/img2754/";

    try
    {
       final String vfmURLPrefix = new String("http://www.blah.blah2.com/imageRepo/");

       BufferedInputStream stream = null; 
       FileOutputStream destination = null;
       URLConnection urlc = null;
       int readSize = 1024 * 50;
       int bytes_read;
       URL url = null;

        url = new URL(vfmURLPrefix + vfmURL);
        urlc = url.openConnection();
        urlc.setDoOutput(true);
        stream = new BufferedInputStream(url.openStream());
        System.out.println(url.toString());
        destination = new FileOutputStream(sPath+origName);
        byte[] buffer = new byte[readSize];

        while(true)
        {       

           bytes_read = stream.read(buffer);
           if (bytes_read == -1) {break;}
             destination.write(buffer, 0, bytes_read);
             stream.close(); 
        }

        if(stream != null)
            stream.close();

        if(destination != null) 
           destination.close();

    }

    catch(Exception e)
    {
       System.out.println("Exception #3 " + e);
    }
    return bOK;

  }

Tried using Apache HttpComponents. Get the following output: Okay -- I tried that...now I get:

Exception #3     org.apache.http.conn.HttpHostConnectException: Connection to http://www.blah.blah2.com/ refused. 
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at pullVFMImagesFromVFMURLs.writeImage(pullVFMImagesFromVFMURLs.java:100)
        at pullVFMImagesFromVFMURLs.pullvfmimagesfromvfmurls(pullVFMImagesFromVFMURLs.java:55)
        at pullVFMImagesFromVFMURLs.main(pullVFMImagesFromVFMURLs.java:28)
    Caused by: java.net.ConnectException: Connection timed out: connect
        at java.net.DualStackPlainSocketImpl.connect0(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.PlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 9 more

Answer 1

EDIT 2

Your code has an error in the read loop. InputStream.read does not necessary return the entire byte stream being received. (It might not all be available yet.) The way you have coded it, the input stream is going to often be prematurely closed. That results in a different error ("stream closed") than what you have posted though.

In any case, after fixing your loop, I was able to run your code to download this image without any problems.

Original

As an aside, your code is a kind-of confusing. In particular, your code creates a URLConnection, but opens the stream directly from the URL. The URLConnection is basically unused. It's also confusing because you call setDoOutput, but not setDoInput. I would rewrite your code more like this:

url = new URL(vfmURLPrefix + vfmURL);
urlc = url.openConnection();
stream = new BufferedInputStream(urlc.getInputStream()); // connect/request will be made.

I doubt that will solve your current problem though. To debug your current problem, I would recommend using Firebug or Chrome's Developer Tools to sniff the network connection/request/response when you request the same image URL in your browser. You might find unexpected redirects, status codes, cookies, or similar. With that extra knowledge in hand, you then might try casting the connection to an HttpURLConnection so that you have more options available for customizing the request.

Answer 2

When its fine with your web browser and a problem with your application, try adding user-agent to the java request, as follows:

request.setHeader("User-Agent","Mozilla/3.0");

Answer 3

There can be a few problems: 1. You must be logged in. 2. You have to have a Cookie to get the Images 3. The request of URL() does not match a browser-request. ...

Try to simulate a real browser using the

  Apache HttpComponents

Regards