Re: Use of http proxy?

Hi Nick:

DConnect2 is my (NetCDF-Java library) version of the OPenDAP Java client HTTP 
handler, that uses http commons library, so we should work together to make 
this work.

It sounds like we need to add proxy support by configuring the http commons 
library. I havent used an http proxy before, so I will need your help. If you 
have a clear idea of how to proceed, let me know.

John

Nick Bower wrote:
Bummer. Found the real exception and the source - DODS libraries are using the commons http client, which requires explicit proxy setting at an application level (ie proxies must be a supported feature of DAP). But even when skirting DNS availability using hosts file, we go from this, 

java.net.UnknownHostException: www.marine.csiro.au
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:177)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
       at java.net.Socket.connect(Socket.java:519)
       at java.net.Socket.connect(Socket.java:469)
       at java.net.Socket.<init>(Socket.java:366)
       at java.net.Socket.<init>(Socket.java:239)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:79) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:121) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1321) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324) 
       at opendap.dap.DConnect2.openConnection(DConnect2.java:218)

to this,
C:\Program Files\IDV_2.2>jre\bin\java -Dhttp.proxyHost=proxyhost -Dhttp.proxyPor 
t=3128 -Xmx512m -jar idv.jar
25/05/2007 14:52:15 org.apache.commons.httpclient.HttpMethodDirector executeWith 
Retry
INFO: I/O exception (java.net.ConnectException) caught when processing request: 
Connection timed out: connect
25/05/2007 14:52:15 org.apache.commons.httpclient.HttpMethodDirector executeWith 
Retry
INFO: Retrying request
25/05/2007 14:53:00 org.apache.commons.httpclient.HttpMethodDirector executeWith 
Retry
INFO: I/O exception (java.net.ConnectException) caught when processing request: 
Connection timed out: connect
Looking at the source below, it would indeed seem DAP does not support proxies. This is hard to believe - it's more effective than any EULA for curtailing commercial use. I'll take my grumbles to the DAP group I guess... :( 

opendap.dap.DConnect2.java:
private void openConnection(String urlString, Command command) throws IOException, DAP2Exception, ParseException { 

   initHttpClient();
   GetMethod method = new GetMethod(urlString);
   method.setFollowRedirects(true);

   if (acceptCompress)
method.setRequestHeader(new Header("Accept-Encoding", "deflate,gzip")); 

   // enable sessions
   if (allowSessions)
     method.setRequestHeader(new Header("X-Accept-Session", "true"));

   InputStream is = null;
   try {
     // Execute the method.

     int statusCode = _httpClient.executeMethod(method);
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^





Nick Bower wrote:
IDV seems to be having trouble with our http proxy (squid). I've set the following global parameters in runIDV.bat in the usual way; 

-Dhttp.proxyHost=proxyhost -Dhttp.proxyPort=3128
however IDV fails to retrieve any data in the Data Chooser. See traceback below. Could it have something to do with IDV insisting DODS URI's begin with dods:// perhaps? What is the recommended solution for using a http proxy?
By the way, I can navigate the server in the "Display" / web browsing tab. It's only the Data Chooser that fails. 

Thanks, Nick

An error has occurred:
There was an error loading the data:
Error creating data source:dods.grid with: http://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ Grid data source failed making data set: dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ Cant open dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ or as DODS dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ 
www.marine.csiro.au
java.io.FileNotFoundException: Cant open dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ or as DODS dods://www.marine.csiro.au/dods/nph-dods/dods-data/bl/SPINUP4/temp/ 
www.marine.csiro.au
   at ucar.nc2.dataset.NetcdfDataset.openDODS(NetcdfDataset.java:380)
   at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:316)
   at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:175)
at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:203) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:183) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:171) 
   at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:71)
at ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:726) at ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:743) at ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:335) at ucar.unidata.data.DataManager.createDataSource(DataManager.java:1186) at ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1698) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1617) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1595) at ucar.unidata.idv.IntegratedDataViewer.handleAction(IntegratedDataViewer.java:1446) 
   at ucar.unidata.idv.DefaultIdv.handleAction(DefaultIdv.java:112)
at ucar.unidata.idv.IntegratedDataViewer.handleAction(IntegratedDataViewer.java:1378) at ucar.unidata.idv.chooser.UrlChooser.loadURLInner(UrlChooser.java:250) 
   at ucar.unidata.idv.chooser.UrlChooser.loadURL(UrlChooser.java:222)
at ucar.unidata.idv.chooser.UrlChooser.doLoadInThread(UrlChooser.java:269) 
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
   at java.lang.reflect.Method.invoke(Method.java:585)
   at ucar.unidata.util.Misc$2.run(Misc.java:919)
   at ucar.unidata.util.Misc$3.run(Misc.java:947)


And another example from your own servers...
java.io.FileNotFoundException: Cant open dods://motherlode.ucar.edu:8080/thredds/dodsC/modelsNc/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20070524_1200.nc or as DODS dods://motherlode.ucar.edu:8080/thredds/dodsC/modelsNc/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_20070524_1200.nc 
motherlode.ucar.edu
   at ucar.nc2.dataset.NetcdfDataset.openDODS(NetcdfDataset.java:380)
   at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:316)
   at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:175)
at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:203) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:183) at ucar.nc2.dataset.NetcdfDatasetCache.acquire(NetcdfDatasetCache.java:171) 
   at ucar.nc2.dt.grid.GridDataset.open(GridDataset.java:71)
at ucar.unidata.data.grid.GeoGridDataSource.doMakeDataSet(GeoGridDataSource.java:726) at ucar.unidata.data.grid.GeoGridDataSource.getDataset(GeoGridDataSource.java:743) at ucar.unidata.data.grid.GeoGridDataSource.initAfterCreation(GeoGridDataSource.java:335) at ucar.unidata.data.DataManager.createDataSource(DataManager.java:1186) at ucar.unidata.idv.IntegratedDataViewer.createDataSource(IntegratedDataViewer.java:1698) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1617) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1595) at ucar.unidata.idv.IntegratedDataViewer.makeDataSource(IntegratedDataViewer.java:1576) at ucar.unidata.idv.chooser.IdvChooser.makeDataSource(IdvChooser.java:610) at ucar.unidata.idv.chooser.ThreddsHandler.processNodes(ThreddsHandler.java:842) at ucar.unidata.idv.chooser.ThreddsHandler.processNode(ThreddsHandler.java:782) 
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
   at java.lang.reflect.Method.invoke(Method.java:585)
   at ucar.unidata.util.Misc$2.run(Misc.java:921)
   at ucar.unidata.util.Misc$3.run(Misc.java:947)