CODEWITHSUNDEEP
proxymarklogic
Gowtham
Gowtham

Reputation: 395

Will xdmp:http-get() support proxies?

xdmp:http-get($url,
    <options xmlns="xdmp:document-get">
        <format>binary</format>
    </options>)[2]

Hi all,

The above query does not return a response from the proxy server. I knew the IP address and the port number to get the response in proxy server. Does anyone know where to add the IP and the port number?

MarkLogic version: 7.x

Recently, I have tried to configure proxy as discussed in http://markmail.org/message/sbfj44jtmpsyopyh with the below code.

let $proxy := "http://171.21.15.60:3128"
let $uri := "http://www.austlii.edu.au/cgi-bin/sinosrch.cgi?results=200;query=damage"
let $host := tokenize($uri,'/')[3] 
let $proxyuri := concat($proxy, '/', tokenize($uri, '/')[last()]) 
return xdmp:http-post(
  $proxyuri,
  <options xmlns="xdmp:http">
    <headers>
      <Host>{$host}</Host>          
    </headers>
  </options>
)

But I got a bad request as a response.

<response xmlns="xdmp:http">
 <code>400</code>
 <message>Bad Request</message>
 <headers>
  <server>squid/3.1.4</server>
  <mime-version>1.0</mime-version>
  <date>Thu, 20 Nov 2014 04:09:50 GMT</date>
  <content-type>text/html</content-type>
  <content-length>3071</content-length>
  <x-squid-error>ERR_INVALID_URL 0</x-squid-error>
  <vary>Accept-Language</vary>
  <content-language>en</content-language>
  <x-cache>MISS from l076ddms1</x-cache>
  <x-cache-lookup>NONE from l076ddms1:3128</x-cache-lookup>
  <via>1.0 l076ddms1 (squid/3.1.4)</via>
  <proxy-connection>close</proxy-connection>
 </headers>
</response>

Look at the below error

The following error was encountered while trying to retrieve the URL: /sinosrch.cgi?results=200;query=damage

Can anyone help me with this issue?

Thanks.

Hi all,

I am still getting the same response after I had tried the steps told by @mblakele.

declare namespace http="xdmp:http";

declare function local:http-options(
  $options as element(http:options)?,
  $extra as element(http:options)?)
as element()?
{
  if (empty($extra)) then $options
  else if (empty($options)) then $extra
  else element http:options {
    (: TODO - needs to handle conflicting options. :)
    $options/*,
    $extra/* }
};

declare function local:http-get(
  $proxy as xs:string,
  $uri as xs:string,
  $options as element(http:options)?)
 as item()+
{

  let $uri-toks := tokenize($uri, '/+')
  let $uri-host := $uri-toks[2]
  let $options := local:http-options(
    $options,
    element http:options {
      element http:headers {
        element http:host { $uri-host } } })
  (: TODO support TLS proxies using https. :)
  let $uri-proxy := concat(
    'http://', $proxy,
    substring-after($uri, $uri-host))
  return xdmp:http-get($uri-proxy, $options)
};

local:http-get(
  '171.21.15.60:3128', 'http://www.austlii.edu.au/cgi-bin/sinosrch.cgi?results=200;query=damage', ())

The value of $uri-proxy of the above code:

http://171.21.15.60:3128/cgi-bin/sinosrch.cgi?results=200;query=damage

The value of $uri-host of the above code is:

www.austlii.edu.au

The value of $options in the above code is:

<http:options xmlns:http="xdmp:http">    
 <http:headers>    
 <http:host>www.austlii.edu.au</http:host>
</http:headers></http:options>

The error is

The following error was encountered while trying to retrieve the URL: /cgi-bin/sinosrch.cgi?results=200;query=damage

Upvotes: 2

Views: 291

Answers (2)

Gowtham
Gowtham

Reputation: 395

As I can't handle proxy with MarkLogic. I developed REST API using .NET to access the external site through proxy tunnel and I made MarkLogic to call my local webservice.

Hope MarkLogic http-get() will support proxy in future.

Thanks dev for your valuable suggestion.

Upvotes: 0

mblakele
mblakele

Reputation: 7842

I don't think there's any direct support, but http://markmail.org/message/sbfj44jtmpsyopyh might help.

[EDIT] Since that code has some problems, here's a simple rewrite. This still isn't completely general-purpose, but it may be easier to debug and enhance.

declare namespace http="xdmp:http" ;

declare function local:http-options(
  $options as element(http:options)?,
  $extra as element(http:options)?)
as element()?
{
  if (empty($extra)) then $options
  else if (empty($options)) then $extra
  else element http:options {
    (: TODO - needs to handle conflicting options. :)
    $options/*,
    $extra/* }
};

declare function local:http-get(
  $proxy as xs:string,
  $uri as xs:string,
  $options as element(http:options)?)
 as item()+
{
  let $_ := (
    if (matches($proxy, '^\w+(:\d+)?$')) then ()
    else error(
      (), 'BADPROXY',
      ('Must be a string host:port', xdmp:describe($proxy))))
  let $uri-toks := tokenize($uri, '/+')
  let $uri-host := $uri-toks[2]
  let $options := local:http-options(
    $options,
    element http:options {
      element http:headers {
        element http:host { $uri-host } } })
  (: TODO support TLS proxies using https. :)
  let $uri-proxy := concat(
    'http://', $proxy,
    substring-after($uri, $uri-host))
  return xdmp:http-get($uri-proxy, $options)
};

local:http-get(
  'localhost:8118', 'http://www.google.com/', ())

Upvotes: 1

Related Questions