Fix connection pool exhaustion from proxy requests (#292)

* Fix connection pool exhaustion from ?uri= proxy requests (#287)

- Add 30 s connectionRequestTimeout to both HTTP client builders in
  Application so pool exhaustion fails fast instead of blocking forever
- Replace allMatch(HTMLMediaTypePredicate) with Request.selectVariant()
  in ProxyRequestFilter so real browser Accept headers (text/html,
  application/xml;q=0.9, */*;q=0.8) correctly trigger the early return,
  leaving (X)HTML responses to the downstream handler and Varnish cache
- In client.xsl ldh:rdf-document-response, detect external ?uri= URIs
  and replace-content on #content-body with bs2:Row rendering of the
  fetched RDF instead of iterating stale home-page blocks

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Server-side condition

* Server-side progress bar

* Fix external URI proxy bypass and client-side rendering

- ProxyRequestFilter: use Core-only MediaTypes (no HTML) with combined
  Model+ResultSet writable variant list; selectVariant==null is the sole
  bypass signal so Accept:*/* correctly reaches the proxy instead of
  falling through to the HTML handler
- Thread pre-computed Variant through all getResponse() overloads to
  avoid a second selectVariant call inside Core's Response constructor
- client.xsl onsubmit: skip the XHTML round-trip for external URIs and
  call PushState + RDFDocumentLoad directly, advancing the progress bar
  to 66% between the two steps; fixes the double-click issue
- client.xsl ldh:rdf-document-response: respect the #layout-modes mode
  selector for client-side rendered external resources; refactor the
  duplicate id('content-body') lookup out of both xsl:choose branches
- ProxyRequestFilterTest: stub Request.selectVariant() to return a
  non-null Variant so both tests reach the logic they exercise

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* ProxyRequestFilter: document HTML bypass rationale; cache MediaTypes instance

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* ProxyRequestFilter: clarify HTML bypass as resource exhaustion defence

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Make HTTP client connectionRequestTimeout configurable

Defaults to 30000 ms (via Dockerfile ENV). Passed through the
CATALINA_OPTS path (same as allowInternalUrls) to avoid exceeding
the ~30-param libxslt limit already reached by context.xsl.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix ProxyRequestFilter HTML bypass: check Accept header explicitly

Replace the selectVariant==null bypass with an explicit check for
non-wildcard text/html or application/xhtml+xml in the Accept header.
Browsers list these types explicitly (q=1.0) and get bypassed to the
app shell; API clients that send only */* reach the proxy.

The old approach (Core MediaTypes, selectVariant==null) failed for
browsers because their */*;q=0.8 wildcard matched RDF variants,
causing the proxy to return RDF instead of the (X)HTML app shell.

Add testHtmlAcceptBypassesProxy to cover the bypass path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix ContentMode block rendering for proxied external resources

Proxied resources' ContentMode blocks (charts, maps) were querying the
local SPARQL endpoint instead of the remote one because ProxyRequestFilter
discarded all external response headers and ResponseHeadersFilter then
injected the local sd:endpoint Link.

- ApplicationFilter: register external ?uri= target in request context
  (AC.uri property) as authoritative proxy marker
- ProxyRequestFilter: forward all Link headers from external response
- ResponseHeadersFilter: skip local sd:endpoint/ldt:ontology/ac:stylesheet
  for proxy requests; removes now-unused parseLinkHeaderValues/getLinksByRel
- client.xsl (ldh:rdf-document-response): extract sd:endpoint from Link
  header and store in LinkedDataHub.endpoint, mirroring acl:mode pattern
- functions.xsl (sd:endpoint()): return LinkedDataHub.endpoint when set,
  fall back to local /sparql — no changes needed in view.xsl or chart.xsl
- CLAUDE.md: document the proxy/client-side rendering architecture

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Replace proxy-detection heuristics with ac:uri() / $ac:uri throughout

- ApplicationFilter: store external URI as AC.uri context property; strip ?uri= from UriInfo
- ProxyRequestFilter: read proxy target from AC.uri context property; bypass HTML requests
- XsltExecutableFilter: remove SYSTEM_ID_PROPERTY; XSLTWriterBase reads AC.uri directly
- XSLTWriterBase: pass $ac:uri to server-side XSLT when proxying
- layout.xsl: declare $ac:uri param; use it for export links and search input pre-fill
- document.xsl: remove proxy spinner branch from bs2:ContentBody
- client/functions.xsl: add ac:uri() function (dynamic read of ixsl:query-params()?uri);
  ldh:base-uri() now calls ac:uri() instead of stale global $ac:uri
- client.xsl: drop global $ac:uri param; ldh:HTMLDocumentLoaded passes ldh:base-uri(.)
  to ldh:RDFDocumentLoad after pushState so URL is already updated
- ProxyRequestFilterTest: update mocks to use AC.uri context property

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add server-side ac:uri() function; refactor ActionBar templates into document.xsl

- Add ac:uri() server-side function to imports/default.xsl (mirrors acl:mode() pattern)
- Move ActionBarLeft/ActionBarMain/ActionBarRight/BreadCrumbBar/ModeList/MediaTypeList templates from layout.xsl to document.xsl
- Fix $effective-mode type error (xs:string → xs:anyURI) and simplify with [1] idiom
- Use ac:uri() instead of $ac:uri in MediaTypeList hrefs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: namedgraph

CLAUDE.md

@@ -91,8 +91,28 @@ The application runs as a multi-container setup:
 1. Requests come through nginx proxy
 2. Varnish provides caching layer
 3. LinkedDataHub application handles business logic
-4. Data persisted to appropriate Fuseki triplestore
-5. XSLT transforms data for client presentation
+4. RDF data is read/written via the **Graph Store Protocol** — each document in the hierarchy corresponds to a named graph in the triplestore; the document URI is the graph name
+5. Data persisted to appropriate Fuseki triplestore
+6. XSLT transforms data for client presentation
+
+### Linked Data Proxy and Client-Side Rendering
+
+LDH includes a Linked Data proxy that dereferences external URIs on behalf of the browser. The original design rendered proxied resources identically to local ones — server-side RDF fetch + XSLT. This created a DDoS/resource-exhaustion vector: scraper bots routing arbitrary external URIs through the proxy would trigger a full server-side pipeline (HTTP fetch → XSLT rendering) per request, exhausting HTTP connection pools and CPU.
+
+The current design splits rendering by request origin:
+
+- **Browser requests** (`Accept: text/html`): `ProxyRequestFilter` bypasses the proxy entirely. The server returns the local application shell. Saxon-JS then issues a second, RDF-typed request (`Accept: application/rdf+xml`) from the browser.
+- **RDF requests** (API clients, Saxon-JS second pass): `ProxyRequestFilter` fetches the external RDF, parses it, and returns it to the caller. No XSLT happens server-side.
+- **Client-side rendering**: Saxon-JS receives the raw RDF and applies the same XSLT 3 templates used server-side (shared stylesheet), so proxied resources look almost identical to local ones.
+
+Key implementation files:
+- `ProxyRequestFilter.java` — intercepts `?uri=` and `lapp:Dataset` proxy requests; HTML bypass; forwards external `Link` headers
+- `ApplicationFilter.java` — registers external proxy target URI in request context (`AC.uri` property) as authoritative proxy marker
+- `ResponseHeadersFilter.java` — skips local-only hypermedia links (`sd:endpoint`, `ldt:ontology`, `ac:stylesheet`) for proxy requests; external ones are forwarded by `ProxyRequestFilter`
+- `client.xsl` (`ldh:rdf-document-response`) — receives the RDF proxy response client-side; extracts `sd:endpoint` from `Link` header; stores it in `LinkedDataHub.endpoint`
+- `functions.xsl` (`sd:endpoint()`) — returns `LinkedDataHub.endpoint` when set (external proxy), otherwise falls back to the local SPARQL endpoint
+
+The SPARQL endpoint forwarding chain ensures ContentMode blocks (charts, maps) query the **remote** app's SPARQL endpoint, not the local one. `LinkedDataHub.endpoint` is reset to the local endpoint by `ldh:HTMLDocumentLoaded` on every HTML page navigation, so there is no stale state when navigating back to local documents.
 
 ### Key Extension Points
 - **Vocabulary definitions** in `com.atomgraph.linkeddatahub.vocabulary`

Dockerfile

@@ -109,6 +109,8 @@ ENV MAX_TOTAL_CONN=40
 
 ENV MAX_REQUEST_RETRIES=3
 
+ENV CONNECTION_REQUEST_TIMEOUT=30000
+
 ENV IMPORT_KEEPALIVE=
 
 ENV MAX_IMPORT_THREADS=10

docker-compose.yml

@@ -65,6 +65,7 @@ services:
       - SIGN_UP_CERT_VALIDITY=180
       - MAX_CONTENT_LENGTH=${MAX_CONTENT_LENGTH:-2097152}
       - ALLOW_INTERNAL_URLS=${ALLOW_INTERNAL_URLS:-}
+      - CONNECTION_REQUEST_TIMEOUT=${CONNECTION_REQUEST_TIMEOUT:-}
       - NOTIFICATION_ADDRESS=LinkedDataHub <notifications@localhost>
       - MAIL_SMTP_HOST=email-server
       - MAIL_SMTP_PORT=25

platform/entrypoint.sh

@@ -1037,6 +1037,10 @@ if [ -n "$ALLOW_INTERNAL_URLS" ]; then
     export CATALINA_OPTS="$CATALINA_OPTS -Dcom.atomgraph.linkeddatahub.allowInternalUrls=$ALLOW_INTERNAL_URLS"
 fi
 
+if [ -n "$CONNECTION_REQUEST_TIMEOUT" ]; then
+    export CATALINA_OPTS="$CATALINA_OPTS -Dcom.atomgraph.linkeddatahub.connectionRequestTimeout=$CONNECTION_REQUEST_TIMEOUT"
+fi
+
 if [ -n "$MAX_CONTENT_LENGTH" ]; then
     MAX_CONTENT_LENGTH_PARAM="--stringparam ldhc:maxContentLength '$MAX_CONTENT_LENGTH' "
 fi

src/main/java/com/atomgraph/linkeddatahub/Application.java

@@ -214,6 +214,7 @@
 import org.apache.http.HttpClientConnection;
 import org.apache.http.HttpHost;
 import org.apache.http.client.HttpRequestRetryHandler;
+import org.apache.http.client.config.RequestConfig;
 import org.apache.http.config.Registry;
 import org.apache.http.config.RegistryBuilder;
 import org.apache.http.conn.socket.ConnectionSocketFactory;
@@ -358,6 +359,8 @@ public Application(@Context ServletConfig servletConfig) throws URISyntaxExcepti
             servletConfig.getServletContext().getInitParameter(LDHC.maxConnPerRoute.getURI()) != null ? Integer.valueOf(servletConfig.getServletContext().getInitParameter(LDHC.maxConnPerRoute.getURI())) : null,
             servletConfig.getServletContext().getInitParameter(LDHC.maxTotalConn.getURI()) != null ? Integer.valueOf(servletConfig.getServletContext().getInitParameter(LDHC.maxTotalConn.getURI())) : null,
             servletConfig.getServletContext().getInitParameter(LDHC.maxRequestRetries.getURI()) != null ? Integer.valueOf(servletConfig.getServletContext().getInitParameter(LDHC.maxRequestRetries.getURI())) : null,
+            System.getProperty("com.atomgraph.linkeddatahub.connectionRequestTimeout") != null ? Integer.valueOf(System.getProperty("com.atomgraph.linkeddatahub.connectionRequestTimeout")) :
+            servletConfig.getServletContext().getInitParameter(LDHC.connectionRequestTimeout.getURI()) != null ? Integer.valueOf(servletConfig.getServletContext().getInitParameter(LDHC.connectionRequestTimeout.getURI())) : null,
             servletConfig.getServletContext().getInitParameter(LDHC.maxImportThreads.getURI()) != null ? Integer.valueOf(servletConfig.getServletContext().getInitParameter(LDHC.maxImportThreads.getURI())) : null,
             servletConfig.getServletContext().getInitParameter(LDHC.notificationAddress.getURI()) != null ? servletConfig.getServletContext().getInitParameter(LDHC.notificationAddress.getURI()) : null,
             servletConfig.getServletContext().getInitParameter(LDHC.supportedLanguages.getURI()) != null ? servletConfig.getServletContext().getInitParameter(LDHC.supportedLanguages.getURI()) : null,
@@ -445,7 +448,7 @@ public Application(final ServletConfig servletConfig, final MediaTypes mediaType
             final String baseURIString, final String proxyScheme, final String proxyHostname, final Integer proxyPort,
             final String uploadRootString, final boolean invalidateCache,
             final Integer cookieMaxAge, final boolean enableLinkedDataProxy, final boolean allowInternalUrls, final Integer maxContentLength,
-            final Integer maxConnPerRoute, final Integer maxTotalConn, final Integer maxRequestRetries, final Integer maxImportThreads,
+            final Integer maxConnPerRoute, final Integer maxTotalConn, final Integer maxRequestRetries, final Integer connectionRequestTimeout, final Integer maxImportThreads,
             final String notificationAddressString, final String supportedLanguageCodes, final boolean enableWebIDSignUp, final String oidcRefreshTokensPropertiesPath,
             final String frontendProxyString, final String backendProxyAdminString, final String backendProxyEndUserString,
             final String mailUser, final String mailPassword, final String smtpHost, final String smtpPort,
@@ -709,10 +712,10 @@ public Application(final ServletConfig servletConfig, final MediaTypes mediaType
                 trustStore.load(trustStoreInputStream, clientTrustStorePassword.toCharArray());
             }
 
-            client = getClient(keyStore, clientKeyStorePassword, trustStore, maxConnPerRoute, maxTotalConn, null, false);
-            externalClient = getClient(keyStore, clientKeyStorePassword, trustStore, maxConnPerRoute, maxTotalConn, null, false);
-            importClient = getClient(keyStore, clientKeyStorePassword, trustStore, maxConnPerRoute, maxTotalConn, maxRequestRetries, true);
-            noCertClient = getNoCertClient(trustStore, maxConnPerRoute, maxTotalConn, maxRequestRetries);
+            client = getClient(keyStore, clientKeyStorePassword, trustStore, maxConnPerRoute, maxTotalConn, null, false, connectionRequestTimeout);
+            externalClient = getClient(keyStore, clientKeyStorePassword, trustStore, maxConnPerRoute, maxTotalConn, null, false, connectionRequestTimeout);
+            importClient = getClient(keyStore, clientKeyStorePassword, trustStore, maxConnPerRoute, maxTotalConn, maxRequestRetries, true, connectionRequestTimeout);
+            noCertClient = getNoCertClient(trustStore, maxConnPerRoute, maxTotalConn, maxRequestRetries, connectionRequestTimeout);
 
             if (maxContentLength != null)
             {
@@ -1527,7 +1530,7 @@ public void submitImport(RDFImport rdfImport, com.atomgraph.linkeddatahub.apps.m
      * @throws UnrecoverableKeyException key loading error
      * @throws KeyManagementException key loading error
      */
-    public static Client getClient(KeyStore keyStore, String keyStorePassword, KeyStore trustStore, Integer maxConnPerRoute, Integer maxTotalConn, Integer maxRequestRetries, boolean buffered) throws NoSuchAlgorithmException, KeyStoreException, UnrecoverableKeyException, KeyManagementException
+    public static Client getClient(KeyStore keyStore, String keyStorePassword, KeyStore trustStore, Integer maxConnPerRoute, Integer maxTotalConn, Integer maxRequestRetries, boolean buffered, Integer connectionRequestTimeout) throws NoSuchAlgorithmException, KeyStoreException, UnrecoverableKeyException, KeyManagementException
     {
         if (keyStore == null) throw new IllegalArgumentException("KeyStore cannot be null");
         if (keyStorePassword == null) throw new IllegalArgumentException("KeyStore password string cannot be null");
@@ -1592,7 +1595,11 @@ public void releaseConnection(final HttpClientConnection managedConn, final Obje
         config.property(ClientProperties.FOLLOW_REDIRECTS, true);
         config.property(ClientProperties.REQUEST_ENTITY_PROCESSING, RequestEntityProcessing.BUFFERED); // https://stackoverflow.com/questions/42139436/jersey-client-throws-cannot-retry-request-with-a-non-repeatable-request-entity
         config.property(ApacheClientProperties.CONNECTION_MANAGER, conman);
-        
+        if (connectionRequestTimeout != null)
+            config.property(ApacheClientProperties.REQUEST_CONFIG, RequestConfig.custom().
+                setConnectionRequestTimeout(connectionRequestTimeout).
+                build());
+
         if (maxRequestRetries != null)
             config.property(ApacheClientProperties.RETRY_HANDLER, (HttpRequestRetryHandler) (IOException ex, int executionCount, HttpContext context) ->
             {
@@ -1629,7 +1636,7 @@ public void releaseConnection(final HttpClientConnection managedConn, final Obje
      * @param maxRequestRetries maximum number of times that the HTTP client will retry a request
      * @return client instance
      */
-    public static Client getNoCertClient(KeyStore trustStore, Integer maxConnPerRoute, Integer maxTotalConn, Integer maxRequestRetries)
+    public static Client getNoCertClient(KeyStore trustStore, Integer maxConnPerRoute, Integer maxTotalConn, Integer maxRequestRetries, Integer connectionRequestTimeout)
     {
         try
         {
@@ -1688,7 +1695,11 @@ public void releaseConnection(final HttpClientConnection managedConn, final Obje
             config.property(ClientProperties.FOLLOW_REDIRECTS, true);
             config.property(ClientProperties.REQUEST_ENTITY_PROCESSING, RequestEntityProcessing.BUFFERED); // https://stackoverflow.com/questions/42139436/jersey-client-throws-cannot-retry-request-with-a-non-repeatable-request-entity
             config.property(ApacheClientProperties.CONNECTION_MANAGER, conman);
-            
+            if (connectionRequestTimeout != null)
+                config.property(ApacheClientProperties.REQUEST_CONFIG, RequestConfig.custom().
+                    setConnectionRequestTimeout(connectionRequestTimeout).
+                    build());
+
             if (maxRequestRetries != null)
                 config.property(ApacheClientProperties.RETRY_HANDLER, (HttpRequestRetryHandler) (IOException ex, int executionCount, HttpContext context) ->
                 {
@@ -1708,7 +1719,7 @@ public void releaseConnection(final HttpClientConnection managedConn, final Obje
                     }
                     return false;
                 });
-        
+
             return ClientBuilder.newBuilder().
                 withConfig(config).
                 sslContext(ctx).

src/main/java/com/atomgraph/linkeddatahub/server/filter/request/ApplicationFilter.java

@@ -107,7 +107,23 @@ public void filter(ContainerRequestContext request) throws IOException
 
                     requestURI = builder.build();
                 }
-                else requestURI = request.getUriInfo().getRequestUri();
+                else
+                {
+                    request.setProperty(AC.uri.getURI(), graphURI); // authoritative external proxy marker
+
+                    // strip ?uri= from the effective request URI — server-side sees only the path;
+                    // the ContainerRequestContext property is the sole indicator of proxy mode
+                    MultivaluedMap<String, String> externalQueryParams = new MultivaluedHashMap();
+                    externalQueryParams.putAll(request.getUriInfo().getQueryParameters());
+                    externalQueryParams.remove(AC.uri.getLocalName());
+
+                    UriBuilder externalBuilder = UriBuilder.fromUri(request.getUriInfo().getAbsolutePath());
+                    for (Entry<String, List<String>> params : externalQueryParams.entrySet())
+                        for (String value : params.getValue())
+                            externalBuilder.queryParam(params.getKey(), value);
+
+                    requestURI = externalBuilder.build();
+                }
             }
             catch (URISyntaxException ex)
             {