Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] SSLConfigurationReloaderTests testReloadingKeyStore failing #108774

Open
cbuescher opened this issue May 17, 2024 · 4 comments
Open

[CI] SSLConfigurationReloaderTests testReloadingKeyStore failing #108774

cbuescher opened this issue May 17, 2024 · 4 comments
Assignees
Labels
needs:risk Requires assignment of a risk label (low, medium, blocker) :Security/Security Security issues without another label Team:Security Meta label for security team >test-failure Triaged test failures from CI

Comments

@cbuescher
Copy link
Member

Seems to be related to JDK23 somehow, although I don't see any direct indication in the logs and the failure.

Build scan:
https://gradle-enterprise.elastic.co/s/ml2fwj3ywiafc/tests/:x-pack:plugin:core:test/org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests/testReloadingKeyStore

Reproduction line:

./gradlew ':x-pack:plugin:core:test' --tests "org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.testReloadingKeyStore" -Dtests.seed=BAFBC3F9A8DA6D20 -Dtests.locale=da -Dtests.timezone=Indian/Comoro -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
Yes

Failure history:
Failure dashboard for org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests#testReloadingKeyStore

Failure excerpt:

java.lang.RuntimeException: Exception starting or connecting to the mock server

  at __randomizedtesting.SeedInfo.seed([BAFBC3F9A8DA6D20:D3864BF8438FE7D3]:0)
  at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.lambda$testReloadingKeyStore$1(SSLConfigurationReloaderTests.java:138)
  at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.validateSSLConfigurationIsReloaded(SSLConfigurationReloaderTests.java:577)
  at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.testReloadingKeyStore(SSLConfigurationReloaderTests.java:164)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1575)

  Caused by: javax.net.ssl.SSLHandshakeException: (certificate_required) Received fatal alert: certificate_required

    at sun.security.ssl.Alert.createSSLException(Alert.java:130)
    at sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:365)
    at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287)
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
    at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1481)
    at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests$1.receiveResponseHeader(SSLConfigurationReloaderTests.java:714)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.lambda$testReloadingKeyStore$0(SSLConfigurationReloaderTests.java:136)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.lambda$privilegedConnect$31(SSLConfigurationReloaderTests.java:791)
    at java.security.AccessController.doPrivileged(AccessController.java:571)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.privilegedConnect(SSLConfigurationReloaderTests.java:790)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.lambda$testReloadingKeyStore$1(SSLConfigurationReloaderTests.java:136)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.validateSSLConfigurationIsReloaded(SSLConfigurationReloaderTests.java:577)
    at org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.testReloadingKeyStore(SSLConfigurationReloaderTests.java:164)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke(Method.java:580)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1575)

@cbuescher cbuescher added :Security/Security Security issues without another label >test-failure Triaged test failures from CI labels May 17, 2024
cbuescher added a commit that referenced this issue May 17, 2024
@elasticsearchmachine elasticsearchmachine added Team:Security Meta label for security team needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 17, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

parkertimmins pushed a commit to parkertimmins/elasticsearch that referenced this issue May 17, 2024
@jakelandis jakelandis self-assigned this May 21, 2024
@jakelandis
Copy link
Contributor

I can reliable reproduce with Java 23, but not Java 22

Java.net | | 23.ea.23 | open | installed | 23.ea.23-open

To avoid some misleading warnings, rule out the security manager, and provide better logging here is a better reproduction line:

./gradlew ':x-pack:plugin:core:test' --tests "org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.testReloadingKeyStore" -Dtests.seed=BAFBC3F9A8DA6D20 -Dtests.locale=da -Dtests.timezone=Indian/Comoro -Druntime.java=23 -Dtests.security.manager=false -Dtests.jvm.argline="-Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false --enable-native-access=ALL-UNNAMED" -Dtests.es.logger.org.apache.http=debug -Dtests.es.logger.level=DEBUG

fails with Java 23, but works with Java 22.

Java 22 (works)

1> [2024-05-22T01:01:34,116][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] [testReloadingKeyStore] Connection established org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests$1@371b6acc
  1> [2024-05-22T01:01:34,116][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Executing request GET / HTTP/1.1
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Target auth state: UNCHALLENGED
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Proxy auth state: UNCHALLENGED
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> GET / HTTP/1.1
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> Host: localhost:52242
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> Connection: Keep-Alive
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.14 (Java/22)
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> Accept-Encoding: gzip,deflate
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "GET / HTTP/1.1[\r][\n]"
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Host: localhost:52242[\r][\n]"
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/22)[\r][\n]"
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
  1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "[\r][\n]"
  1> [2024-05-22T01:01:34,120][DEBUG][o.e.t.h.MockWebServer    ] [[HTTP-Dispatcher]] [127.0.0.1:52242] incoming HTTP request [GET /], returning status [200] body [body]
  1> [2024-05-22T01:01:34,125][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
  1> [2024-05-22T01:01:34,126][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 << "Date: Tue, 21 May 2024 22:01:34 GMT[\r][\n]"
  1> [2024-05-22T01:01:34,126][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 << "Content-length: 4[\r][\n]"
  1> [2024-05-22T01:01:34,126][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 << "[\r][\n]"
  1> [2024-05-22T01:01:34,129][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 << HTTP/1.1 200 OK
  1> [2024-05-22T01:01:34,129][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 << Date: Tue, 21 May 2024 22:01:34 GMT
  1> [2024-05-22T01:01:34,129][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 << Content-length: 4
  1> [2024-05-22T01:01:34,130][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Connection can be kept alive indefinitely
  1> [2024-05-22T01:01:34,132][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [testReloadingKeyStore] http-outgoing-0: Close connection

Java 23 (fails)

  1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] [testReloadingKeyStore] Connection established org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests$1@ade8676
  1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Executing request GET / HTTP/1.1
  1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Target auth state: UNCHALLENGED
  1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Proxy auth state: UNCHALLENGED
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> GET / HTTP/1.1
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> Host: localhost:52320
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> Connection: Keep-Alive
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.14 (Java/23-ea)
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers            ] [testReloadingKeyStore] http-outgoing-0 >> Accept-Encoding: gzip,deflate
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "GET / HTTP/1.1[\r][\n]"
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Host: localhost:52320[\r][\n]"
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/23-ea)[\r][\n]"
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "[\r][\n]"
  1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "[write] I/O error: Broken pipe"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [testReloadingKeyStore] http-outgoing-0: Close connection
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "GET / HTTP/1.1[\r][\n]"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Host: localhost:52320[\r][\n]"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/23-ea)[\r][\n]"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "[\r][\n]"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire               ] [testReloadingKeyStore] http-outgoing-0 >> "[write] I/O error: Connection or outbound has closed"
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [testReloadingKeyStore] http-outgoing-0: Shutdown connection
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Connection discarded
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [testReloadingKeyStore] Connection released: [id: 0][route: {s}->https://localhost:52320][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
  1> [2024-05-22T01:04:18,098][INFO ][o.a.h.i.e.RetryExec      ] [testReloadingKeyStore] I/O exception (java.net.SocketException) caught when processing request to {s}->https://localhost:52320: Broken pipe
  1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.e.RetryExec      ] [testReloadingKeyStore] Broken pipe

I am pretty clueless to what the root cause may be. It is either an issue with out MockWebServer which delegates down to com.sun.net.httpserver.HttpsServer or an issue with apache http client. I ran another test that uses the MockWebServer with HTTPS and it worked..so I am pretty clueless but take another look soon.

@jakelandis
Copy link
Contributor

I can get the test to pass in Java23 by changing:

--- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ssl/SSLConfigurationReloaderTests.java
+++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ssl/SSLConfigurationReloaderTests.java
@@ -130,7 +130,7 @@ public class SSLConfigurationReloaderTests extends ESTestCase {
         // Load HTTPClient only once. Client uses the same store as a truststore
         try (CloseableHttpClient client = getSSLClient(keystorePath, "testnode")) {
             final Consumer<SSLContext> keyMaterialPreChecks = (context) -> {
-                try (MockWebServer server = new MockWebServer(context, true)) {
+                try (MockWebServer server = new MockWebServer(context, false)) {
                     server.enqueue(new MockResponse().setResponseCode(200).setBody("body"));
                     server.start();
                     privilegedConnect(() -> client.execute(new HttpGet("https://localhost:" + server.getPort())).close());

So, the problem with the test is that in Java23 we are wiring up an apache http client to a mock web server with mutual TLS and for some reason the mock web server does not trust the apache http client.

It isn't clear if mTLS is intentional with this test since it really is not an important detail of what is being tested. It also not clear if it worked because the mock web server and apache http client are configured correctly for mTLS or just happened to work in the past. We tend to conflate key and trust stores, so maybe we aren't wiring up mTLS correctly and are relying on a Java bug (pre java 23) for this to work? I'll keep chipping away, but starting to gain more confidence this is not a production concern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs:risk Requires assignment of a risk label (low, medium, blocker) :Security/Security Security issues without another label Team:Security Meta label for security team >test-failure Triaged test failures from CI
Projects
None yet
Development

No branches or pull requests

3 participants