Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Servirtium fails to respond with correct headers when using Apollo Kotlin and large response #1065

Open
snowe2010 opened this issue Feb 23, 2024 · 8 comments

Comments

@snowe2010
Copy link

When trying to record responses when using the Apollo Kotlin client, if the response body is over a certain size then a gzip compression is applied. When this gzip compression is applied, Servirtium (or Apollo Kotlin) fails to set the Content-Encoding header properly when the response makes it back to the Apollo client. This results in the wrong body trying to be unwrapped, and a failure of

Expected leading [0-9a-fA-F] character but was 0x7b

This problem could be in Apollo Kotlin or even the underlying OkHttp library, but I spent several hours trying to track down the issue with little gained. I believe the problem 'begins' here in the BridgeInterceptor. This is at least the spot where you can place debug points on these two lines and in the InteractionOptions object and see that the request is sent, the response is received by Servirtium, the headers are set, and then the response sent back. But the response that is sent back has all the headers set to the original values (I think), so it's like the interaction options didn't even happen.

I believe the problem could be due to OkHttp doing something weird with the headers here in the Http1ExchangeCodec, maybe they are copying the headers from the request (that doesn't seem to make sense), but I'm not totally sure.

If I should create an issue with either OkHttp or Apollo Kotlin please let me know, else this mostly seems like a Servirtium error (as it works if you just directly run without the Servirtium proxy).

I've attached an MCVE with 3 test classes that demonstrate the errors:

  • DirectMainTest shows that direct calls without Servirtium succeed
  • DiskRecordingMainTest shows that placing Servirtium in the middle immediately causes a failure of the large payload case, while the small payload works perfectly fine
  • DiskRecordingSetGzipManuallyMainTest demonstrates the exact same as the DiskRecordingMainTest, but tries to set the headers manually so that it's possible to see the points at which the proxy is hit, the interceptors are hit, and the response headers are modified.

http4k-servirtium-graphql-mcve.zip

@daviddenton
Copy link
Member

Thanks for the investigation. As a very quick way of halving the problem space - have you tried changing to a different Http client implementation and seeing it the problem persists?

@snowe2010
Copy link
Author

Changing Apollo to use a different client? or changing Servirtium to use a different client?

@snowe2010
Copy link
Author

Ah, ok, so testing with different http4k clients:

  • == Succeeds

  • JavaHttpClient
  • Java8HttpClient
  • Apache 4
  • Apache 5
  • Fuel
  • Helidon
  • Jetty
  • OkHttp

This will definitely unblock me for now.

@daviddenton
Copy link
Member

So it is something to do with the client. I thought you were using OkHttp and it was failing though? 😕

@snowe2010
Copy link
Author

So I wasn't specifying a proxyClient in the servirtium instantiation. I was just depending on whatever the default was. I thought that including the implementation("org.http4k:http4k-client-okhttp:$http4k_version") dependency would handle that. Still used to 'automagic' compared to http4k's functional approach. So I'm guessing servirtium just defaulted to javahttpclient?

@daviddenton
Copy link
Member

yes - we try to use defaults with no dependencies whenever possible :)

@snowe2010
Copy link
Author

Hm. so the okhttp client works in the mcve i provided, but it does not work in our actual codebase, but now the message is even less clear. Seems like the gzipping is encoding incorrectly somewhere?

java.io.IOException: ID1ID2: actual 0x00001fef != expected 0x00001f8b
	at okio.GzipSource.checkEqual(GzipSource.kt:197)
	at okio.GzipSource.consumeHeader(GzipSource.kt:110)
	at okio.GzipSource.read(GzipSource.kt:62)
	at okio.Buffer.writeAll(Buffer.kt:1303)
	at okio.RealBufferedSource.readByteString(RealBufferedSource.kt:215)
	at com.apollographql.apollo3.network.http.LoggingInterceptor.intercept(LoggingInterceptor.kt:125)
	at com.apollographql.apollo3.network.http.DefaultHttpInterceptorChain.proceed(HttpInterceptor.kt:22)
	at com.apollographql.apollo3.network.http.HttpNetworkTransport$execute$1.invokeSuspend(HttpNetworkTransport.kt:65)
	at com.apollographql.apollo3.network.http.HttpNetworkTransport$execute$1.invoke(HttpNetworkTransport.kt)
	at com.apollographql.apollo3.network.http.HttpNetworkTransport$execute$1.invoke(HttpNetworkTransport.kt)
	at kotlinx.coroutines.flow.SafeFlow.collectSafely(Builders.kt:61)
	at kotlinx.coroutines.flow.AbstractFlow.collect(Flow.kt:230)
	at kotlinx.coroutines.flow.internal.ChannelFlowOperatorImpl.flowCollect(ChannelFlow.kt:195)
	at kotlinx.coroutines.flow.internal.ChannelFlowOperator.collectTo$suspendImpl(ChannelFlow.kt:157)
	at kotlinx.coroutines.flow.internal.ChannelFlowOperator.collectTo(ChannelFlow.kt)
	at kotlinx.coroutines.flow.internal.ChannelFlow$collectToFun$1.invokeSuspend(ChannelFlow.kt:60)
	at _COROUTINE._BOUNDARY._(CoroutineDebugging.kt:46)
	at kotlinx.coroutines.flow.FlowKt__ReduceKt.single(Reduce.kt:57)
	at com.sunrun.pricing.keystone.ProdKeystoneService$listProducts$response$1.invokeSuspend(KeystoneService.kt:32)
Caused by: java.io.IOException: ID1ID2: actual 0x00001fef != expected 0x00001f8b
	at okio.GzipSource.checkEqual(GzipSource.kt:197)
	at okio.GzipSource.consumeHeader(GzipSource.kt:110)
	at okio.GzipSource.read(GzipSource.kt:62)
	at okio.Buffer.writeAll(Buffer.kt:1303)
	at okio.RealBufferedSource.readByteString(RealBufferedSource.kt:215)
	at com.apollographql.apollo3.network.http.LoggingInterceptor.intercept(LoggingInterceptor.kt:125)
	at com.apollographql.apollo3.network.http.DefaultHttpInterceptorChain.proceed(HttpInterceptor.kt:22)
	at com.apollographql.apollo3.network.http.HttpNetworkTransport$execute$1.invokeSuspend(HttpNetworkTransport.kt:65)
	at com.apollographql.apollo3.network.http.HttpNetworkTransport$execute$1.invoke(HttpNetworkTransport.kt)
	at com.apollographql.apollo3.network.http.HttpNetworkTransport$execute$1.invoke(HttpNetworkTransport.kt)
	at kotlinx.coroutines.flow.SafeFlow.collectSafely(Builders.kt:61)
	at kotlinx.coroutines.flow.AbstractFlow.collect(Flow.kt:230)
	at kotlinx.coroutines.flow.internal.ChannelFlowOperatorImpl.flowCollect(ChannelFlow.kt:195)
	at kotlinx.coroutines.flow.internal.ChannelFlowOperator.collectTo$suspendImpl(ChannelFlow.kt:157)
	at kotlinx.coroutines.flow.internal.ChannelFlowOperator.collectTo(ChannelFlow.kt)
	at kotlinx.coroutines.flow.internal.ChannelFlow$collectToFun$1.invokeSuspend(ChannelFlow.kt:60)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

@snowe2010
Copy link
Author

Actually, I do get it in the mcve. I just had to actually add the direct playback test with some interaction options:

object MainInteractionOptions : InteractionOptions {
    override fun modify(request: Request): Request {

        val newRequest = request
            .replaceHeader("Authorization", "00xxx0xxx0xxxxxx")
            .replaceHeader("Host", "localhost")
            .replaceHeader("User-agent", "agent")
            .replaceHeader("Date", "Tue, 28 Jan 2020 14:15:55 GMT")
        return super.modify(newRequest)
    }
    override fun modify(response: Response): Response {

        val newResponse = response
            .replaceHeader("Content-Encoding", "gzip")
            .removeHeaders("X-")
            .removeHeader("Set-Cookie")
            .replaceHeader("Date", "Tue, 28 Jan 2020 14:15:55 GMT")

        return newResponse
    }

    override fun debugTraffic() = false
}


@Tag("playback-api")
class MainDirectPlaybackTests : MainTest {
    override val url by lazy { "http://localhost:${servirtium.port()}" }

    private lateinit var servirtium: Http4kServer

    @BeforeEach
    fun start(info: TestInfo) {
        servirtium = ServirtiumServer.Replay(
            info.markdownName(),
            Disk(File("src/test/resources/servitium-recordings")),
            options = MainInteractionOptions
        )
        servirtium.start()
    }

    @AfterEach
    fun stop() {
        servirtium.stop()
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants