Hiya, has anybody here tested their AWS S3 client ...

# http4k

[Philipp Mayer]

Hiya, has anybody here tested their AWS S3 client (using the official SDK) with http4k as fake server? I can’t seem to manage to route it to my local fake server instance.
Using the http4k client is unfortunately no option for now..

[dave]

I'm not sure if you can get away with changing the endpoint configuration in their SDK. Remember that you can set the AwsSdkClient inside which takes a handler, so there are options there.

[Andrew O'Hara]

So long as you're using the official v2 sdk, the http4k-aws module had an adapter you can use to route your requests in-memory. See this example, and this part in particular. If you want to route to an actual running fake server, then you can just override the client endpoint settings, as David suggested.

[Philipp Mayer]

Upgrading to the v2 sdk helped, thanks for the hint Andrew! 🙂
Though I’m still running into issues that I can’t seem to fix:

import org.http4k.aws.AwsSdkClient
import org.http4k.client.JavaHttpClient
import org.http4k.connect.amazon.core.model.Region
import org.http4k.connect.amazon.s3.FakeS3
import org.http4k.connect.amazon.s3.createBucket
import org.http4k.connect.amazon.s3.model.BucketName
import org.http4k.core.Uri
import org.http4k.core.then
import org.http4k.filter.ClientFilters
import org.http4k.filter.debug
import org.http4k.server.SunHttp
import org.http4k.server.asServer
import software.amazon.awssdk.core.sync.RequestBody
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.PutObjectRequest

//
val bucketName = BucketName.of("example-bucket")
val s3 = FakeS3().apply {
    s3Client().createBucket(bucketName, Region.EU_CENTRAL_1)
}

s3.debug().asServer(SunHttp(port = 26467)).start()

val client = ClientFilters.SetBaseUriFrom(Uri.of("<http://example-bucket.s3.eu-central-1.localhost:26467>")).then(JavaHttpClient()
)

val s3Client: S3Client = S3Client.builder()
    .httpClient(AwsSdkClient(client))
    .region(software.amazon.awssdk.regions.Region.EU_CENTRAL_1)
    .build()

val request = PutObjectRequest.builder()
    .bucket(bucketName.value)
    .key("this-is-a-key")
    .build()

val body = RequestBody.fromString("test")

s3Client.putObject(request, body)

yields:

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 503, Request ID: null)

just posting via curl works, though:

curl --location '<http://dealer-attachments.s3.eu-central-1.localhost:26467>'

This is now rather off-topic because it’s more on the AWS sdk side, but maybe someone encountered the same issue 🙂
I guess it’s because of the dns resolution - if I point the client against <http://localhost>:... it works, but isn’t forwarded to the right bucket.

The readme also covers that, but I don’t know how to fix it:
“In the case of a missing header (if for instance a non-http4k client attempts to push some data into it without the x-forwarded-for header, it creates a global bucket which is then used to store all of the data for these unknown requests.“

[Andrew O'Hara]

So, the AwsSdkClient is only necessary if you want to use an in-memory server. In that case, you don't need to start FakeS3, and would just do .httpClient(AwsSdkClient(FakeS3()))

If you want to run against a fake server running with a port, you don't need to override the httpClient(...). Instead you should do an .endPointOverride(URI("<http://localhost>:${server.port()}")) on the client builder. You don't need to override the server's port; it will use any available one.

I've updated the example code to show both methods. Here's how to do the latter.

[Andrew O'Hara]

However, overriding the official sdk's official client to use JavaHttpClient could be a novel way to eliminate the heavyweight apache client. I don't normally bother with that outside lambdas.

[Philipp Mayer]

I actually tried .endPointOverride already, resulting in:
.endpointOverride(URI("<http://localhost:26467>")) to`Caused by: java.net.UnknownHostException: dealer-attachments.localhost`
.endpointOverride(URI("<http://s3.eu-central-1.localhost:26467>")) to Caused by: java.net.UnknownHostException: dealer-attachments.s3.eu-central-1.localhost
etc. I can’t pin point it for now, but I still have the gut feeling around DNS issues.
Again, this is what I want to end up with basically: curl --location --request PUT '<http://dealer-attachments.s3.eu-central-1.localhost:26467/test1234>'

[dave]

hmm - this is the official client being "clever". You can force a rewrite of the URL by doing AwsHttpClient(SetHostFrom(Uri.of('<http://localhost:80123>")).then(JavaHttpClient))

[Philipp Mayer]

This is what I did in the example earlier, no? (tried it with different URLs) 🙂

[Philipp Mayer]

Here’s an example repo that I will use to play around with: https://github.com/p10r/aws-sdk-http4k-fake

(I included debug().asServer(SunHttp(port = 26467)).start() to show that it’s not reaching the actual endpoint)

val client = ClientFilters.SetBaseUriFrom(Uri.of("<http://localhost:26467>"))
            .then(JavaHttpClient())

Will log the request to FakeS3, but it’s targeting the incorrect endpoint.

[Andrew O'Hara]

Ah yes, because s3 has subdomains... I'll see what I can come up with.

[Philipp Mayer]

Exactly 🙂 I’ll take a walk, maybe I’ll come up with a new idea there 😄

[Philipp Mayer]

I thought about a reverse proxy, but haven't looked into it yet.

[Andrew O'Hara]

It occurs to me that this issue might also occur when using the fake server in port mode with the http4k-connect client as well. i.e. I don't think the problem is unique to the official SDK.

Is there a particular reason you want to test against a running server, and not in-memory?

[dave]

There is also X-forwarded-for which we use to get around some limitations with the subdomains. Might be worth investigating that as well... We use it in the main S3 client to route requests to the buckets.

[Philipp Mayer]

Is there a particular reason you want to test against a running server, and not in-memory?

I’m introducing tests to a typical spring boot code base and want to use it in an E2E context - the local server version is better fitting there and easier to grasp for the team (which is new to Kotlin and thorough testing overall). Hope that makes sense. 🙂

[Philipp Mayer]

After some back and forth we're now having our first http4k dependency not only for testing, but also for production! I'm super happy.

[Andrew O'Hara]

Grats! I'd be interested to see how you managed to do it in the end.

[dave]

@Andrew O'Hara I presume you mean "convince people that http4k can really be used in production" rather than "take a punt on the famously unstable http4k" 🙂

[Andrew O'Hara]

Every time I see someone ask if something is "production ready", I die a little inside.

I'm actually curious how he managed to get his s3 client working against an http4k-connect FakeS3 in port mode, which I guess is a bit of the latter. 🤔

[Philipp Mayer]

Some more context is needed: there's a service that has a heavy dependency on S3 and the simple strategy of "just mock it" wasn't really working anymore because they needed a real fake.
While I would've loved to get it to run on a port, in the end I just went for the http4k client, injecting the fake S3 directly.
This was quite new for the team, but we had some time pressure and it simply worked, so everyone is happy.

[Philipp Mayer]

About the production ready part: it's actually not that some teams can't imagine a library to work - it's more that they don't even know something outside of the Spring universe. It's completely normal for them to work around the framework and constantly fight it.

[Andrew O'Hara]

Ok cool. Later on I might try to come up with some samples that get it to work on a port.

[Andrew O'Hara]

Perhaps eventually you'll be able to show them the light: bit by bit.

[Philipp Mayer]

This is the way! Introducing http4k from tests is actually quite a compelling strategy.

[Andrew O'Hara]

I'm finally following up on this. When running FakeS3 as a server, it works as-is with the http4k-connect client. In order to use the official V2 SDK, David's workaround worked for me. I'm adding this as an example to the examples repo.

val http= ClientFilters.SetHostFrom(FakeS3::class.defaultLocalUri).then(JavaHttpClient())
val client = S3Client.builder()
    .httpClient(AwsSdkClient(http))
    .credentialsProvider { AwsBasicCredentials.create("key_id", "secret_key") }
    .build()

https://github.com/http4k/http4k-connect-examples/pull/49

[Philipp Mayer]

Heya, sorry for getting back just now! Thanks a ton for the example. I simply stopped trying and went http4k-native, so didn’t bother with it again.