https://kotlinlang.org logo
Join Slack
Powered by
is there any way to get the actual body from a res...
# http4k
s
is there any way to get the actual body from a response when using 
.asResult
in your lens? i.e.
Copy code
val ssmResponse = Body.auto<SSMObject>().toLens().asResult()
return ssmResponse(client(request))
            .map {
                println("SSM call was successful, returning the client token decrypted hopefully")
                it.parameter.value
            }
            .mapFailure {
                throw IllegalStateException(
                    "unable to find the client secret $clientSecretParameterUrl:  $it" // i need access to the bodyString() here
                )
            }.get()
figured it out you have to cast the lensfailure target to a MemoryResponse and then pull the body out from there.
a
Glad to hear you figured it out. But did you know there's a prebuilt Http4k SDK for SSM and many other AWS services? https://github.com/http4k/http4k-connect/tree/master/amazon/systemsmanager
s
I did not, I’ll take a look at that, but the above code is actually calling the 
AWS Parameters and Secrets Lambda Extension
, rather than SSM directly, for faster response and caching. I do have to call ssm to put some data, but I’m using quarkiverse’s SSM service, which autowires all the credentials automatically so I don’t have to actually configure anything. It would be nice to keep the binary size low, but not sure the tradeoffs are worth it.
a
Ah yes, so the extension isn't a bad idea. Perhaps http4k-connect should support it 🤔 . If you're looking to get secrets into your lambda, cloudformation/SAM can resolve them at deploy-time. Supposedly they're encrypted at rest. This will put the params into your env on init, so long as you don't expect them to change.
s
I don’t expect some of them to change, but the final one will change. I don’t want to use cdk/cf/sam to embed the creds at deploy time as that has a few downsides: • need to embed based on environment • changing them requires a redeploy in case of a leak • env variables are clearly visible in lambda console, which is fine for some stuff, not for other stuff.
what does the Fake do in the http4k-aws-sdk?
d
The fakes are substitute implementations of the services which can be used for testing. They all run totally in-memory or you can launch them on a port. They're like localstack without docker (and many many times faster).
(if you're talking about the http4k-connect fakes that is)
s
oh nice. so they still do ‘function’? like if you put a parameter and then get it, you’ll actually get a parameter back?
d
Yep
s
oh that’s really neat. what about preloading data? is that possible?
d
They're all stateful
It's just code - you can do anything! 🙃
s
haha ok, wondered if that was gonna be the answer.
d
@Andrew O'Hara helped us to create a really cool fake dynamo which supports pretty much all of the functionality. 🙃
s
i’ll need to look into that. that sounds really useful.
d
Each fake has a client built in so you can just create it and then call whatever operations to populate.
We've even got a fake Kms which has pregenerated keys so you can do encryption and signing etc.
s
ooh…
d
And of course the connect adapters all run with Moshi and no reflection so you also get to not rely on Jackson (which I know you wanted for your native stuff)
s
yeah, I’m not going to be able to remove jackson until I get a PR through that removes the need for it in the quarkus handler stuff, but jackson doesn’t really work properly in native at all so there’s other reasons lol. we’ve been using kotlinx-serialization for years now.
hm. man I really want to use the http4k connect adapters, but it looks like it’s not really going to work for me. I don’t see any ability to specify the endpointUrl when creating the client, which means that I can’t use localstack, which means that if I have multiple lambdas and one writes to SSM and the other reads from SSM then they will not be able to reference the same underlying datastore. The same problem occurs with the fake, as each fake is unique and so if I have a quarkustest that starts up the application, hits the lambda, and the fake is used, that if I try to validate what’s in SSM it will be using a different fake, rather than the same instance available in the running application. damn…
a
You can override the endpoint the same way you would point it to a fake. You just pass in an http client with a ClientFilters.SetHostFrom wrapped around it. Can't really give a better example since I'm afk. Also, the fake kms keys aren't pre-generated anymore. Now each key is generated on demand and won't conflict with the other keys anymore.
👍 1
d
you can supply a custom storage implementation to to share the state of you want, or you can just start up the fake server and override the endpoint as Andrew suggested. How are you running your lambdas? If they are all in memory in a single process then this should be trivial to get them to use the same in memory version. It's all about having everything in a single test env. 🙃
s
sorry, off for christmas! hope y’all had a good holiday. ok, so I can either add a filter, or share a custom storage impl. Custom storage impl sounds the best, but not sure how I would do that.
How are you running your lambdas? If they are all in memory in a single process then this should be trivial to get them to use the same in memory version. It’s all about having everything in a single test env.
so running in aws is fine, all the http4k stuff works great, saves to ssm, works perfectly. It’s when I’m trying to test with quarkus tests that it’s a problem. So I autowire a fake using something like this:
Copy code
@Dependent
class HttpConfig {
    @Produces
    @IfBuildProfile(anyOf = ["test", "dev"])
    fun ssmClient(): SystemsManager {
        val http = FakeSystemsManager()

        return SystemsManager.Http(Region.of("us-east-1"), { AwsCredentials("test", "test") }, http.debug())
    }

    @Produces
    @DefaultBean
    fun prodSsmClient(): SystemsManager {
        val http = JavaHttpClient()

        return SystemsManager.Http(Region.of("us-east-1"), { AwsCredentials("test", "test") }, http.debug())
    }
}
(haven’t actually tested that the prod bean works yet) then my test is a quarkustest so it starts the quarkus application (i’m pretty sure in a separate thread… not quite sure how all the underlying quarkus stuff runs), and hits the running application from a restassured call.
Copy code
@QuarkusTest
@TestProfile(UpdateTokenHandlerProfile::class)
open class UpdateServiceAuthTokenLambdaTest {
    @Inject
    lateinit var ssmClient: SystemsManager

    @Test
    @Throws(Exception::class)
    fun `test auth token saved to SSM`() {
        RestAssured.given()
            .contentType("application/json")
            .accept("application/json")
            .`when`()
            .post()
            .then()
            .statusCode(200)
            .body(CoreMatchers.containsString("""success":true"""))
        val authToken = ssmClient.getParameter(SSMParameterName.of(authTokenPath),true).valueOrNull()
        expectThat(authToken)
            .isNotNull()
            .get(ParameterValue::Parameter)
            .get(Parameter::Value)
            .isA<String>()
            .isNotBlank()
            .isNotEmpty()
    }
}
d
so you either want to run the entire thing IN memory, or you can start the SSM with 
FakeSystemsManager().start()
in your test and it will start on the default port. Then you'd use the JavaHttpClient as usual. If you're missing data between the tests then I suspect that your magic wiring (this is why we hate magic!) is maybe creating multiple copies of the FakeSystemsManager and so you're getting a different one for each test? The storage could help here, but it's all the same thing - you need to get a single instance of either if you want to keep it in memory, or to just use a networked version. - there are storage implementations for. Redis, S3, File etc... see: https://github.com/http4k/http4k-connect?tab=readme-ov-file#supported-storage-backends-named-http4k-connect-storage-technology)
In http4k-land we'd just create a single instance of the Fake and inject it anywhere we needed an HTTP client
so I suspect that amounts to injecting a common HTTP Handler (of which the FakeSSM is one 🙂
s
yeah I can do that, I think it’s just that it starts quarkus as an entirely separate process or something so it’s not even linked in any way besides the rest calls in my test
d
if you want to stick with the dependency stuff
s
I kinda have to stick with the dependency stuff in order to use QuarkusTest… 😞
i was going to do the http4k lambdas as I’d been talking with y’all about, but think it’s a bit too much to switch my team on at this point.
from workflows and stuff to just general knowledge around quarkus we already have.
d
fiar enough - you just need to show them how much better life can be 😉
s
haha
d
Try starting the Fake on a port and then using the JavaHttpClient with a custom baseUrl
s
I can’t figure out the custom base url. I thought it would be like this, but getting an error.
d
it's the Fake SSM you want to wrap
not the SSM Client
All of the connect fakes are just HTTP handlers with a set of routes preinstalled instead of an HTTP client
Copy code
SetBaseUriFrom(apiUrl).then(FakeSystemsManager())
(but you can also start them up on a port, because both clients and servers in http4k (and ServerAsAFunction) are just HttpHandlers
s
sorry, that is what I”m doing, it’s just called ssm client. but the type is SystemsManager.
d
yes - you're trying with the wrong thing
you need something of type HttpHandler, not SystemsManager 🙂
s
oh oh oh.
d
On the above you can write a simple proxy with:
Copy code
fun main() {
    SetBaseUriFrom(Uri.of("<http://github.com>")).then(JavaHttpClient()).asServer(SunHttp(8080)).start()
}
it's all just functions!
😅 1
s
agh, my bean isn’t returning the fakesystemsmanager lol
d
I'm not going to hear you let me say "you're doing this to yourself" 😉
s
haha
rofl. i didn’t need to do any of that. I just made the fakesystemsmanager bean a 
@Singleton
and everything just works 🤦🏽‍♂️
d
it's not a 🤦 when there's magic involved - it's more like....
image.png
😉
but - I'm glad you got it working anyway
s
dude, if you could make an extension for drools that takes and makes drools cloud native like Kogito does, then I’d have my team drop quarkus/kogito in a microsecond.
d
I seem to remember looking at knative when we were doing the other serverless implementations. IIRC, we decided that we already had all of the bits for it as it's just a webserver (with no custom types). WRT to Drools - I haven't seen a BPMN engine for about 5 years, but if it's anything like the one I was forced to use (at least temporarily until we dropped it 😉 ), I think you could quite easily create a simple system to create and mount the drools engine into it, then just provide adapters to map between HTTP messages and the whatever types you need for the request. (... and no, I don't think we want to write that - for one because then we'd be on the hook for supporting a BPMN engine integration 😂 )
🤣 1
s
so, I think I might be seeing a bug with the fake SSM, unless I’m misunderstanding something. I’ve got this code:
Copy code
@Test
    fun `test 1`() {
        ssmManager.putParameter(
            SSMParameterName.of("abc"), "123", ParameterType.SecureString,
            Overwrite = true,
            KeyId = KMSKeyId.parse("alias/aws/ssm")
        )
    }
    @Test
    fun `test 2`() {
        ssmManager.putParameter(
            SSMParameterName.of("abc"), "123", ParameterType.SecureString,
            Overwrite = true,
            KeyId = KMSKeyId.parse("alias/aws/ssm")
        )
    }
note it’s the exact same test copied and pasted, but the first test succeeds at saving the parameter, the second fails:
Copy code
***** REQUEST: POST: <https://ssm.us-east-1.amazonaws.com/> *****
POST <https://ssm.us-east-1.amazonaws.com/> HTTP/1.1
X-Amz-Target: AmazonSSM.PutParameter
Content-Type: application/x-amz-json-1.1
X-Forwarded-Host: <http://ssm.us-east-1.amazonaws.com|ssm.us-east-1.amazonaws.com>
host: <http://ssm.us-east-1.amazonaws.com|ssm.us-east-1.amazonaws.com>
x-amz-content-sha256: dd07741b6cfd5f48da0c8fdb657445b84d80765e1216449f2bc5250c3e09b45e
x-amz-date: 20231228T004328Z
content-length: 91
Authorization: AWS4-HMAC-SHA256 Credential=test/20231228/us-east-1/ssm/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target;x-forwarded-host, Signature=a3d3818e6a439dd402c46a31382092351324e6257e63955a8e36d4be08026a77

{"Name":"abc","Value":"123","Type":"SecureString","KeyId":"alias/aws/ssm","Overwrite":true}
***** RESPONSE 200 to POST: <https://ssm.us-east-1.amazonaws.com/> *****
HTTP/1.1 200 OK


{"Tier":"Standard","Version":1}
Copy code
***** REQUEST: POST: <https://ssm.us-east-1.amazonaws.com/> *****
POST <https://ssm.us-east-1.amazonaws.com/> HTTP/1.1
X-Amz-Target: AmazonSSM.PutParameter
Content-Type: application/x-amz-json-1.1
X-Forwarded-Host: <http://ssm.us-east-1.amazonaws.com|ssm.us-east-1.amazonaws.com>
host: <http://ssm.us-east-1.amazonaws.com|ssm.us-east-1.amazonaws.com>
x-amz-content-sha256: dd07741b6cfd5f48da0c8fdb657445b84d80765e1216449f2bc5250c3e09b45e
x-amz-date: 20231228T004328Z
content-length: 91
Authorization: AWS4-HMAC-SHA256 Credential=test/20231228/us-east-1/ssm/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target;x-forwarded-host, Signature=a3d3818e6a439dd402c46a31382092351324e6257e63955a8e36d4be08026a77

{"Name":"abc","Value":"123","Type":"SecureString","KeyId":"alias/aws/ssm","Overwrite":true}
***** RESPONSE 400 to POST: <https://ssm.us-east-1.amazonaws.com/> *****
HTTP/1.1 400 Bad Request


{"__type":"ResourceNotFoundException","Message":"AmazonSSM can't find the specified item."}
d
Well you can easily debug it, but yes - the current implementation doesn't allow overwriting the value (it converts a null here into a 400). Should be simple to fix by checking the overwrite param, feel free to PR it and we'll release. 🙃 https://github.com/http4k/http4k-connect/blob/9ae644825ccb8be0d230e4881fa8378516314824/amazon/systemsmanager/fake/src/main/kotlin/org/http4k/connect/amazon/systemsmanager/endpoints.kt#L42
In the meantime before release it's easy to take a copy of the fake and run it locally with the fix in the fixed endpoint 🙃
s
looks like you beat me to it. was just about to PR it lol
d
yep - it['s released now in the latest version 🙂
Open in Slack
PreviousNext
Powered by