We’ve been historically using the Upload scalar to...

# apollo-kotlin

[Stylianos Gakis]

We’ve been historically using the Upload scalar to upload things. Backend has started using apollo-router, and apparently it simply does no longer work at all according to https://github.com/apollographql/router/issues/842 and alternatives are suggested instead here https://www.apollographql.com/blog/backend/file-uploads/file-upload-best-practices/.
Are we missing something here, or do we really have to simply stop using the Upload scalar completely? This is going to be quite a dev experience downgrade if so, the Upload scalar and the support you’ve built for it for apollo-kotlin has been super convenient to use for the past years.

[bod]

👋 Thanks for the links - I've never thought about the router not supporting this before. And I also had not seen this updated blog post.

Well... it looks like indeed Apollo doesn't recommend going the multipart Upload route anymore.

Maybe we should deprecate it in Apollo Kotlin / at least warn in the doc.

I'll ask around a bit to know more about this (wonder what they do on the iOS and JS clients).

[Stylianos Gakis]

Please do! Any more information on this would be much appreciated.
I am really bummed up about this tbh 😅
It really was so convenient to use. But I do understand if it’s for security concerns maybe there’s no way around it 🤷‍♂️

I think the behavior might wanna stay in apollo-kotlin, since non-apollo-router implementations support this just fine. In fact this works perfectly fine in one of our legacy backend services which are not on apollo-router already

[bod]

the behavior might wanna stay in apollo-kotlin, since non-apollo-router implementations support this just fine

you're probably right, maybe just a warning about it not being supported by router + a link to the blog post. Looks like it may have security concerns.

[Stylianos Gakis]

Yep, exactly

[mbonnin]

I think it makes sense. From a backend point of view the requirements are a bit different:
• GraphQL is short lived structured-data requests
• Upload is long lived binary-data request
The requirements in terms of caching/load balancing/execution environments are quite different

[mbonnin]

From a client point of view, I don't think the migration should be that difficult? Call you favorite HTTP client instead of using GraphQL?

[mbonnin]

As a bonus, it's easier to implement stuff like upload progress or resumable uploads

[Stylianos Gakis]

Call you favorite HTTP client instead of using GraphQL

Idontwanna 😂

But jokes aside, we did actually have to do this once before, because some service was using netflix DGS + WebFlux which surprise surprise, also does not support the Upload scalar. That was for audio files in that scenario.

Now we’ll just do the same, but for file uploads, and to a different service.

The way we had it before, is that we initiated the process with a HTTP request there, and we got back a unique ID back. Then GQL receives in a mutation just that ID, which we only get back if the process succeeded already. It will work here too, and you are right, it’s not that hard to do.

It is what it is 😄

[mbonnin]

Oh you mean you could create the object and upload the object data all in the same HTTP request? I can see how the atomicity is useful there.
With the new approach, it means the backend has to deal with possibly incomplete objects

[mbonnin]

If you really want to, but don't tell anyone I suggested that, you could "just" base64 encode your data to keep the same behaviour 😅

[Stylianos Gakis]

Haha yeah, this was brought up, but also got shut down quickly enough 😅

[Stylianos Gakis]

Oh no no, not the same request. Just:

1. We start by getting the file we need in the app
2. Upload that using HTTP to some service
3. Once this is complete and succeeds, we receive back an ID of that result
4. Take this ID and pass it in a mutation to our apollo-router backed backend service
5. Backend service uses that ID to find the resource
6. If it’s there, all is good and we get a success for the mutation
7. Life is good
Does what I write here make sense?

[mbonnin]

Makes sense 👍 . I usually expect the opposite:
1. call a mutation that creates a resource (audio track)
2. the mutation returns a pre-signed url
3. the client upload the data to the pre-signed url
4. life is good

[mbonnin]

But if there's an error between 2 & 3 you end up with an "incomplete" object in your backend, which your use case prevents

[mbonnin]

Although with your workflow you might end up with orphaned audio track files (tradeoffs 😄 )

[bod]

(looks like the iOS client already has a warning about discouraging Upload, and also gives the 1st workflow as an example of what to do instead)

[mbonnin]

I like workflow #2 because you can handle auth in your GraphQL API. You don't have to handle auth in your upload service and/or let anyone upload files

[mbonnin]

But 🤷 I am not (yet) a backend engineer 😄

[bod]

ahaha yeah as often there are multiple ways to do this

[Stylianos Gakis]

gives the 1st workflow as an example of what to do

The 1st one mentioned in that blogpost right?

Regarding that Martin, you would still need to use that URL in your Retrofit(or whatever) instance. How different is that from initiating the call to the service you know it’s gonna go to instead?

[Stylianos Gakis]

If you got the signed URL and use that to go to that other service, would you then just skip authentication then? Since you know you got a valid URL, so if that’s the case then no need for extra auth?

[mbonnin]

yes, exactly, the url contains the secret

[mbonnin]

And you can upload directly to S3/cloud storage

[mbonnin]

No need to proxy the upload through your GraphQL gateway

[Stylianos Gakis]

I think due to my lack of understanding of how this may look like, I am having a bit of a hard time understanding how this works.
So that URL will contain the place where it will go, but also some sort of secret which will allow us to go to S3 directly. In that scenario, we just do a multipart POST to that URL, and just put the file itself as contents, kinda like how Upload in GQL does it right now here https://github.com/apollographql/apollo-kotlin/blob/1ed167a5b8396bbcc232fdcdf3e85f[…]om/apollographql/apollo3/api/http/DefaultHttpRequestComposer.kt ?

And in that scenario, how do we in the client get notified that the backend has now processed this? Do we need to poll through a GQL query perhaps?

[mbonnin]

And in that scenario, how do we in the client get notified that the backend has now processed this? Do we need to poll through a GQL query perhaps?

You don't know 🙈 . So either polling like you said or having another sync mutation that tells the server to process the resource

[mbonnin]

This also my very superficial understanding of things, I have never really used this in production so take it with a grain of salt

[mbonnin]

In both cases, the fact that upload and GraphQL are different network requests means that there is potential for incomplete data on the server.

[Stylianos Gakis]

I do kinda start liking this approach more then 😅
Auth is not a problem in our context, since we got a auth solution that services can just opt-in and know if someone who’s trying to do anything is logged in or not.
I understand it may not be that nice for some reasons, but it’s the one that I can reason about the easiest in my brain at least. Definitely gonna have to discuss this with my backend colleagues.

And yeah there may then be orphaned files uploaded, but those for our case I think will be able to be cleared out every X days, since the ones that do go through properly should then be referenced from our database in some way.
I would guess S3 would have a way to find all the ones in the bucket that are not part of our list of used ones.

[Stylianos Gakis]

And talking about taking things with a grain of salt, take my ideas with even more salt because I’ve never done any real backend work 😅

[mbonnin]

It could be a mix of both actually
1. you do a mutation to get a presigned url together with some uploadId mutation { getUploadUrl { id url } }
2. Upload that using HTTP to some service
3. Once this is complete and succeeds, we note the uploadId
4. Take this ID and pass it in a mutation to our apollo-router backed backend service
5. Backend service uses that ID to find the resource
6. If it’s there, all is good and we get a success for the mutation
7. Life is good

[mbonnin]

But if auth is not a concern, Agree you can skip 1.

[Stylianos Gakis]

Aha! Yeah that would definitely work.
This way you just need 2 mutations to support this, one to get the ball rolling, and 1 for what you’d need anyway in the approach that I suggested.
And with this approach you get to skip auth and get to interact with the file uploading service only through getting the URL back and using it blindly, instead of hooking up your Retrofit instance to point to that service.
But now doing this HTTP request is a bit more “fire and hope it works” since you just hope that the way you send it is in the same shape as the backend wants it. I guess that’s true for any HTTP request though, maybe I am spoiled by GQL here 😄

I think all this makes total sense though, I got a pretty clear picture of why this limitation exists in the first place, and what we can do instead.
Always fun to chat with you two, and thank you a lot for the help! 🤗

Opened this just to help you out if this is something you want to keep track of for the future, to maybe mention in the docs for example.