Basic question: `urlStr.urlEncoded()` is turning m...

# http4k

[Gopal S Akshintala]

Basic question: urlStr.urlEncoded() is turning my URL into this: https%3A%2F%<http://2Fpokeapi.co|2Fpokeapi.co>%2Fapi%2Fv2%2Fpokemon%3Flimit%3D10 which is invalid. What’s the right and http4k idiomatic way to do it?

[s4nchez]

you normally only need to url encode "values" (e.g. query parameters) rather than the full url. If you have a full example of what you're trying to do, maybe it'll be easier to spot where the problem is.

[Gopal S Akshintala]

Sure… one of my urls has spaces something like {{baseUrl}}/services/data/v{{version}}/query/?q=SELECT Id FROM Profile where Name = 'Standard User'
I don’t wanna string replace spaces with %20 as there can be all types of urls with spl characters etc, so I wanna leverage encoder

[Gopal S Akshintala]

val uri = Uri.of(itemRequest.url.raw).let { it.query(it.query.urlEncoded()) }

I tried this, but it is replacing = sign also which is causing a problem
like limit%3D10

[s4nchez]

what problems is it causing? The way I'd write is:

Uri.of("{{baseUrl}}/services/data/v{{version}}/query/")
    .query("q", "SELECT Id FROM Profile where Name = 'Standard User'")

which translates to {{baseUrl}}/services/data/v{{version}}/query/?q=SELECT+Id+FROM+Profile+where+Name+%3D+%27Standard+User%27

[Gopal S Akshintala]

But I get these urls as params at runtime, I don’t know them prior.

[s4nchez]

I still can't follow. You get a full URL and what do you need to do with it? The whole setup looks strange (let alone the fact that sending SQL as parameter looks very suspicious security-wise)

[Gopal S Akshintala]

Ah that’s SOQL query (Salesforce). It’s normal to pass… so basically I am trying to parse postman collection json file (something like the one attached). Each item in this json has a url to fire. So that’s how I get the url at run time.

Pokemon.postman_collection.json

[Gopal S Akshintala]

Tried this, seems to be working for my existing usecases:

fun encodeUri(url: String): Uri =
  Uri.of(url).let { uri ->
    uri.queries().map { query -> query.first to query.second?.urlEncoded() }
      .foldRight(uri) { encodedQuery, _ -> uri.query(encodedQuery.first, encodedQuery.second) }
}

[s4nchez]

Glad you find a workaround. We could consider adding a way to parse Uris (including query etc), but another alternative would be using java.net.URI to do that job and then converting to its various parts to org.http4k.Uri

[Gopal S Akshintala]

Thanks @s4nchez but if you see this (UrlEncode only query values) as a valid use case, I can find some time to contribute this within http4k 🙂

[s4nchez]

@Gopal S Akshintala that's a good spot! Do you want to submit a PR for a fix or are you ok if I just do it myself?

[Gopal S Akshintala]

I wish to submit a PR, but can only do it on the weekend 🙂 does that work?

[s4nchez]

Or course. That code is almost five years old, so I think it can survive being like that for a few more days 😉 Thank you @Gopal S Akshintala!

[James Richardson]

🤔 i'm not 100% sure about this. I think your query parameter should be properly encoded as:
query/?q=SELECT%20Id%20FROM%20Profile%20where%20Name%20%3D%20'Standard%20User'

[James Richardson]

note that there is no '=' in the parameter value because it was encoded.

[Gopal S Akshintala]

@James Richardson but to encode, I need to split q part and SELECT … part first and encode only the value. The problem is with the split("=") which returns the value as SELECT Id FROM Profile where Name

[Gopal S Akshintala]

trimming the 'Standard User' as there is an = before that

[James Richardson]

well, where did that come from - i think you're talking about parsing the URL from your json file, right?

[James Richardson]

not parsing the parameters in the request handler... if i understand right

[Gopal S Akshintala]

Yes right, I have these urls coming from a JSON file

[James Richardson]

the JSON file was produced and is managed by postman. usually it uses placeholder variables for parameters (which it will encode properly before sending) - perhaps the request that is captured in the postman file is itself incorrectly encoded. You could try this by using a variable for the request parameter, then substitute the value at runtime (in postman) and see what is actually sent over the wire. i suspect (i'm not 100%) that it will match what i posted.

[James Richardson]

Also for extra fun - the stuff that goes into the JSON file will also be json encoded, so needs to be read with care 🙂

[Gopal S Akshintala]

This is the corresponding json in the collection file:

{
			"name": "Standard User Profile Query",
			"event": [
				{
					"listen": "test",
					"script": {
						"exec": [
							"var statusCode = pm.response.code;",
							"",
							"var jsonData = JSON.parse(responseBody);",
							"if(jsonData.records[0]!=null){",
							"    pm.environment.set(\"standardUserProfileId\", jsonData.records[0].Id);",
							"}"
						],
						"type": "text/javascript"
					}
				}
			],
			"protocolProfileBehavior": {
				"disableBodyPruning": true
			},
			"request": {
				"method": "GET",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/json"
					}
				],
				"body": {
					"mode": "raw",
					"raw": ""
				},
				"url": {
					"raw": "{{baseUrl}}/services/data/v{{version}}/query/?q=SELECT Id FROM Profile where Name = 'Standard User'",
					"host": [
						"{{baseUrl}}"
					],
					"path": [
						"services",
						"data",
						"v{{version}}",
						"query",
						""
					],
					"query": [
						{
							"key": "q",
							"value": "SELECT Id FROM Profile where Name = 'Standard User'"
						}
					]
				}
			},
			"response": []
		},

[Gopal S Akshintala]

I hv a logic to replace those placeholder variables, using Moshi factory

[James Richardson]

Yes, I believe your url in the postman file is incorrectly encoded.

[Gopal S Akshintala]

No the json file has whatever is in the screenshot, Postman doesn’t do anything in between

[James Richardson]

Yes, the query that has been typed in, in the screenshot, is incorrectly encoded

[James Richardson]

See my comment above about using a parameter variable, and see if you get a different request. I believe you will. I think you'll find that the select query is encoded as I posted above. I'm not 100% sure on this, but I think so

[James Richardson]

well - i tried it and postman doesn't do what i expected at all! the encoding it uses for parameter values doesn't encode = 🤷 - apologies - might have just thrown confusion into the mix.
encodeURIComponent on javascript, and also some other encoding thing do, but postman doesn't. hmmm

[James Richardson]

so maybe the thing to do is what you said right at the start. sorry for that.

[s4nchez]

Build is failing. Did it pass locally?

[Gopal S Akshintala]

I noticed some failing unit tests and I fixed them an hour ago, but when I do gradle clean build I get this: Dust.kt: (3, 12): Unresolved reference: nashorn which might not be related to the change I did

[s4nchez]

Which jdk are you using? We expect 11

[Gopal S Akshintala]

I see, yes ./gradlew clean build succeed in my machine

[s4nchez]

Ok I'll check the pr later again (not close to a computer now). Thank you for submitting it!