A modern programming language that makes developers happier.

kotlinlang

Bit of a random question:
&gt; For the `percentile` operation, DataFrame uses estimation method R3 when the given percentile needs to be selected from the values (like for self-comparable columns), and R8 when the given percentile can be interpolated from the values (of a numeric column).
Is there a way to force-use R3 for percentiles on numeric columns (or as in my case, Durations) ? My use case is _very_ small datasets, where p80 gives me what some my find surprising results: with [1,1,1,1,5] i'd expect p80 to be `1`. Google <https://www.google.com/search?q=80th+percentile+of+%281%2C1%2C1%2C1%2C4%29|sometimes agrees> and <https://www.google.com/search?q=80th+percentile+of+%281%2C1%2C1%2C1%2C5%29|sometimes doesn't>. Every other (more) legit source agrees that it's _not_ actually 1, something between 1 and the larger value i have in position 5...
In my case, I "just" want to eliminate outliers and work with the real values - rather than assume my data is uniformly distributed (it isn't)

Hi! Good question :smile: Currently we don't offer a way to pick which quantile estimation method to use; `percentile` uses R8 when it encounters numeric columns, R3 otherwise. However, DataFrame also offers `percentileBy`, which always selects a value from the column instead of calculating it. This essentially forces R3 :slightly_smiling_face:

Ohhh! Subtle. I’ll have a look, thanks 

in my `aggregate {}` block, i'd use it like this -- is that correct?
``` percentileBy(80.0, "value", false).values().single() ```
out of curiosity, how come there's such a difference between percentile() and percentileBy()

oh i also only just noticed the docs do say
&gt; In the future we might add an option to change the quantile estimation method.

image.png

For aggregation, I think you mean `groupBy`  `aggregate` right?
See here for an example of that DSL: <https://kotlin.github.io/dataframe/groupby.html#transformation>

The difference between `-By` and the normal one we explain here (see picture) <https://kotlin.github.io/dataframe/summarystatistics.html> (we still need to write KDocs for it :sweat_smile:)

It's a bit clearer when using the <https://kotlin.github.io/dataframe/extensionpropertiesapi.html|extension properties api>:

Basically,
•  `percentile(pct) { col1 and ... } or percentile(pct, "col1", ...)` returns a single _value_ calculated from the values of the provided column(s together)
• `percentileOf(pct) { row -&gt; row.col1 + 12 }` returns a single _value_ calculated from the results of the provided row expression.
• `percentileFor(pct) { col1 and ... } or percentileFor(pct, "col1", ...)` returns a _row_ containing the calculated percentile of each provided column individually
• `percentileBy(pct) { row -&gt; row.col1 } or percentileBy(pct, "col1")` returns a _row_ selected from the dataframe closest to where the estimated percentile of the provided row expression would lie.
So to use it in aggregation, the correct way to use `percentileBy` in your use case would be `percentileBy(80.0) { value }.value` or using the String API: `percentileBy(80.0, "value")["value"]`

thank you, btw -- i'm not sure if/why i completely missed that bit of the docs

It's a bit hidden, I admit. It will likely become clearer once we add it as KDocs too :)