Everyone's benchmarking this year, but I'd really ...

# advent-of-code

[Jakub Gwóźdź]

Everyone's benchmarking this year, but I'd really like to see a proper algorithm benchmarks.
So: not on how much time a solution will take when run ten thousand times on same input, but rather how much time will it take when run once on a ten thousand times larger input.
Of course such input data would have to be manually crafted separately for each day, but maybe it's worth the effort?

[ephemient]

that would be interesting but difficult to formulate, especially in some later puzzles start running into integer size limitations

[ephemient]

there's usually a few puzzle constructors on reddit every year though

[Thiago]

I was trying since first day but I'm not an expert in benchmark at all and I don't know whether in the right direction using Kotlin Bencharmk.

And I was checking what best algorithm to the solution in time spent only.

[ephemient]

JMH supports running a benchmark with varying @Params, which can be used to chart out the behavior across different input sizes (for example). kotlinx.benchmark exposes the same feature

[ephemient]

for example, you could write a benchmark like

@State(Scope.Benchmark)
class Day1Bench {
    @Param("small.txt", "medium.txt", "large.txt")
    lateinit var filename: String

    lateinit var input: String

    @Setup
    fun setup() {
        input = File(filename).readText()
    }

    @Benchmark
    fun part1() = solveDay1Part1(input)

    @Benchmark
    fun part2() = solveDay1Part2(input)
}

[Neil Banman]

Ephemient’s an old hand at this; he helped me out a couple years ago. Honestly I got it working but it was more trouble than it was worth for these small puzzles. Maybe the process can be automated or streamlined but for me it always took a few minutes to set up. Not to mention puzzling through scores of poorly documented Gradle settings.

[Endre Deak]

Isn't this a good problem for an LLM? Sharing your puzzle input and asking to generate similar inputs but with larger size? Although there might be some distortion that the LLM will use kind of false assumptions based on your specific input. Also there is the copyright and ethics related to AOC.