Returning it in getValue along with the proper value seems to be annoying because I can't destruct a Pair at the top level of the class

You can keep the delegate in a separate field in your class, like this:

class MyClass {
    val delegate = myDelegate()
    val myField: Int by delegate
}

And then you can do whatever you want.