I don't think there is a solution. The root of your problem is the <*> in the Map, which means all typing information is already lost. Using something like Map<KClass<T>, T> would be a solution, but that is not possible in Kotlin (see
this question on SO, which also discusses alternatives).
In the code you provided I don't see how the data is populated. If you control that part and can guarantee the type safety yourself just slap a
@Suppress("UNCHECKED_CAST")
on there. Otherwise either use as? with nullable return type or be fine with throwing exceptions.