If I understand correctly it is working for the axis because you control the Axis’s State. However, those commands that has been painted already in the Canvas are the ones causing issues, because you don’t have/control their state.
That scenario is a bit more complex, there is always the possibility to apply transformations over an existing Canvas Bitmap. Anyways transforming something that you don’t know how it looks might change the original shape, aspect ratio an so on.
Other alternative as you said could be creating a custom ViewComponent class that contains the properties and data of the different objects that painted into this Canvas. You can keep a list of them internally. Let the class have 3 open methods .onMeasure(…) .onLayout(…) and .onDraw(…) and propagate them from CustomView accordingly.