Depends on how your remoteVideoView is implemented. By default SurfaceView content is shown behind the View hierarchy. The SurfaceView itself draws a hole punch, effectively clearing pixels to reveal the surface that has the video content behind the rest of the UI. It might be the case that your remoteVideoView itself has z-above which means the video content is shown on top of the rest of the UI making it invisible despite your box/column ordering.
While using TextureView would work around this issue, there are a few limitations here which include increased resource utilization and battery consumption. The Android team as been pushing for developers to favor SurfaceView usage over TextureView for these reasons. Additionally newer display tech (ex. HDR rendering/adaptive refresh rate) can only be supported through SurfaceView not TextureView