Worked around this by simply not aligning it to the center of the LinearLayout, and letting it match_parent on the width and then wrapped the whole composable in a box to have the box take the size that the ComposeView is given, and then on the child (which for my case was an Image) I had full control of where in the given Box I could place it, and I picked Center.

composeView.setContent {
  Box { // This box is the size given from XML, match_parent for width in my case
    AsyncImage(..., Modifier.align(Alignment.Center).heightIn(x.dp).padding(horizontal = number_to_align_to_what_I_had_in_xml))
  }
}

It’d still be nice to know why I had this problem in the first place and if it’s something that’s just gonna happen all the time. If yes this might hurt the migration story of some people. I might have to report a bug if nobody points out that I may have done an obvious mistake.