Oh, the culprit was BoxWithConstraints that wrapped a custom layout. This composable defers composition until the layout phase. This means that other composables readings its alignment lines during layout will fail because it has not yet initialized its alignment lines.
At least this is what I think caused the issue.