Why cannot we change the x,y coordinates of composables with modifiers?

You can with the offset modifier

Please see this. The touch location and the object being dragged are different. I think this is a problem.

There's no code here, so you are asking to diagnose the problem in the dark

Here I am updating an offset state and using it with the offset() modifier. But, besides the first time I drag the object away from the original location, the pointer location is different from the offset.

Looks like the offset modifier needs to be first, before the pointerInput modifier

Yes it solved the problem. Can I have an explanation why this happens? I know orders matter, but here it's a bit tricky to explain.

In this case, the offset needs to be applied first because everything after is the UI element you want to move. You can paste this snippet into Studio and play around with the order of offset and pointerInput to see the different outcomes:

@Preview
@Composable
fun DragTest() {
    MaterialTheme {
        var offsetX by remember{ mutableStateOf(0f) }
        var offsetY by remember{ mutableStateOf(0f) }

        Surface {
            Box(Modifier.fillMaxSize()) {
                Canvas(
                    Modifier
                        .size(60.dp)
                        .border(4.dp, Color.Green)
                        .pointerInput(Unit) {
                            detectDragGestures{ change, dragAmount->
                                offsetX += dragAmount.x
                                offsetY += dragAmount.y
                            }
                        }
                        .offset { IntOffset(x = offsetX.roundToInt(), y = offsetY.roundToInt()) }
                        .padding(8.dp)
                        .size(100.dp)
                        .border(1.dp, Color.Blue)
                        .background(Color.Cyan)
                        .drawBehind {
                            drawCircle(Color.Red, size.minDimension/3)
                        }
                ) {}
            }
        }
    }
}