AFAIK, it is possible. You need to extract the gesture modifier and pass it as a function parameter to your child composable. Somethink like:
UserCard(Modifier.pointerInput())
UserCard(modifier: Modifier = Modifier) {
Column(modifier = modifier.size().background())
}