https://kotlinlang.org logo
Join Slack
Powered by
I'm trying to use <https://github.com/robocorp/jav...
# compose-desktop
s
I'm trying to use https://github.com/robocorp/java-access-bridge-wrapper to do UI automation of a compose desktop application. However, running the sample code crashes the compose application with some IOOB-Exceptions. I'm pretty sure that external applications should not be able to crash mine just via an accessibility API. How could I fix this? stack trace and sample python code in comments due to length of text.
stack trace:
Copy code
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Exception in thread "Thread-2" java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 5
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7328)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7303)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge.getAccelerator(AccessBridge.java:3855)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge.getAccessibleKeyBindingsCount(AccessBridge.java:4021)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge.runDLL(Native Method)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$dllRunner.run(AccessBridge.java:230)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 5
	at java.base/java.util.Arrays$ArrayList.get(Arrays.java:4225)
	at androidx.compose.ui.platform.a11y.ComposeAccessible$ComposeAccessibleComponent.getAccessibleChild(ComposeAccessible.kt:338)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$141.call(AccessBridge.java:3862)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$141.call(AccessBridge.java:3855)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils$CallableWrapper.run(AccessBridge.java:7384)
	at java.desktop/java.awt.event.InvocationEvent.dispatch$$$capture(InvocationEvent.java:313)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java)
	at --- Async.Stack.Trace --- (captured by IntelliJ IDEA debugger)
	at java.desktop/java.awt.event.InvocationEvent.<init>(InvocationEvent.java:296)
	at java.desktop/java.awt.event.InvocationEvent.<init>(InvocationEvent.java:222)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7339)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7323)
	... 6 more
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Not implemented: getCharacterAttribute
Exception in thread "Thread-2" java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 47
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7328)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7303)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge.getAccelerator(AccessBridge.java:3855)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge.getAccessibleKeyBindingsCount(AccessBridge.java:4021)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge.runDLL(Native Method)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$dllRunner.run(AccessBridge.java:230)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 47
	at java.base/java.util.Arrays$ArrayList.get(Arrays.java:4225)
	at androidx.compose.ui.platform.a11y.ComposeAccessible$ComposeAccessibleComponent.getAccessibleChild(ComposeAccessible.kt:338)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$141.call(AccessBridge.java:3862)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$141.call(AccessBridge.java:3855)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils$CallableWrapper.run(AccessBridge.java:7384)
	at java.desktop/java.awt.event.InvocationEvent.dispatch$$$capture(InvocationEvent.java:313)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java)
	at --- Async.Stack.Trace --- (captured by IntelliJ IDEA debugger)
	at java.desktop/java.awt.event.InvocationEvent.<init>(InvocationEvent.java:296)
	at java.desktop/java.awt.event.InvocationEvent.<init>(InvocationEvent.java:222)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7339)
	at jdk.accessibility/com.sun.java.accessibility.internal.AccessBridge$InvocationUtils.invokeAndWait(AccessBridge.java:7323)
	... 6 more
sample code:
Copy code
from JABWrapper.jab_wrapper import JavaAccessBridgeWrapper
from JABWrapper.context_tree import ContextNode, ContextTree, SearchElement
import ctypes
import queue
import threading
import time

GetMessage = ctypes.windll.user32.GetMessageW
TranslateMessage = ctypes.windll.user32.TranslateMessage
DispatchMessage = ctypes.windll.user32.DispatchMessageW

def pump_background(pipe: queue.Queue):
    try:
        jab_wrapper = JavaAccessBridgeWrapper()
        print(jab_wrapper)
        pipe.put(jab_wrapper)
        message = ctypes.byref(ctypes.wintypes.MSG())
        while GetMessage(message, 0, 0, 0) > 0:
            TranslateMessage(message)
            print("Dispatching msg={}".format(repr(message)))
            DispatchMessage(message)
    except Exception as err:
        pipe.put(None)

def pump_foreground():
    try:
        jab_wrapper = JavaAccessBridgeWrapper()
        print(jab_wrapper)
        message = ctypes.byref(ctypes.wintypes.MSG())
        while GetMessage(message, 0, 0, 0) > 0:
            TranslateMessage(message)
            print("Dispatching msg={}".format(repr(message)))
            DispatchMessage(message)
    except Exception as err:
        print(err)

def main():
    pipe = queue.Queue()
    thread = threading.Thread(target=pump_background, daemon=False, args=[pipe])
    thread.start()
    time.sleep(1)
    jab_wrapper = pipe.get()
    if not jab_wrapper:
        raise Exception("Failed to initialize Java Access Bridge Wrapper")
    time.sleep(0.1) # Wait until the initial messages are parsed, before accessing frames
    jab_wrapper.switch_window_by_title("Untitled")
    context_tree = ContextTree(jab_wrapper)

if __name__ == "__main__":
    main()
a
Please file a bug at https://youtrack.jetbrains.com/issues?u=1&amp;q=project:%20CMP Don’t forget to specify platform and JVM, and a reproducer.
7 Views
Open in Slack
PreviousNext
Powered by