Desktop Activities

Version: 1.0.9  ·  ID: zoan-packages-desktop  ·  Author: Zoan Software

The Desktop package automates Windows desktop applications: installed programs such as an ERP, an accounting system, a desktop banking application or any Windows app. The bot works with their windows and controls —buttons, fields, lists, tables— just like a person with the keyboard and mouse.

Under the hood it uses UI Automation (UIA3), the Windows accessibility technology, which lets it reliably “see” an application’s controls (not by screen coordinates, but by the actual element). It also supports Java desktop applications (Swing).

Desktop or browser?

• For systems used in the browser (web pages, portals), use the Browser package.
• For programs installed on Windows (desktop windows), use this package.

Many concepts mirror Browser: there is a session, selectors, and activities to click, type, read and wait.

Needs a Windows desktop

Every activity in this package works on an interactive desktop: in unattended runs, the robot gets one through a robot session. Zoan Cloud’s headless robots have no desktop, and the Designer hides these activities in projects for that platform.

Concept 1: the window and the session

To automate an app you first connect to its window. There are two pieces:

• Open App: launches the application (if it is not open) and returns its process identifier (PID).
• Win Attach: connects to a window and returns a session (ZoanDesktopSession).

As in Browser, there are two ways to work with the session:

Option A — Win Attach with body

The activities inside body inherit the session automatically; you don’t set session on any of them.

Win Attach   selector = (window "Calculator")
 └─ body:
      Win Click   selector = (button "5")
      Win Click   selector = (button "+")
      Win Get Text   selector = (display)   → output: result

Option B — Session in a variable

Win Attach stores the session in a variable; each activity receives it in session, and at the end you close it with Win Close.

Win Attach   selector = (window "ERP")   → output: app
Win Type   session = app   selector = (field "User")   text = "admin"
Win Close  session = app

session is optional

Unlike Browser, in Desktop the session parameter is optional in element activities. If you leave it out (and you are not inside a Win Attach body), the selector must include the window node (wnd, which the visual picker captures). With a session, selectors are shorter because the window is already fixed. For flows with several actions on the same window, a session is more convenient and robust.

With a session, the search happens inside its window

When there is a session (the one in session or the one inherited from body), the selector’s wnd node is ignored and the control is searched inside the session’s window. To act on another top-level window (for example, the main window that appears after a login), close the body and connect to it with another Win Attach.

Concept 2: desktop selectors

As in the browser, a selector identifies the control to act on (this button, that field). You capture it with UI Discover by pointing at the element with the mouse; Zoan Automation stores a robust reference based on the control’s UIA properties (its controlType, name, automationId…), which survives changes in the window’s position or size.

A desktop selector is a list of nodes from the window down to the control:

Node	What it represents	Attributes used when searching
wnd	The application’s top-level window	process (process name, with or without .exe), title (window title), cls (window class)
ctrl	Each control from the window down to the target	controlType, automationId, name, cls, idx (position among siblings, starting at 1)

How it is resolved:

• Windows: only visible windows are considered; the first one that matches every attribute of the wnd node wins.
• Value matching: case-insensitive; supports * and ? wildcards (Invoice *) and regular expressions between slashes (/^Order-\d+$/).
• Dynamic values: write {{ expression }} inside a value (for example Row {{ index }}) and it is resolved at run time.
• No guessing: if several controls match and nothing tells them apart, or none satisfies the anchors, the activity does not pick one at random: it fails and the diagnostics say why (add an anchor or an idx).
• Retries: the search repeats every 250 ms until timeout runs out, including after momentary UI Automation errors (an app busy opening a dialog).

Read this first

Two dedicated pages explain selectors in depth:

• Selectors — attributes, wildcards, anchors and good practices to locate controls without a name of their own.
• UI Discover › Desktop mode — how to capture controls (including Java apps), pause to navigate the app, and validate.

In this documentation, when we write selector = (button "Save") we mean “the Save button”, which you will have captured with UI Discover.

Java applications

When Win Attach connects to a Java application, it also connects its own Swing bridge (waiting up to 30 s for it to be ready) so it can see the controls even when Windows does not expose them. The machine needs a Java with the jdk.attach module; if the bridge cannot connect, the log says so and Java Access Bridge is tried instead.

Concept 3: interaction methods (UIA vs physical)

Several activities offer two ways to act on a control, and it helps to understand the difference:

Action	”UIA” method (default)	“Physical” method
Win Click	invoke — on buttons, activates them without moving the mouse (a button that opens a dialog does not block the flow). On other controls it makes a real click.	mouse — moves the mouse and makes a real click.
Win Type	value — sets the text through the UIA Value pattern. Fast and safe.	simulate — focuses the field and simulates keystrokes.

Start with the default method (invoke/value): it is more reliable. Switch to the physical one (mouse/simulate) only if a “stubborn” control does not respond to UIA (some custom controls, or fields that validate every key).

Before acting, the interaction activities (click, type, check, select, scroll) bring the window to the front and, if the control is scrolled out of view, try to scroll it into view; if they can’t, they fail instead of acting on something else. In Java applications, clicking, typing and selecting are always physical.

Concept 4: waits and timeouts

• Finding the element: each activity waits up to timeout ms for its element to exist. If you leave it empty, it uses the UI timeout from the project settings or, if the project doesn’t set one, 30000 ms.
• Instant checks: Win Element Exists does not wait by default (0); give it a timeout if you want it to wait.
• Bringing the window to the front: Win Focus waits up to 15000 ms (or the project’s timeout); Win Key Press with a selector, up to 3 s.
• Opening an application: Open App gives Windows 30000 ms (or the project’s timeout, if longer).
• Delays: the project’s delays before and after each action apply to every element activity.

To wait for something to appear or disappear, use Win Wait For instead of a fixed Delay.

Common errors in element activities

These messages are shared by every activity that searches for a control. Each activity’s own messages are on its page.

Message	Cause and fix
Element not found with selector {…}. followed by Diagnostics: …	The element did not appear within timeout. The diagnostics say at which node it was lost. Check that the window is open, validate the selector in UI Discover, or wait first with Win Wait For.
… elements match the selector and nothing tells them apart, so none was chosen (not guessing). Add an anchor or an index in the selector. (in the diagnostics)	The selector is ambiguous. Add an automationId, an idx or an anchor.
'selector' is required and must be a valid desktop selector.	The selector is missing or is not a desktop one (for example, a browser selector). Capture it again with UI Discover.
The selector must contain a 'wnd' node when there is no session.	Without a session, the selector needs the window. Use the activity inside a Win Attach or pass session.
The element is outside the visible area and could not be scrolled into view, …	The control is hidden by its container’s scrolling. Add a Win Scroll before it.
The Java control could not be located unambiguously, so nothing was done. …	In a Java app, the bridge could not tell the control apart. Capture the selector again or add an anchor.

To retry after momentary failures, wrap the steps in a Retry or a Try / Catch.

Activities

Launch, connect and recover

Activity	Type	Output	What it does
Open App	open-app	int	Launches an application and returns its PID
Kill Process	kill-process	int	Terminates processes by name or PID (a hung app)
Win Attach	win-attach	ZoanDesktopSession	Connects to a window
Win Focus	win-focus	—	Brings a window to the front
Win Close	win-close	—	Closes the window and releases the session

Interact

Activity	Type	Output	What it does
Win Click	win-click	—	Clicks an element
Win Type	win-type	—	Types text into a field
Win Set Value	win-set-value	—	Sets a field’s value (Value pattern)
Win Check	win-check	—	Checks/unchecks a checkbox
Win Select	win-select	—	Selects an item in a combo box/list
Win Key Press	win-key-press	—	Sends a key, a combination or a sequence of keys
Win Scroll	win-scroll	—	Scrolls an element

Read data

Activity	Type	Output	What it does
Win Get Text	win-get-text	string	Text or value of an element
Win Get Attribute	win-get-attribute	object	A property of the element (enabled, value…)
Win List Items	win-list-items	string[]	Available items of a combo box/list
Win Element Exists	win-element-exists	boolean	Does the element exist?
Win Extract Table	win-extract-table	DataTable	Extracts a table from a window into a DataTable

Wait and capture

Activity	Type	Output	What it does
Win Wait For	win-wait-for	boolean	Waits for an element to appear or disappear
Win Screenshot	win-screenshot	—	Captures the window or an element to a file
Take Screenshot	take-screenshot	string	Captures the whole desktop to a file and returns its path

A typical flow

Open a system, log in and read a value:

Open App   path = "C:/ERP/erp.exe"   → output: pid
Win Attach   selector = (window "ERP — Login")
 └─ body:
      Win Type   selector = (field "User")       text = credential("erp")["user"]
      Win Type   selector = (field "Password")   text = credential("erp")["password"]
      Win Click  selector = (button "Sign in")
Win Attach   selector = (window "ERP — Home")      // waits up to timeout for it to appear
 └─ body:
      Win Get Text   selector = (label "Balance")   → output: balance

And if the application hangs, the usual recovery is to keep evidence with Take Screenshot, terminate it with Kill Process and open it again. The built-in templates already include that pattern in their Recuperar playbook.

Next steps

• Open App and Win Attach — the starting point.
• Browser — the equivalent package for the web (parallel concepts).
• Win Wait For — key to reliable desktop automations.