Skip to main content

Image Automation

Overview

Clicknium automation is integrated with many image automation functions. Image detection is the best choice when it is hard to locate UI elements with attributes.

Capture image locator

Open a project in Visual Studio Code by pressing Ctrl + F10 or clicking the button Capture to invoke Clicknium recorder.
recorder button

  • Hover the mouse over the element to highlight it.
  • Press Ctrl and click the element to capture its locator.
  • Press Shift and select one area by the mouse to capture the image. The locator will be stored as an image locator.

recorder helper

For example

  • Choose the target element

recorder sample1

  • Press Shift and drag to select an area

recorder sample1

You will get the locator as following:
recorder sample1

The image locator contains two parts.
The first part is the anchor element. In the running process, first, find the anchor element, then capture the anchor image depending on the image method attributes, and find the target image in the anchor image by image matching algorithm. The second part is image itself with the following attributes:

NameDescriptionequalscontainsstartWithendWith
accuracyThe minimum similarity between the target image and the one in image locator. It is useful when the image to be found is slightly different from the one in the locator. Measurement unit is between 0 and 1 with default value 0.75.
Yes
No
No
No
dpiwe recommend the DPI settings of Windows OS in the process is the same as the ones when recording
Yes
No
No
No
methodenum values define the algorithm of image matching. The value is autoset when recording
Yes
No
No
No
filePaththe selection area of image recording, used in the process of image matching
Yes
No
No
No
matchIndexwhich will be selected when more than one area is found during image matching, default is 0, it means the first matching area
Yes
No
No
No
elementRectstore the anchor element area during recording. No need to modify the value
Yes
No
No
No
selectionRectstore the selection area during recording. No need to modify the value
Yes
No
No
No
timeoutSpecify the maximum time interval in image matching, default is 5000 milliseconds
Yes
No
No
No

the definition of methods:

  • HighestAccuracy: compare the images found in whole Windows screen.
  • InRegionHighestAccuracy: compare the images found in anchor element area.
  • OutRegionHighestAccuracy: comapre the images found out of the anchor element area.

method attributes are automatically set based on the selection area and corresponding anchor area in the process of recording If the selection area is inside the anchor element area, set to 'InRegionHighestAccuracy';
If the selection area intersects with the anchor element area, set to 'HighestAccuracy'; If the selection area is outside the anchor element area, set to 'OutRegionHighestAccuracy';

Use image locator in project

Image locator can be used in the same way as other locators, for example

from clicknium import clicknium as cc, locator, ui


#open new browser window
driver = cc.chrome.open("https://www.bing.com")
driver.find_element(locator.chrome.img1).click

ui(locator.notepad.menuitem).click()

The following functions support image locator:

  • click
  • double_click
  • mouse_up
  • mouse_down
  • drag_drop
  • get_position
  • get_size
  • highlight
  • hover
  • send_hotkey
  • set_text (parameter by should be sendkey-after-click)