Windows GUI自动化控制工具之python uiAutomation

对 Windows GUI进行自动化控制的工具有很多，比如pywinauto、pyautogui、pywin32、Autoit、airtest、UIAutomation等，UI Automation API是微软提供的自动化框架，可在支持 Windows Presentation Foundation (WPF) 的所有操作系统上使用，支持的应用类型更多。本文介绍封装了UI Automation API的Python uiautomation 模块的使用方法。

目录

Python uiautomation 模块由yinkaisheng 开发，封装了微软 UI Automation API，支持自动化Win32，MFC，WPF，Modern UI(Metro UI), Qt, IE, Firefox, Chrome和基于Electron开发的应用程序。

环境准备

uiautomation安装

最新版uiautomation2.0只支持Python 3版本，但不要使用3.7.6和3.8.1这两个版本，因为comtypes包在这两个版本中不能正常工作。

pip安装uiautomation：

$ pip install uiautomation

检查是否安装成功：

$ pip list | findstr uiautomation
uiautomation       2.0.18

安装完成后，在Python的Scripts(我的路径为C:\Program Files\Python37\Scripts)目录中会有一个文件automation.py，是用来枚举控件树结构的一个脚本。

可运行

automation.py -h

查看命令帮助:

$ python automation.py -h
UIAutomation 2.0.18 (Python 3.7.2, 64 bit)
usage
-h      show commandhelp
-t      delay time, default 3 seconds, begin to enumerate after Value seconds, this must be an integer
        you can delay a few seconds and make a window active so automation can enumerate the active window
-d      enumerate tree depth, this must be an integer, if it is null, enumerate the whole tree
-r      enumerate from root:Desktop window, if it is null, enumerate from foreground window
-f      enumerate from focused control, if it is null, enumerate from foreground window
-c      enumerate the control under cursor, if depth is <0, enumerate from its ancestor up to depth
-a      show ancestors of the control under cursor
-n      show control full name, if it is null, show first 30 characters of control's name in console,
        always show full name in log file @AutomationLog.txt
-p      show process id of controls

if UnicodeError or LookupError occurred when printing,
try to change the active code page of console window by using chcp or see the log file @AutomationLog.txt
chcp, get current active code page
chcp 936, set active code page to gbk
chcp 65001, set active code page to utf-8

examples:
automation.py -t3
automation.py -t3 -r -d1 -m -n
automation.py -c -t3

进程查看器

对 Windows GUI进行自动化控制需要使用进程查看器工具对GUI界面元素进行定位，定位工具有很多，这里推荐使用微软提供的inspect.exe 或者 Accessibility Insights 这两款工具。

inspect.exe

inspect.exe 是 Windows SDK 自带的一个进程查看器，可以用来查看系统正在运行的进程信息、模块、线程、堆栈跟踪等详细数据。

Windows SDK下载地址为：https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/

建议直接到这里下载inspect.exe：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/tree/master/inspect

64位系统版本的inspect.exe也可以点击这里下载。

Accessibility Insights

Accessibility Insights 是微软开发的一款辅助功能测试工具。它可以帮助开发者测试 web 应用、Windows 桌面应用和 Android 应用的可访问性，确保这些应用程序符合无障碍标准。

Accessibility Insights获取的控件属性信息没有inspect.exe全面，使用起来更加流畅。下载为：https://accessibilityinsights.io/downloads/

控件对象模型

微软 UIAutomation API定义了支持的控件类型和对应的模型（Pattern），所有支持的控件类型可参考：https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpatternmapping
控件类型必须支持的模型可选模型Does not supportButtonNoneExpandCollapse, Invoke, Toggle, ValueNoneCalendarGrid, TableScroll, SelectionValueCheckBoxToggleNoneNoneEditNoneRangeValue, Text, ValueNoneListNoneGrid, MultipleView, Scroll, SelectionTableListItemSelectionItemCustomNavigation, ExpandCollapse, GridItem, Invoke, ScrollItem, Toggle, ValueNoneMenuNoneNoneNoneMenuBarNoneDock, ExpandCollapse, TransformNoneMenuItemNoneExpandCollapse, Invoke, SelectionItem, ToggleNoneRadioButtonSelectionItemNoneToggleSplitButtonExpandCollapse, InvokeNoneNoneTabSelectionScrollNoneTabItemSelectionItemNoneInvokeTableGrid, GridItem, Table, TableItemNoneNoneTextNoneGridItem, TableItem, TextValueTitleBarNoneNoneNoneToolBarNoneDock, ExpandCollapse, TransformNone
python uiautomation库对UIAutomation API定义的各个Control和Pattern进行了封装。

下面来看使用python uiautomation操作Windows自带计算器的例子。

uiautomation库示例

控制计算器

可以使用inspect.exe来定位计算器元素：

示例脚本如下：

import os
import uiautomation as auto
import subprocess

classuiautoCalc(Loggers):"""uiautomation控制计算器
    """def__init__(self):super().__init__()
        self.logger = Loggers().myLogger()
        auto.uiautomation.DEBUG_SEARCH_TIME =True 
        auto.uiautomation.SetGlobalSearchTimeout(2)# 设置全局搜索超时时间
        self.calcWindow = auto.WindowControl(searchDepth=1, Name='计算器', desc='计算器窗口')# 计算器窗口ifnot self.calcWindow.Exists(0,0):
            subprocess.Popen('calc')# 设置窗口前置
            self.calcWindow = auto.WindowControl(
            searchDepth=1, Name='计算器', desc='计算器窗口')
        self.calcWindow.SetActive()# 激活窗口
        self.calcWindow.SetTopmost(True)# 设置为顶层defgotoScientific(self):
        self.calcWindow.ButtonControl(AutomationId='TogglePaneButton', desc='打开导航').Click(waitTime=0.01)        
        self.calcWindow.ListItemControl(AutomationId='Scientific', desc='选择科学计算器').Click(waitTime=0.01)
        clearButton = self.calcWindow.ButtonControl(AutomationId='clearEntryButton', desc='点击CE清空输入')if clearButton.Exists(0,0):
            clearButton.Click(waitTime=0)else:
            self.calcWindow.ButtonControl(AutomationId='clearButton', desc='点击C清空输入').Click(waitTime=0.01)defgetKeyControl(self):
        automationId2key ={'num0Button':'0','num1Button':'1','num2Button':'2','num3Button':'3','num4Button':'4','num5Button':'5','num6Button':'6','num7Button':'7','num8Button':'8','num9Button':'9','decimalSeparatorButton':'.','plusButton':'+','minusButton':'-','multiplyButton':'*','divideButton':'/','equalButton':'=','openParenthesisButton':'(','closeParenthesisButton':')'}        
        calckeys = self.calcWindow.GroupControl(ClassName='LandmarkTarget')
        keyControl ={}for control, depth in auto.WalkControl(calckeys, maxDepth=3):if control.AutomationId in automationId2key:
                self.logger.info(control.AutomationId)
                keyControl[automationId2key[control.AutomationId]]= control
        return keyControl

    defcalculate(self, expression, keyControl):
        expression =''.join(expression.split())ifnot expression.endswith('='):
            expression +='='for char in expression:
                keyControl[char].Click(waitTime=0)
        self.calcWindow.SendKeys('{Ctrl}c', waitTime=0.1)return auto.GetClipboardText()defcalc_demo(self):"""计算器示例
        :return : 
        """        
        self.gotoScientific()# 选择科学计算器        
        keyControl = self.getKeyControl()# 获取按键控件
        result     = self.calculate('(1 + 2 - 3) * 4 / 5.6 - 7', keyControl)print('(1 + 2 - 3) * 4 / 5.6 - 7 =', result)
        self.calcWindow.CaptureToImage('calc.png', x=7, y=0, width=-14, height=-7)# 截图
        self.calcWindow.GetWindowPattern().Close()# 关闭计算机if __name__ =="__main__":
    ui = uiautoCalc()
    ui.calc_demo()

脚本执行动图：

参考文档

1. https://github.com/pywinauto/pywinauto
2. https://cloud.tencent.com/developer/article/2213048
3. https://github.com/yinkaisheng/Python-UIAutomation-for-Windows
4. Python UIAutomation文档：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/blob/master/readme_cn.md
5. https://www.cnblogs.com/Yinkaisheng/p/3444132.html
6. GitHub - jacexh/pyautoit: Python binding for AutoItX3.dll
7. GitHub - mhammond/pywin32: Python for Windows (pywin32) Extensions
8. Accessibility tools - Inspect - Win32 apps | Microsoft Learn
9. Accessibility Insights

--THE END--