{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "initial_id",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 动手学大模型：GUI智能体构建"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e56a0ac7",
   "metadata": {},
   "source": [
    "## 本教程目标\n",
    "1.了解GUI智能体领域的技术路线和研究现状\n",
    "\n",
    "2.尝试基于开源模型Qwen2-VL-7B依托OS-Kairos数据集构建自适应人机交互GUI智能体的方法\n",
    "\n",
    "3.尝试基于构建的GUI智能体进行推理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5673d29",
   "metadata": {},
   "source": [
    "## 1.准备工作\n",
    "### 1.1 了解GUI智能体领域的技术路线和研究现状\n",
    "阅读教程：[[Slides](https://github.com/Lordog/dive-into-llms/blob/main/documents/chapter8/GUIagent.pdf)]\n",
    "\n",
    "### 1.2 了解什么是自适应人机交互GUI智能体\n",
    "参考论文：[[Paper](https://arxiv.org/abs/2503.16465)]\n",
    "\n",
    "### 1.3 数据集准备\n",
    "本教程以OS-Kairos为例，基于此工作开源的数据集构建并推理简单的GUI智能体。\n",
    "\n",
    "从[[Data](https://github.com/Wuzheng02/OS-Kairos)]的README.md文件中的下载链接下载数据集，并解压到环境中。\n",
    "\n",
    "数据格式示例如下：\n",
    "```json\n",
    "    {\n",
    "        \"task\": \"打开网易云音乐，搜索《Shape of You》，并播放这首歌。\",\n",
    "        \"image_path\": \"/data1/wuzh/cloud_music/images/1736614680.6518524_1.png\",\n",
    "        \"list\": [\n",
    "            \" 打开网易云音乐  \",\n",
    "            \" 点击首页顶部的搜索框  \",\n",
    "            \" 输入：Shape of You  \",\n",
    "            \" 选择正确的搜索结果  \",\n",
    "            \" 点击歌名以播放  \"\n",
    "        ],\n",
    "        \"now_step\": 1,\n",
    "        \"previous_actions\": [\n",
    "            \"CLICK <point>[[381,367]]</point>\"\n",
    "        ],\n",
    "        \"score\": 5,\n",
    "        \"osatlas_action\": \"CLICK <point>[[454,87]]</point>\",\n",
    "        \"teacher_action\": \"CLICK <point>[[500,100]]</point>\",\n",
    "        \"success\": false\n",
    "    },\n",
    "```\n",
    "### 1.4 模型准备\n",
    "从[[Model](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)]下载Qwen2-VL-7B模型权重。\n",
    "\n",
    "## 1.5 有监督微调代码准备\n",
    "本教程采用LLaMa-Factory对Qwen2-VL-7B进行有监督微调，因此先从[[Code](https://github.com/hiyouga/LLaMA-Factory/)]下载源码。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95ed05b",
   "metadata": {},
   "source": [
    "## 2.GUI智能体构建\n",
    "### 2.1 数据预处理\n",
    "将以下代码存储为get_sharpgpt.py，并向其填出Kairos_train.json和预计存储处理后的训练集json的路径，然后运行该文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5de387e9",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "with open('', 'r', encoding='utf-8') as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "\n",
    "preprocessed_data = []\n",
    "for item in data:\n",
    "    task = item.get(\"task\", \"\")\n",
    "    previous_actions = item.get(\"previous_actions\", \"\") \n",
    "    image_path = item.get(\"image_path\",\"\")\n",
    "    prompt_text = f\"\"\"\n",
    "    You are now operating in Executable Language Grounding mode. Your goal is to help users accomplish tasks by suggesting executable actions that best fit their needs and give a score. Your skill set includes both basic and custom actions:\n",
    "\n",
    "    1. Basic Actions\n",
    "    Basic actions are standardized and available across all platforms. They provide essential functionality and are defined with a specific format, ensuring consistency and reliability. \n",
    "    Basic Action 1: CLICK \n",
    "        - purpose: Click at the specified position.\n",
    "        - format: CLICK <point>[[x-axis, y-axis]]</point>\n",
    "        - example usage: CLICK <point>[[101, 872]]</point>\n",
    "\n",
    "    Basic Action 2: TYPE\n",
    "        - purpose: Enter specified text at the designated location.\n",
    "        - format: TYPE [input text]\n",
    "        - example usage: TYPE [Shanghai shopping mall]\n",
    "\n",
    "    Basic Action 3: SCROLL\n",
    "        - Purpose: SCROLL in the specified direction.\n",
    "        - Format: SCROLL [direction (UP/DOWN/LEFT/RIGHT)]\n",
    "        - Example Usage: SCROLL [UP]\n",
    "\n",
    "    2. Custom Actions\n",
    "    Custom actions are unique to each user's platform and environment. They allow for flexibility and adaptability, enabling the model to support new and unseen actions defined by users. These actions extend the functionality of the basic set, making the model more versatile and capable of handling specific tasks.\n",
    "\n",
    "    Custom Action 1: PRESS_BACK\n",
    "        - purpose: Press a back button to navigate to the previous screen.\n",
    "        - format: PRESS_BACK\n",
    "        - example usage: PRESS_BACK\n",
    "\n",
    "    Custom Action 2: PRESS_HOME\n",
    "        - purpose: Press a home button to navigate to the home page.\n",
    "        - format: PRESS_HOME\n",
    "        - example usage: PRESS_HOME\n",
    "\n",
    "    Custom Action 3: ENTER\n",
    "        - purpose: Press the enter button.\n",
    "        - format: ENTER\n",
    "        - example usage: ENTER\n",
    "\n",
    "    Custom Action 4: IMPOSSIBLE\n",
    "        - purpose: Indicate the task is impossible.\n",
    "        - format: IMPOSSIBLE\n",
    "        - example usage: IMPOSSIBLE\n",
    "\n",
    "    In most cases, task instructions are high-level and abstract. Carefully read the instruction and action history, then perform reasoning to determine the most appropriate next action.\n",
    "\n",
    "    And I hope you evaluate your action to be scored, giving it a score from 1 to 5. \n",
    "    A higher score indicates that you believe this action is more likely to accomplish the current goal for the given screenshot.\n",
    "    1 means you believe this action definitely cannot achieve the goal.\n",
    "    2 means you believe this action is very unlikely to achieve the goal.\n",
    "    3 means you believe this action has a certain chance of achieving the goal.\n",
    "    4 means you believe this action is very likely to achieve the goal.\n",
    "    5 means you believe this action will definitely achieve the goal.\n",
    "\n",
    "    And your final goal, previous actions, and associated screenshot are as follows:\n",
    "    Final goal: {task}\n",
    "    previous actions: {previous_actions}\n",
    "    screenshot:<image>\n",
    "    Your output must strictly follow the format below, and especially avoid using unnecessary quotation marks or other punctuation marks.(where osatlas action must be one of the action formats I provided and score must be 1 to 5):\n",
    "    action:\n",
    "    score:\n",
    "    \"\"\"\n",
    "\n",
    "    action = item.get(\"osatlas_action\", \"\")\n",
    "    score = item.get(\"score\", \"\")\n",
    "    preprocessed_item = {\n",
    "        \"messages\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": prompt_text,\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"assistant\",\n",
    "                \"content\": f\"action: {action}\\nscore:{score}\",\n",
    "            }\n",
    "        ],\n",
    "        \"images\":[image_path]\n",
    "    }\n",
    "    preprocessed_data.append(preprocessed_item)\n",
    "\n",
    "\n",
    "with open('', 'w', encoding='utf-8') as f:\n",
    "    json.dump(preprocessed_data, f, ensure_ascii=False, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29804be0",
   "metadata": {},
   "source": [
    "经过该步骤，我们将OS-Kairos的训练集成功转化为符合sharpgpt格式的数据，便于进行下一步训练，数据预处理完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31fa41b8",
   "metadata": {},
   "source": [
    "## 2.2 有监督微调\n",
    "在上一步中，我们已经得到适配LLaMa-Factory训练的格式的Kairos数据集。然后我们要修改LLaMa-Factory以注册数据集和配置训练信息。\n",
    "首先修改data/dataset_info.json，添加如下内容注册数据集：\n",
    "```json\n",
    "\"Karios\" :{\n",
    "  \"file_name\": \"Karios_qwenscore.json\",\n",
    "  \"formatting\": \"sharegpt\",\n",
    "  \"columns\": {\n",
    "    \"messages\": \"messages\",\n",
    "    \"images\": \"images\"\n",
    "  },\n",
    "  \"tags\": {\n",
    "    \"role_tag\": \"role\",\n",
    "    \"content_tag\": \"content\",\n",
    "    \"user_tag\": \"user\",\n",
    "    \"assistant_tag\": \"assistant\"\n",
    "  }\n",
    "},   \n",
    "```\n",
    "继续修改/examples/train_full/qwen2vl_full_sft.yaml来进行配置训练信息：\n",
    "```yaml\n",
    "### model\n",
    "model_name_or_path: \n",
    "stage: sft\n",
    "do_train: true\n",
    "finetuning_type: full\n",
    "deepspeed: examples/deepspeed/ds_z3_config.json\n",
    "\n",
    "### dataset\n",
    "dataset: Karios\n",
    "template: qwen2_vl\n",
    "cutoff_len: 4096\n",
    "max_samples: 9999999\n",
    "#max_samples: 999\n",
    "overwrite_cache: true\n",
    "preprocessing_num_workers: 4\n",
    "\n",
    "### output\n",
    "output_dir: \n",
    "logging_steps: 10\n",
    "save_steps: 60000\n",
    "plot_loss: true\n",
    "overwrite_output_dir: true\n",
    "\n",
    "### train\n",
    "per_device_train_batch_size: 2\n",
    "gradient_accumulation_steps: 2\n",
    "learning_rate: 1.0e-5\n",
    "num_train_epochs: 5.0\n",
    "lr_scheduler_type: cosine\n",
    "warmup_ratio: 0.1\n",
    "bf16: true\n",
    "ddp_timeout: 180000000\n",
    "\n",
    "### eval\n",
    "val_size: 0.1\n",
    "per_device_eval_batch_size: 1\n",
    "eval_strategy: steps\n",
    "eval_steps: 20000\n",
    "```\n",
    "以上是一个示例配置，model_name_or_path是Qwen2-VL-7B的路径，output_dir是存放断点和log的路径。\n",
    "配置好后即可配置推理，示例命令行：\n",
    "```\n",
    "CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 FORCE_TORCHRUN=1 llamafactory-cli train examples/train_full/qwen2vl_full_sft.yaml\n",
    "```\n",
    "此处至少要使用3张80GB的A100计算资源。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea7b6201",
   "metadata": {},
   "source": [
    "## 3.OS-Kairos推理验证\n",
    "训练完毕后，我们即可进行推理，推理也有很多种方法可以选择，可以自行通过transformers库和torch库构建推理代码，也可以采用LLaMa-Factory进行一键式推理，以下演示如何LLaMa-Factory进行一键式推理。\n",
    "首先要修改/examples/inference/qwen2_vl.yaml，一个示例如下\n",
    "```yaml\n",
    "model_name_or_path: \n",
    "template: qwen2_vl\n",
    "```\n",
    "其中model_name_or_path是训练后的断点路径。\n",
    "然后即可通过\n",
    "```\n",
    "CUDA_VISIBLE_DEVICES=0 FORCE_TORCHRUN=1 llamafactory-cli webchat examples/inference/qwen2_vl.yaml\n",
    "```\n",
    "进行一键式推理，你可以自己拿出OS-Kairos测试集的图片，或者个人手机的截图，结合get_sharpgpt.py中格式的文本prompt输入给智能体，OS-Karios推理后会给出它认为当前应该进行的action和对于这个action的置信度分数。置信度分数越低，则当前指令越超出OS-Kairos的能力边界，需要人类介入来进行人机交互。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}