CSE3209 – Robotics Process Automation(RPA)
School of Computer Science and Engineering and Information Science
Module 3: Intelligent Automation
Data Manipulation, Automation of Virtual Machines, Introduction to
Native Citrix Automation, Text and Image Automation, PDF
Automation, Computer Vision, Programming, Debugging, Error
Handling, Logging, Extensions, Project Organization
2
What is intelligent automation?
• Intelligent automation is software technology that incorporates AI
and ML to automate complex workflows.
• It can be perceived as an extension of the traditional RPA, geared to
broaden its capabilities.
• When combined with AI, RPA takes on a whole new dimension of
possibilities.
• It can even automate tasks that previously required human
involvement, like document understanding and unstructured data
analysis.
• In the simplest sense, intelligent automation can be broken down into
three different components: AI, RPA, and business process
management (BPM).
•
3
Components of intelligent automation
AI and ML
• Process mining and task mining analyze your processes to
determine which steps should be automated.
RPA
• You can easily build, deploy, and manage software robots that
mimic human interactions with digital systems.
BPM
• You can create complex automations that connect different
departments, systems, and apps. This enables you to see where
automations can best be applied to maximize your productivity.
4
Steps
Discover
• Before automations are created or employed, you first need to figure out which
processes can and should be automated.
Automate
• Once suitable processes have been identified, stakeholders can create the
necessary automations to enact them.
• Operate
• Once you’ve created and tested your new automations, you're ready to
implement them into the appropriate workflows. These automations provide a
foundation on which you can continuously build and deploy new automations,
enterprise-wide.
5
Data Manipulation
• Data manipulation is the process through which the data is altered
using various operations in order to facilitate its usage.
6
• Data manipulation allows the user to change data from one form to
another by concatenating, converting, extracting, splitting, or
segregating to make it ready to be used by RPA in workflows.
• Example: A bulk data (such as a financial database) could be
organized in alphabetical order, making individual entries easier to
locate. Data manipulation provides to extract only the relevant data
and use it in other documents or correlate it with information from
other sources.
7
8
Data Conversion
9
String Manipulation
10
[Link] :-
• Concatenates the string representations of two specified
objects.
Expression: [Link] (VarName1, VarName2)
[Link]:
13
Replace :-
• This activity allows you to replace specific characters or
substrings within a string with a new value. It is commonly used
to remove unwanted characters or replace certain patterns in a
string
[Link]
• Splits a string into substrings based on certain criteria set by the user.
This criterion could be a space, comma, or full stop
[Link](“|“c)(index)
15
Substring
• Substring: Extracts a substring from a string using the starting index
and the length. It is used to isolate or separate a substring from the
original string • Expression: [Link](startIndex, length)
16
17
[Link]
• To convert a string to Uppercase. It is a string method that helps in
performing case-insensitive operations on strings
18
[Link]
• To convert a string to lowercase. It is a string method that helps in
performing case-insensitive operations on strings
20
Automation of Virtual Machines
• Automation of virtual machines (VMs) using UiPath RPA (Robotic Process Automation)
involves creating workflows that interact with VM environments as if a human were performing
the tasks.
• UiPath can control VM-based applications through its built-in activities, screen scraping, image
recognition, and hotkey features.
22
What is Virtual Machine?
• These are the machines where memory or storage is borrowed from a physical
host computer or server.
• Host is the actual machine
• VM provides an image of actual set up.
• All set ups and installation done on client side.
23
24
25
26
Image and Text Automation
• Text and image automation in UiPath involves creating workflows that can
interact with various UI elements, read and write text, and recognize images to
simulate user actions or extract information.
27
Text Automation
• Text automation in UiPath is about reading from and writing to different applications, including
web pages, PDFs, Word documents, and desktop apps.
Key Activities for Text Automation:
• Read Text:
• “Get Text”: Extracts text from a specific UI element.
• “Read PDF Text” and “Read PDF with OCR”: Extracts text from PDF documents, with
OCR being used for scanned or image-based PDFs.
• “Screen Scraping”: Provides structured text extraction using FullText, Native, or OCR
methods.
28
• Write Text:“Type Into”: Simulates keyboard typing to input text into fields.
• “Set Text”: Directly sets text into a UI field (if supported by the app).
• “Write Line” and “Log Message”: Used to display text in the output panel for debugging
purposes.
• “Write to CSV”, “Append to CSV”, and “Write Range” for writing data into Excel or CSV
files.
29
Image Automation
Key Activities for Image Automation:
• “Click Image”: Searches for an image on the screen and clicks it when found. Useful for button
presses or navigation in non-selectable UIs.
• “Find Image”: Checks if a specific image is present on the screen and returns its position.
• “Image Exists”: Determines whether an image is visible on the screen and returns a Boolean
value.
• “Click OCR Text” and “Get OCR Text”: Used for interacting with and extracting text that is
embedded in images, leveraging OCR engines like Google OCR or Tesseract OCR.
30
• “Take Screenshot”: Captures the current screen or a specific area and
saves it as an image file.“Hover Image”: Moves the mouse over a
specified image.
Example Use Cases:
• Reading Text from an Application:
• Use “Get Text” to extract data from an order form or report.
• Save the extracted text into an Excel sheet using “Write Range”.
• Automating Button Clicks in a VM:
• Use “Click Image” to click a button within a remote desktop session
where the selector cannot be accessed directly.
• Use “Image Exists” to verify that the button is present before clicking it,
ensuring reliability.
31
• Extracting Information from an Image-Based PDF:
• Use “Read PDF with OCR” to extract text from scanned documents.
• If accuracy is a concern, tweak the OCR engine's settings or try different OCR options for better
results.
• Form-Filling:
• Use “Type Into” or “Set Text” to input information into forms.
• Automate filling and submitting forms by combining text automation with image-based clicks.
32
Extracting Text from Images Using the UiPath Screen OCR
Activity
Diagram illustrating the OCR process from image to text extraction.
33
[Link] Package
[Link] [Link]:
1. Open UiPath Studio.
2. Go to Manage Packages.
3. Search for [Link]
4. install it.
34
OutPut
Input Image
35
Extract text from any Image
36
PDF Automation
Table 1. The activities available in the [Link] package
Activities Description
Get PDF Page Count Provides the total number of pages in a PDF file.
Read PDF Text Reads all characters from a specified PDF file and stores them in a string
variable.
Read PDF With OCR Reads all characters from a specified PDF file and stores it in a string variable
by using OCR technology.
Export PDF Page As Image Creates an image from a page in a specified PDF file.
Extract Images From PDF Extracts images from a specified PDF file.
Extract PDF Page Range Extracts a specified range of pages from a PDF document.
Join PDF Files Joins multiple PDF files stored in an array of strings into a single PDF file.
Manage PDF Password Changes the password of a specified PDF file.
37
Read OCRPDF & Text extraction
38
AI Computer Vision
• AI Computer Vision is an AI Skill
that enables all UiPath Robots to
see and understand every
element on a computer screen.
• Now, businesses and RPA
developers can automate tasks
on most virtual desktop interface
(VDI) environments – regardless
of framework or operating
system
39
Computer Vision Recorder
• UiPath Computer Vision Recorder is a tool that is available in
the Computer Vision activities package. It is used to create
workflows that make advantage of the capabilities of the CV
activities automatically.
40
• CV Check •CV Get Text
• CV Click •CV Highlight
• CV Dropdown Select •CV Hover
• CV Element Exists •CV Screen Scope
• CV Extract Table •CV Type Into
41
• CV Click
• UIPath Computer Vision neural network targets a certain UI element, which is
clicked.
• CV Element Exists
• uses the UiPath Computer Vision neural network to look for a specific UI
element on the screen in the foreground and then returns a Boolean variable
that indicates whether or not it was found.
42
• CV Get Text
• Users can extract text from a
chosen UI element with the help of
“CV Get Text“. Users who use this
activity can choose their preferred
technique for getting the target text
from a drop-down menu.
• For automating the extraction of
text data from UI components like
scanned documents or web pages,
CV Get Text provides a quick and
effective solution.
43
• CV Highlight
• You may use this activity to visually draw
attention to a particular UI component
inside your automation workflows.
• After the element has been recognized,
the activity will highlight it in a way that
makes it visually stand out, making it
simple for you to validate that the
relevant element has been targeted. This
activity can help you When you need to
verify whether the selected UI Element is
the right one or not.
44
• CV Hover
• UiPath Computer Vision Hover
activity allows you to
interact with a targeted UI
element by hovering over it.
45
• CV Type Into
• CV Type into activity is used
to type any specific text or
values into a text box or
window. These activities
are mostly used in data
entry jobs where the data
form excel has to be read
and add it to the data
entry form.
46
• CV Refresh
• A CV refresh refers to refreshing
the scope of an application or
a window. During the design
process, you may manually
accomplish this by using the
Refresh button in the activity’s
body.
47
• CV Dropdown Select
• chooses a particular item from a drop-down menu.
• CV Check
• checkboxes can be selected, cleared, or toggled. Only check boxes can
communicate with this activity.
• CV Extract Table
• takes a screen-visible table and extracts it, then stores it in a DataTable
variable.
48
Debugging
• Debugging in simple terms is the process of identifying and removing
errors from the project.
49
50
Command Shortcut Action When to Use
Executes current step and To inspect inside a
Step Into F11
enters inner logic workflow or container
Executes current step and To skip internal details of
Step Over F10
moves to next known activities
Executes remaining inner
To exit a nested scope
Step Out Shift + F11 steps and returns to
quickly
parent
Runs until next To resume normal
Continue F5
breakpoint or end execution
51
Debug Section:
• Steps are used to execute your project step by step. So, when you click on Step Into, it
executes the next step and then it waits.
• Validate button is used to validate your project and check if you have any errors or not. So,
when you choose this option, UiPath will check if your automation has any errors and if it has
any errors it will return you the error.
• Breakpoints are the points at which you want to stop the execution and start debugging
step by step. The breakpoints button offers two options:
• Toggle Breakpoints
• Remove All Breakpoints
• Slow Step slows down your execution so that, you get a track of what is happening.
• Options provide various highlighting options to highlight the activities. So, you can use this
when you want to highlight any activity while you are debugging your project.
52
Extensions
• UiPath extensions help you broaden your automation capabilities to
web browsers, Java applications, Silverlight applications, and Citrix
Virtual Apps and Desktops, as well as over RDP connections, and
more.
53
• Chrome, Firefox , Microsoft Edge
• Java
• Citrix - also requires the UiPath Remote Runtime installed on the Citrix
Application server.
• Windows Remote Desktop - also requires the UiPath Remote
Runtime installed on the Windows server.
• VMware Horizon - also requires the UiPath Remote Runtime installed on
the VMware Horizon Application server.
• SAP Solution Manager plugin
• Excel Add-in
54
Exception Handling
55
Exception Handling
• Exception Handling mainly deals with handling errors with respect to
various activities in UiPath. The Error Handling activity offers four
options: Rethrow, Terminate Workflow, Throw, Try Catch.
• Rethrow is used when you want activities to occur before the
exception is thrown.
• Terminate workflow is used to terminate the workflow the moment
the task encounters an error.
56
• Throw activity is used when you want to throw error before the
execution of the step.
• Try Catch activity is used when you want to test something and handle
the exception accordingly.
• try
• catch
• Finally
57
Try Catch
58
59
60