UIJRT | United International Journal for Research & Technology | Volume 02, Issue 08, 2021

API Validation Testing Using Karate

Shashwati Jha1, Srinivasula Katigari2, S.B. Prapulla3 and Ramakanth Kumar4
1
Student, Department of Computer Science, RV College of Engineering, India
2
Applications Tech Lead, LBrands Mast Global, India
3
Assistant Professor, Department of Computer Science, RV College of Engineering, India
4
Professor & HOD, Department of Computer Science, RV College of Engineering, Bangalore, India
Email: 1shashwatijha.cs17@[Link]

Abstract — Functional Automation is a widely adopted In this paper, we detail the development of an API
domain in the evergreen field of testing and is poised to testing framework using Karate Domain Specific
grow at an average growth rate of more than 15% from Language (DSL). The organization of this paper is as
2020 to 2027. As testing is the most time and cost follows; In Section 3, relevant information about the
intensive phase of a project, there is an active effort to framework is provided for clearer understanding. In
develop a framework which can reduce associated costs section 4, an overview of API automation testing and
and increase time efficiency of both, development, and Karate DSL is supplied; Section 5 provides system
deployment. Karate is one such open source automation architecture followed by section 6 which describes the
framework developed in 2017 which can be used for functioning of each module in system architecture.
API testing, performance testing and UI automation. It Section 7 details the methodology of the framework; In
combines the advantages of a neutral language that's Section 8, the results and analysis of framework is
easy to understand by even a non-developer with detailed and Finally, Section 9 carries the conclusion.
powerful assertions and inbuilt multithreading. In this
The following section provides a brief description of
paper, we detail the development of a Karate framework
previous work in the domain of automation testing.
for API testing and analyze its performance in sequential
and parallel execution and compare and contrast Karate II. STATE OF ART
with the popular Cucumber BDD (Behavior Driven In the paper [1] it is conveyed that while manual testing
Development) framework. had helped to refine the requirements as per user stories,
Keywords— API (Application Programming Interface), one aspect it didn’t perform well in was regression
Automation Testing, Cucumber, Feature File, testing as Manual testing via a graphical user interface
Functional Validation, Karate, Scenario. also missed these bugs as it was conducted for newer
features and testing regressively was neglected. Here,
I. INTRODUCTION the process of automating user stories was also
In an Agile Project Development Environment, testing documented, and a cost vs effort estimation was made
of the project at hand to validate its functionality, for automation.
performance and stability is a necessary albeit time
Automation testing is adopted by industries as testing is
consuming task prone to Budgetary and Time
a cost and resource intensive process. In The research
constraints as well as human operator error. Successive
article [2] the authors discuss proposed software’s time
development cycles necessitate execution of similar
to market, by measuring cost of project in terms of the
testcases and performing validations on common APIs
COCOMO model (Constructive Cost Model) cost in
(Application Programming Interface). Using a test
terms of effort per person/month and as a majority of
automation framework along with DevOps Tools, it's
software’s cost lies in testing, automation of such tests
possible to Script a test suite and execute it at regular
can give good return in the long term. Ramler et al in the
timeframes.
paper [3] documented the benefits of automation testing.
With a growing adoption of microservice architecture, Similar to paper [2], automation and manual testing
the development and use of API’s are more were compared to reduce costs and a cost-based model
commonplace. API’s or Application Programming was discussed to reduce overall costs of development.
Interfaces serve as an interface between a backend data
These Proposed improvements over time have
store such as a database or cloud and a front-end
persuaded organizations to further employ automation.
software where data is requested and manipulated by an
To enable automation, Various tools exist with most
end-user. To assure system reliability and functionality,
popular one being Cucumber Framework.
we need to ensure an API works as per requirement and
does not send invalid or corrupted data to the user. In the article [4], the Authors discuss a prosed solution
of automation using Cucumber and Junit, in back end

All rights are reserved by [Link]. 105

 UIJRT | United International Journal for Research & Technology | Volume 02, Issue 08, 2021

automation which explores the viability of Cucumber Maven: Maven is often used from the command line and
framework as an automation tool. As The project aimed it serves to make the building process easier by shielding
to use Karate which has the feature of multithreading to developers from knowing about underlying mechanism.
speed up execution, in the research [5] the author details It also provides runtime information about the build,
a method to execute A BDD framework in parallel by thus increasing quality by creating change logs, creating
using selenium grid, which allows flexibility in reports, and installing dependencies. Dependencies are
choosing tests to execute, and reducing batch processing specified in a [Link] file which is built before
time. It utilizes the Cucumber framework to make test execution.
cases readable by even a domain expert who may not be
API (Application Programming Interface): API’s are
technically skilled.
software tools that are used to pass messages between
In the paper [6] the authors have discussed a way to two applications and to interact with these systems.
implement performance tests on a microservice, where
API status code: Every API upon execution returns a
the aim was to develop a microservice which be
Status code. Status codes in the range 200-299 depict a
integrated without difficulty making minor changes to
successful request. Status codes in 400-499 and 500-599
application code.
represent client and server-side errors respectively.
In [7], the author discusses the various tools available
for automation testing such as Selenium, Test Complete IV. OVERVIEW OF API TESTING USING
etc. thus providing a good overview of the advantages KARATE
of each tool. Finally, Test automation has made APIs or Endpoints serve as a message transfer link
significant process over time from using record and play between back-end and front-end systems. When using
techniques to now using tools such as Cucumber and APIs over a server, we need to ensure that the API is
Junit that support developing automation scripts at code functioning as per contract.
level, the paper [8] highlights these improvements and This entails checking an APIs return status which must
also discusses a technique to make automation testing be 200 for success and between the range of 400-500 for
intelligent via the use of AI to simulate rules and create Failure. Also, we need to ensure that the response we
a knowledge base of an agent. obtain from the API in a positive scenario must always
Automation testing has wide scope and reduces costs match a contract.
while significantly increasing performance. The studied A contract is a binding schema of the API that must
papers reflect the innovations to this domain and provide always be followed. For example, if we specify that in a
an insight into various techniques to carry out response body with fields Name, ID, and Date, that the
automation. However, no work has been done to carry name and ID must be in string format, and the Date must
out a review of Automation processes using Karate be within a specified range and that the ID is a
DSL, a newer tool which can reduce development time mandatory field.
while still maintaining powerful assertion capabilities of
Then each response must contain an ID field that has a
java.
string format, and whenever name and Date are passed,
III. RELEVANT INFORMATION they too must be passed in the specified format.
Cucumber: Cucumber is a Behavior Driven Organizations need to regularly check the performance
Development (BDD) tool which utilizes natural and functionality of APIs managed by them.
language syntax to create test scripts that are easy for a
product owner or business analyst to understand. Scripts Using Manual testing, it would be cumbersome to verify
are created in an easy given, when, then syntax which and validate multiple endpoints with each having
follows a natural language. However, for most step complex multi-lined responses. Here, automation testing
definitions, Back end code must be supplied in Java to serves as the best approach to testing.
automate these steps. Karate is a modern framework that has the capability to
Karate: Karate is bult on top of the cucumber assert complex schema matching and perform Endpoint
framework and this has the language neutrality syntax of testing in a short span of time using multiple threads.
cucumber, however it does not require any step Karate has a simple-to-use syntax that allows developers
definitions to be written in java unlike cucumber and has to write test cases faster.
ease of compatibility with third party software. Which
This speedup coupled with the speedup in execution due
makes using this framework to do API validation tests
to parallelism makes Karate a lightweight framework to
simple, powerful, and flexible.
deploy cases over a cloud regularly.

All rights are reserved by [Link]. 106

 UIJRT | United International Journal for Research & Technology | Volume 02, Issue 08, 2021

V. SYSTEM DESIGN 5. Scenarios: Scenarios contain steps in Given,

when, then format as shown in Fig. 2. These are of
positive, negative and contract valuation format.

6. API gateway: API gateway serves as an interface

between a client store and the front-end system at
hand. It transfers requests to and from a data store
or a server.

7. Target Reports: When Execution has completed,

Reports are generated by test runner which are
stored in target. Reports are created in various
formats, the most common ones being Docker
Fig.1: System Design of Karate framework Images, Cucumber reports and Surefire reports.
Docker Images contain code, dependencies etc.
Figure 1 illustrates the system design of a Karate and are used to make an executable image,
Framework for API Test Automation; the following Cucumber reports and Surefire reports contain
section describes the functions of each module. documentation of each step.

VI. MODULE DESCRIPTION

8. Docker Images: Docker creates an image of the
entire project and is useful for execution over
integrated systems and cloud systems.

9. Additionally, a Maven jar file is created on

execution of jar file which is used to reduce code
Fig.2: Sample code snippet for API Validation
duplication.
using Karate syntax
VII. METHODOLOGY
1. End User/ Automated Execution: An end user or a
pipeline starts executing the program by
specifying tags and number of threads. Tags depict
the function of the scenario. These tags can later
be used to selectively order a scenario or feature
file for execution.

2. Page Object Model: The page Object model file,

which contains all the dependencies required by
the program, installs these dependencies, and
equips local system for execution.

3. Test runner: Test Runner is a java file that is akin

to a main file. It is used to specify execution order
using Hooks. It calls the feature file or group of
feature files as specified by tags, and once
execution has returned from feature files, it calls
the report generator using a ‘@after’ hook which
generates the reports into the target folder.

4. Feature Files: Feature files consist of multiple

scenarios. These are used to combine those
Fig.3 Flow of Development and Execution of
scenarios of positive, negative and contract
Karate Framework
validation which call a common API. These also
contain some background steps which are
executed before each scenario.

All rights are reserved by [Link]. 107

 UIJRT | United International Journal for Research & Technology | Volume 02, Issue 08, 2021

Initially framework dependencies are identified and a

Maven [Link] (Page Object Model) is set up. This @KarateOptions(tags =
contains all dependencies used by the system. Upon {“@Test1,@test2”,”@Positive”,"~@ignore"})
execution the [Link] file will execute first and will
install any dependencies and configurations required by Follows the logical sequence of:
the program. ((@Test1 OR @Test2) AND @Positive AND NOT
@ignore)
Post Setup of the Karate framework the APIs were
identified and chosen. Similar endpoints which have a  “@Test1,@Test2”: Any combination of tags within
common access pattern form an API collection and one double quotes (“*”)run in OR configuration. That
such collection was chosen for the project. means any scenario with either tag @Test1 or
@Test2 will be executed.
For Each API within the collection a feature file was
created. Each feature file should contain positive  “@Test1,@Test2”,”@Positive”:Any combina-tion
validation scenarios, negative validation scenarios and a of tags separated by double quotes and a comma(,)
schema validation scenario as well as background run in AND configuration. Here, any scenario
information. marked with @Test1 OR @Test2 AND @positive
will be run.
The background information will contain common
information that is executed before each scenario. For  "~@ignore": Any scenario having associated
the positive validation scenarios, the application will ~(NOT) Tag will Not be executed.
match a 200-status code when the correct configurations
of path and query parameters and payload are sent. In Further, Parallelism is inbuilt into Karate and the
the negative validation scenario, the scenario will match number of tags can be specified ranging from 1 which is
a status code in the range of 400 to 500 on passing wrong sequential execution to multiple.
or empty parameters.
A limit on the number of threads will be user’s system
However, if a 200-status code is returned, the program processor and ability to simultaneously multithread.
must check if the response payload is empty. This will Other software and hardware factors will also limit the
also constitute a negative validation case. In the schema degree of parallelism.
validation case, the program matches the obtained
responses schema against the required schema known as Upon execution, Target Reports are built in Cucumber
contract. For better documentation and understanding, and Surefire formats. These reports detail each step-in
both scenario and feature files are named according to execution, the time taken to execute each step and if any
their purpose. step has passed, failed or been skipped.

Also, tags can be given to each scenario and feature file They also show the response that was received after
which specify the scenarios function and make execution of an API.
execution easier. Tag ‘@Timezone’ has been shown in
figure 2, the tag describes the scenario which is carrying VIII. RESULTS AND ANALYSIS
out positive validation for the world time API. Table. I: Tabular data of Time vs Feature files
No. of Time (sec) to Complete Execution
Finally, upon development of feature files a feature Threads Threads Threads Threads
[Link] file is created This file serves as a files =1 =2 =4 =6
central point of execution. It is in this file that the user 1 10.97 6.8 5.49 6.54
can specify which feature file or scenario must be run or
2 14.42 8.83 6.67 7.01
not run by using tags. A user can also specify a
collection of feature files to be run. 3 18.83 11.09 8.76 7.64
5 30.36 18.25 11.76 10.58
For example, if the user wishes to not execute any 10 58.12 31.78 18.38 14.57
scenario marked with ‘@ignore’, they must specify
‘~@ignore’ in the [Link]. Tags follow logical 20 103.75 52.1 29.6 20.65
operators hence when tags are supplied in the following 40 199.26 100.11 53.29 36.68
pattern:

All rights are reserved by [Link]. 108

 UIJRT | United International Journal for Research & Technology | Volume 02, Issue 08, 2021

supports neutral language but requires

corresponding steps to be written in Java. This
reduces code complexity and development time.

 Assertions: To match schemas, Karate has powerful

inbuilt assertion which can match complex
schemas. In cucumber, assertions have to be done
through parsing ‘*.yaml’ and ‘*.json’ files.

IX. CONCLUSIONS
Chart.1: Linear graph of No. of Feature Files vs. The primary motive for automation testing is to reduce
Time(sec) time in and increase software output. Both these goals
were achieved as well as increasing software quality and
reliability by automating tests that evaluate these
services.
In Conclusion, Karate DSL demonstrated good
performance when analyzed with varying number of test
cases. The reports Karate generated contain execution
details of each step as well as the overall percentage of
test cases that have passed vs failed which helps in
gauging a quick idea of how many services are up and
running. it also had the advantage of being easy to use
which allowed us to rapidly automate and test each
endpoints functionality and Schema and hence
Chart.2: Log10 Graph of No. of Feature Files vs. decreases both, test production time as well as test
Time(sec) deployment time.
The Framework was run with varying number of Feature
files and Threads. The following section contains the REFERENCES
[1] C. Klammer and R. Ramler, "A Journey from
data used to execute and its time performance.
Manual Testing to Automated Test Generation in an
Chart.1 describes a graph plotted with data from
Industry Project," 2017 IEEE International
Table.1. It is Observed that utilizing 2 threads decreases
Conference on Software Quality, Reliability and
the run time by half of using 1 thread. It is also observed
Security Companion (QRS-C), 2017, pp. 591-592,
that using 6 threads on a 4-core system (intel i5) resulted
doi: 10.1109/QRS-C.2017.108.
in a 5-factor speedup.
[2] Divya Kumar, K.K. Mishra, “The Impacts of Test
Chart.2 Describes the same data but with logarithmic Automation on Software's Cost, Quality and Time
scale. Here. Difference in execution of 1 feature file can to Market”, Procedia Computer Science, Volume
be visualized. Running 1 feature file with 6 threads took 79,2016,Pages 8-15,ISSN 1877-
more time that running 1 feature file with 4 threads. This 0509,[Link]
is due to added latency by 2 extra threads on a 4-core [3] Rudolf Ramler and Klaus Wolfmaier. 2006.
system. “Economic perspectives in test automation:
balancing automated and manual testing with
Some notable advantages of Karate framework exist opportunity cost”. In Proceedings of the 2006
over Cucumber Framework; these have been detailed international workshop on Automation of software
below: test (AST '06). Association for Computing
Machinery, New York, NY, USA, 85–91. DOI:
 Parallelism: Karate supported inbuilt parallelism [Link]
whereas Cucumber framework requires 3rd party [4] Mr Tarik Sheth , Ms. Priyanka Bugade , Ms. Sneha,
tools to support threading. Overall, Karate is a Pokharkar, Analysis Of Code Coverage Through
flexible and lightweight tool to implement faster Gui Test Automation And Back End Test
development Automation, IJISET - International Journal of
Innovative Science, Engineering & Technology,
 Step Definition: Karate requires step definitions Vol. 3 Issue 3, March 2016. ISSN 2348 – 7968
only in neutral language whereas Cucumber

All rights are reserved by [Link]. 109

 UIJRT | United International Journal for Research & Technology | Volume 02, Issue 08, 2021

[5] R. Anand, ArulPrakash Ma, Business driven

automation testing framework, March 2018,
International Journal of Engineering & Technology
7(2.8):345, DOI: 10.14419/ijet.v7i2.8.10438
[6] André de Camargo, Ivan Salvadori,Ronaldo dos
Santos Mello, Frank Siqueira, An architecture to
automate performance tests on microservices,
iiWAS '16: Proceedings of the 18th International
Conference on Information Integration and Web-
based Applications and Services November 2016
Pages 422–429 [Link]
3011141.3011179
[7] Neha Bhateja, A Study on Various Software
Automation Testing Tools, International Journal of
Advanced Research in Computer Science and
Software Engineering, Volume 5, Issue 6, June
2015, ISSN: 2277 128X
[8] Li, J...J., Ulrich, A., Bai, X. et al. Advances in test
automation for software with special focus on
artificial intelligence and machine learning.
Software Qual J 28, 245–248 (2020).
[Link]
[9] Contan, C. Dehelean and L. Miclea, "Test
automation pyramid from theory to practice," 2018
IEEE International Conference on Automation,
Quality and Testing, Robotics (AQTR),2018,pp.1-
5,doi: 10.1109/AQTR.2018-.8402699.
[10] Z. Sun, Y. Zhang and Y. Yan, "A Web Testing
Platform Based on Hybrid Automated Testing
Framework," 2019 IEEE 4th Advanced Information
Technology, Electronic and Automation Control
Conference (IAEAC), 2019, pp. 689-692, doi:
10.1109/IAEAC47372.2019.8997684.
[11] M. Iyama, H. Kirinuki, H. Tanno and T.
Kurabayashi, "Automatically Generating Test
Scripts for GUI Testing," 2018 IEEE International
Conference on Software Testing, Verification and
Validation Workshops (ICSTW), 2018, pp. 146-
150, doi: 10.1109/ICSTW.2018.00043.
[12] V. H. Kiranagi and G. K. Shyam, "Feature Driven
Hybrid Test Automation Framework (FDHTAF)
for web based or cloud based application testing,"
2017 International Conference On Smart
Technologies For Smart Nation (SmartTechCon),
2017, pp. 1555-1559, doi:
10.1109/SmartTechCon.2017.8358626.
[13] K. Sneha and G. M. Malle, "Research on software
testing techniques and software automation testing
tools," 2017 International Conference on Energy,
Communication, Data Analytics and Soft
Computing (ICECDS), 2017, pp. 77-81, doi:
10.1109/ICECDS.2017.8389562

All rights are reserved by [Link]. 110