https://www.mdu.se/

mdu.sePublications
1234563 of 6
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Exploring API Behaviours by Example Generation
Mälardalen University, School of Innovation, Design and Engineering.ORCID iD: 0000-0003-0342-7715
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Understanding the behaviour of complex software-intensive systems is a hard task. For developers of such systems, understanding the actual behaviours is critical in order to successfully create, extend, and maintain them.

The goal of the work in this thesis is to support explorations of the behaviour of software systems through their APIs. We fulfil this goal by generating examples of behaviours the system exhibits. An example is expressed as a sequence of API operations---with parameters, if required---that conforms to a specific behaviour.

Examples of behaviours, such as sequences of operations performed on the system, have been shown to be a good way to further the understanding of software systems for both end users and developers. However, manually creating examples requires effort. In addition, manually created examples only contain what a human can imagine---which might miss important cases, such as unintended behaviours.

The main proposed approach in this thesis is to support users in exploring the behaviour of their software system by automatically generating examples of actual behaviour. By only interacting with the system by the exposed API, we assess the behaviours as exposed to an end user of the API. The input to the approach is a set of API operations and schema of operation parameters. Sequences of operations are generated containing these provided operations. The observed responses from executing the generated sequences are used to assess if the API show an example of a sought behaviour. Found examples go through a shrinking process---trying to find a more minimal sequence showing the same behaviour---and are then reported to the user of the approach. 

The approach is capable of both generating examples of faults in the system and of generating examples of general behaviours. We show evidence of this through multiple evaluations. We have evaluated the fault-finding capabilities by generating examples producing fault-indicating error codes and showing how the configuration of generators affects the interaction with the system. In addition, we evaluate the capability of the approach to generate relevant examples, both in the general API case and in the specific case of REST APIs. By conducting multiple focus group sessions, we conclude that the examples of behaviours produced by the approach indeed aid industry practitioners. The generated examples are deemed relevant for use cases such as testing, documenting, and understanding the behaviour of the system.

Place, publisher, year, edition, pages
Mälardalens universitet, 2024.
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 412
National Category
Software Engineering
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-67064ISBN: 978-91-7485-654-5 (print)OAI: oai:DiVA.org:mdh-67064DiVA, id: diva2:1863573
Public defence
2024-09-10, Zeta, Mälardalens universitet, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2024-05-31 Created: 2024-05-31 Last updated: 2024-06-13Bibliographically approved
List of papers
1. QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs
Open this publication in new window or tab >>QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs
2020 (English)In: Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020, Institute of Electrical and Electronics Engineers Inc. , 2020, p. 131-141, article id 9159071Conference paper, Published paper (Refereed)
Abstract [en]

RESTful APIs are an increasingly common way to expose software systems functionality and it is therefore of high interest to find methods to automatically test and verify such APIs. To lower the barrier for industry adoption, such methods need to be straightforward to use with a low effort. This paper introduces a method to explore the behaviour of a RESTful API. This is done by using automatic property-based tests produced from OpenAPI documents that describe the REST API under test. We describe how this method creates artifacts that can be leveraged both as property-based test generators and as a source of validation for results (i.e., as test oracles). Experimental results, on both industrial and open source services, indicate how this approach is a low effort way of finding real faults. Furthermore, it supports building additional knowledge about the system under test by automatically exposing misalignment of specification and implementation. Since the tests are generated from the OpenAPI document this method automatically evolves test cases as the REST API evolves. 

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc., 2020
Keywords
OpenAPI, Property-based testing, REST, Automatic test pattern generation, Open source software, Verification, Additional knowledge, Automatically test, Open sources, Property-based, Software systems, System under test, Test generations, Test oracles, Software testing
National Category
Computer and Information Sciences Embedded Systems
Identifiers
urn:nbn:se:mdh:diva-51327 (URN)10.1109/ICST46399.2020.00023 (DOI)000621170500015 ()2-s2.0-85091597984 (Scopus ID)9781728157771 (ISBN)
Conference
13th IEEE International Conference on Software Testing, Verification and Validation, ICST 2020, 23 March 2020 through 27 March 2020
Available from: 2020-10-08 Created: 2020-10-08 Last updated: 2024-05-31Bibliographically approved
2. Automatic Property-based Testing of GraphQL APIs
Open this publication in new window or tab >>Automatic Property-based Testing of GraphQL APIs
2021 (English)In: Proceedings - 2021 IEEE/ACM International Conference on Automation of Software Test, AST 2021, 2021Conference paper, Published paper (Refereed)
Abstract [en]

In recent years, GraphQL has become a popular way to expose web APIs. With its raise of adoption in industry, the quality of GraphQL APIs must be also assessed, as with any part of a software system, and preferably in an automated manner. However, there is currently a lack of methods to automatically generate tests to exercise GraphQL APIs.

In this paper, we propose a method for automatically producing GraphQL queries to test GraphQL APIs. This is achieved using a property-based approach to create a generator for queries based on the GraphQL schema of the system under test.

Our evaluation on a real world software system shows that this approach is both effective, in terms of finding real bugs, and efficient, as a complete schema can be covered in seconds. In addition, we evaluate the fault finding capability of the method when seeding known faults. 73% of the seeded faults were found, with room for improvements with regards to domain-specific behavior, a common oracle challenge in automatic test generation.

National Category
Software Engineering Computer Sciences
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-54267 (URN)10.1109/AST52587.2021.00009 (DOI)000695707800001 ()2-s2.0-85113790213 (Scopus ID)
Conference
2nd IEEE/ACM International Conference on Automation of Software Test, AST 2021, Virtual, Online, 20 May 2021
Available from: 2021-05-26 Created: 2021-05-26 Last updated: 2024-05-31Bibliographically approved
3. Exploring API behaviours through generated examples
Open this publication in new window or tab >>Exploring API behaviours through generated examples
Show others...
2024 (English)In: Software quality journal, ISSN 0963-9314, E-ISSN 1573-1367Article in journal (Refereed) Epub ahead of print
Abstract [en]

Understanding the behaviour of a system's API can be hard. Giving users access to relevant examples of how an API behaves has been shown to make this easier for them. In addition, such examples can be used to verify expected behaviour or identify unwanted behaviours. Methods for automatically generating examples have existed for a long time. However, state-of-the-art methods rely on either white-box information, such as source code, or on formal specifications of the system behaviour. But what if you do not have access to either? This may be the case, for example, when interacting with a third-party API. In this paper, we present an approach to automatically generate relevant examples of behaviours of an API, without requiring either source code or a formal specification of behaviour. Evaluation on an industry-grade REST API shows that our method can produce small and relevant examples that can help engineers to understand the system under exploration.

Place, publisher, year, edition, pages
SPRINGER, 2024
Keywords
Property-based testing, Examples, Automated testing, API testing, REST
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:mdh:diva-66548 (URN)10.1007/s11219-024-09668-2 (DOI)001206040400001 ()2-s2.0-85191065404 (Scopus ID)
Available from: 2024-05-08 Created: 2024-05-08 Last updated: 2024-05-31Bibliographically approved
4. Exploring Behaviours of RESTful APIs in an Industrial Setting
Open this publication in new window or tab >>Exploring Behaviours of RESTful APIs in an Industrial Setting
(English)Manuscript (preprint) (Other academic)
Abstract [en]

A common way of exposing functionality in contemporary systems is by providing a Web-API based on the REST API architectural guidelines. To describe REST APIs, the industry standard is currently OpenAPI-specifications. Test generation and fuzzing methods targeting OpenAPI-described REST APIs have been a very active research area in recent years. An open research challenge is to aid users in better understanding their API, in addition to finding faults and to cover all the code. In this paper, we address this challenge by proposing a set of behavioural properties, common to REST APIs, which are used to generate examples of behaviours that these APIs exhibit. These examples can be used both (i) to further the understanding of the API and (ii) as a source of automatic test cases. Our evaluation shows that our approach can generate examples deemed relevant for understanding the system and for a source of test generation by practitioners. In addition, we show that basing test generation on behavioural properties provides tests that are less dependent on the state of the system, while at the same time yielding a similar code coverage as state-of-the-art methods in REST API fuzzing in a given time limit.

National Category
Software Engineering
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-67061 (URN)
Available from: 2024-05-31 Created: 2024-05-31 Last updated: 2024-06-19Bibliographically approved

Open Access in DiVA

The full text will be freely available from 2024-08-20 08:00
Available from 2024-08-20 08:00

Authority records

Karlsson, Stefan

Search in DiVA

By author/editor
Karlsson, Stefan
By organisation
School of Innovation, Design and Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 516 hits
1234563 of 6
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf