The 19th International Workshop on Mutation Analysis
Co-located with ICST
28th May 2024
Tell Me More

About Mutation 2024

Mutation analysis involves mutation of software artifacts that are then used to evaluate the quality of software verification tools and techniques. It is considered the premier technique for evaluating the fault revealing effectiveness of test suites, test generation techniques and other testing approaches. Ideas derived from mutation analysis have also been used to test artifacts at different levels of abstraction, including requirements, formal specifications, models, architectural design notations and even informal descriptions. Recently, mutation has played an important role in software engineering for AI, such as in verifying learned models and behaviors. Furthermore, researchers and practitioners have investigated diverse forms of mutation, such as training data or test data mutation, in combination with metamorphic testing to evaluate model performance in machine learning and detecting adversarial examples.

Mutation 2024 is the 19th in the series of international workshops focusing on mutation analysis. The workshop will be co-located with the International Conference on Software Testing, Verification, and Validation (ICST 2024). Accepted papers will be published as part of the ICST proceedings.

The Mutation workshop aims to be the premier forum for practitioners and researchers to discuss recent advances in the area and propose new research directions. We invite submission of full and short length research papers, full and short length industry papers, and ‘Hot Off the Press’ presentations.

Call for Papers

Call for Papers

**NOTICE (24 Jan 2024)**: To give a bit of extra time to authors, we decided to have a soft deadline and allow the authors to update their submissions by **Feb 04** only if they made initial submissions before **Feb 01**. Please, remember to register your paper and submit the title and abstract before **Feb 01**.

Mutation 2024 aims to be the premier forum for practitioners and researchers to discuss recent advances in the area of mutation analysis and propose new research directions. We invite submissions of full and short length research papers, full and short length industry papers, and ‘Hot Off the Press’ presentations.

Topics of Interest

Topics of interest include, but are not limited to, the following:

  • Evaluation of mutation-based test adequacy criteria, and comparative studies with other test adequacy criteria.
  • Formal theoretical analysis of mutation testing.
  • Empirical studies on any aspects of mutation testing.
  • Mutation based generation of program variants.
  • Higher-order mutation testing.
  • Mutation testing tools.
  • Mutation for mobile, internet, and cloud based systems (e.g., addressing QoS, power consumption, stress testing, performance, etc.).
  • Mutation for security and reliability.
  • Novel mutation testing applications, and mutation testing in novel domains.
  • Industrial experience with mutation testing.
  • Mutation for artificial intelligence (e.g., data mutation, model mutation, mutation-based test data generation, etc.)

Types of Submissions

Five types of submissions can be made to the workshop:

  • Full research papers (10 pages including references): Research, case studies.
  • Short research papers (6 pages including references): Research in progress, tools.
  • Full industrial papers (6 pages including references): Applications and lessons learned in industry.
  • Short industrial papers (2 pages including references): Mutation testing in practice reports.
  • Hot Off the Press presentations (1 page abstract): Presentation of work recently published in other venues and relevant to the workshop.

Each paper must conform to the two columns IEEE conference publication format (please use the letter format template and conference option) and must be submitted in PDF format via EasyChair. Submissions will be evaluated according to the relevance and originality of the work and to their ability to generate discussions between the participants of the workshop. Each submission will be reviewed by three reviewers, and all accepted papers will be published as part of the ICST proceedings (except for Hot Off the Press submissions).

Mutation 2024 will employ a double-anonymous review process (except for Hot Off the Press and industry submissions). Authors must make every effort to anonymise their papers to hide their identities throughout the review process, except for Hot Off the Press submissions. See the double-anonymous QnA page for more information.

Industry paper

Industry papers should be given the keyword "industry" and are not subject to the double anonymity policy.

Hot Off the Press

Hot Off the Press submissions should contain 1) a short summary of the paper's contribution, 2) an explanation of why those results are particularly interesting for MUTATION attendees, 3) a link to the paper. Their title should start with "HOP:" The original paper should be published no earlier than January 1st 2022.

Important Dates

  • Submission deadline (extended): Sun 04 Feb 2024
  • Notification of acceptance: Mon 26 Feb 2024
  • Camera-ready: Fri 1 Mar 2024
  • Workshop date: 28th May 2024

Download: CfPmarkdown (txt)


Program Chairs

Renzo Degiovanni
SnT, University of Luxembourg
Thomas Laurent
SFI Lero, Trinity College Dublin

Program Committee

Manar Alafi
Toronto Metropolitan University (Canada)
Pedro Delgado-Perez
University of Cadiz (Spain)
Lin Deng
Towson University (USA)
Alex Denisov
GitHub Inc. (Germany)
Jinhan Kim
Università della Svizzera italiana (Switzerland)
Miloš Ojdanić
University of Luxembourg (Luxembourg)
Goran Petrovic
Google (USA)
Donghwan Shin
University of Sheffield (UK)
Thierry Titcheu Chekam
SES (Luxembourg)
Serge Demeyer
University of Antwerp (Belgium)
Aayush Garg
University of Luxembourg (Luxembourg)
Fabiano Ferrari
Federal University of São Carlos (Brazil)


Jie M. Zhang

Lecturer/Assistant Profesor
Kings College London


(GMT+1 Time)




Keynote: Mutation for AI and with AI

Prof. Jie M. Zhang


Session 1

(Chair) Renzo Degiovanni

Mutant-Kraken: A Mutation Testing Tool for Kotlin

Josue Morales, Lin Deng, Josh Dehlinger, Suranjan Chakraborty

Timed Model-Based Mutation Operators for Simulink Models

Jian Chen, Manar Alalfi, Thomas Dean

Improving the Efficacy of Testing Scientific Software: Insights from Mutation Testing

Kris Roker, Upulee Kanewala


Lunch (90 min)


Session 2

(Chair) Renzo Degiovanni

Test Harness Mutilation

Samuel Moelius

An Empirical Evaluation of Manually Created Equivalent Mutants

Philipp Straubinger, Alexander Degenhart, Gordon Fraser

A Study of Flaky Failure De-Duplication to Identify Unreliably Killed Mutants

Abdulrahman Alshammari, Paul Ammann, Michael Hilton, Jonathan Bell


Best paper award, dicussion and closing

(Chair) Renzo Degiovanni