YTEP-0007: Automatic Pull Requests’ validation

Abstract

Created: February 21, 2013 Updated: January 25, 2015 Author: Kacper Kowalik

This YTEP describes framework used to automatically run both unit and answer testing for incoming pull requests to main YT repository.

Methods proposed here need to be agnostic with respect to chosen continuous integration system.

Status

Completed

CI server is running at http://tests.yt-project.org. Scripts used in validation process are stored in http://bitbucket.org/xarthisius/yt-validation.

Detailed Description

Background

When a new pull request is issued there is no way to automatically test validity of proposed changes. Every author is forced to run the tests manually, which requires: knowledge of how to perform the tests, downloading required data, possessing significant free computing power. This is troublesome and results in tests being run less frequently than they are supposed to be. With implementation of this YTEP the responsibility of running the testsuite will be shifted from PR’s author to designated, automatic infrastructure.

Required Features

The CI infrastructure will need to:

  • Constantly poll for incoming changes to main repository (or react to POST message if required API will become available)
  • Run both unit and answer testsuite.
  • Notify the author should the tests fail.
  • Notify the author should the PR could not be cleanly merged.
  • Inform people responsible for accepting PR that all tests have passed by sending mail to yt-svn@lists.spacepope.org or broadcasting on #yt irc channel.
  • Results of tests should be publicly available.

Implementation Details

Each PR will pulled and merged with current tip in prepared docker container that consists of yt’s dependencies: yt_analysis/devenv. Resulting container will be tagged as yt_analysis/yt-PR#:commit_hash and will be available for download from public docker registry for developers who wish to test changes locally. Subsequently, yt_analysis/yt-PR#:commit_hash will be used to run both unit and answer tests on CI server. If PR contains commits that modify or add files to doc/ subdirectory in the main yt tree, full documentation build will be performed and its result will be stored at http://tests.yt-project.org.

Base container yt_analysis/devenv will be created using install_script.sh and updated every time changes to aforementioned script is merged to yt branch. Additional containers with development environment based on bleeding edge Linux distributions, such as: Gentoo, Debian Sid, Fedora Rawhide, will available and denoted by appropriate tag: :gentoo, :sid, :rawhide respectively. Testsuite will be run on those containers periodically in order to detect incompatibilities with newer versions of yt’s dependencies.

Dockerfiles for all containers will be a part of yt_analysis/yt repository.

Stumbling Blocks

CI described in this YTEP does not cover integration tests that should be performed on OSX or Windows, nor alternative installations using e.g. wheels, conda. Since CI server is a part of NCSA’s infrastructure, access allowing to modify existing and creating new tests will be restricted to people present in NCSA’s LDAP.

Backwards Compatibility

There are no backwards compatibility issues.