Run JUnit tests in order

JUnit runs test methods in whatever order it wants, which is generally fine as long as all your test methods are independent of each other. This, though, may not be the case when you’re writing integration tests, where you could have a sequence of operations like

login(credentials);
requestProject("foo");
createNewItem("bar");
logout();

in a situation like this, you always want testLogout() to run after testLogin(), and createNewItem() to run after requestProject()!

Yes, you could group everything in a single test method, but it may become huge and very hard to maintain.

Quite surprisingly, JUnit doesn’t have a built-in solution for this. You can run test methods sorted by name using the @FixMethodOrder(MethodSorters.NAME_ASCENDING) tag, but then you need to artificially name your methods so that they appear in the order you want.

I found some alternative solutions in this stackoverflow question, but they require annotating your methods with tags to specify the order in which you want them to run. What I’d like them to do is just run in the same order as they appear in the source code for the test class. Among the answers, I just found this blog post that achieves the same result as I did, only it looks somewhat more complicated (it involves writing/including several classes).

My solution is fairly simple, but there’s 2 warnings:

  1. it uses Javassist, so if you don’t want to add libraries, there’s that
  2. it only works as long as you don’t have test classes that extend other test classes, and you don’t override all @Test-annotated methods in the subclass (I’ve never done that, but I guess as tests get complicated, you may have that); this can be fixed quite easily though, you just need to add the logic for what should come first according to your needs

On with the code!

You can grab the source straight from this pastebin, or copy/paste it from here (I added the MIT License, I think it should be the most permissive.. my intent is to say “do whatever the heck you want with this code”)

/*
 * Copyright (C) <2014> <Michele Bonazza>
 * 
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

/**
 * A test runner that runs tests according to their position in the source file
 * of the test class.
 * 
 * @author Michele Bonazza
 */
public class OrderedTestRunner extends BlockJUnit4ClassRunner {

    /**
     * Creates a new runner
     * 
     * @param clazz
     *            the class being tested
     * @throws InitializationError
     *             if something goes wrong
     */
    public OrderedTestRunner(Class<?> clazz) throws InitializationError {
        super(clazz);
    }

    /*
     * (non-Javadoc)
     * 
     * @see org.junit.runners.BlockJUnit4ClassRunner#computeTestMethods()
     */
    @Override
    protected List<FrameworkMethod> computeTestMethods() {
        // get all methods to be tested
        List<FrameworkMethod> toSort = super.computeTestMethods();

        if (toSort.isEmpty())
            return toSort;

        // a map containing <line_number, method>
        final Map<Integer, FrameworkMethod> testMethods = new TreeMap<>();

        // check that all methods here are declared in the same class, we don't
        // deal with test methods from superclasses that haven't been overridden
        Class<?> clazz = getDeclaringClass(toSort);
        if (clazz == null) {
            // fail explicitly
            System.err
                    .println("OrderedTestRunner can only run test classes that"
                            + " don't have test methods inherited from superclasses");
            return Collections.emptyList();
        }

        // use Javassist to figure out line numbers for methods
        ClassPool pool = ClassPool.getDefault();
        try {
            CtClass cc = pool.get(clazz.getName());
            // all methods in toSort are declared in the same class, we checked
            for (FrameworkMethod m : toSort) {
                String methodName = m.getName();
                CtMethod method = cc.getDeclaredMethod(methodName);
                testMethods.put(method.getMethodInfo().getLineNumber(0), m);
            }
        } catch (NotFoundException e) {
            e.printStackTrace();
        }

        return new ArrayList<>(testMethods.values());
    }

    private Class<?> getDeclaringClass(List<FrameworkMethod> methods) {
        // methods can't be empty, it's been checked
        Class<?> clazz = methods.get(0).getMethod().getDeclaringClass();

        for (int i = 1; i < methods.size(); i++) {
            if (!methods.get(i).getMethod().getDeclaringClass().equals(clazz)) {
                // they must be all in the same class
                return null;
            }
        }

        return clazz;
    }
}

to use this, you need to add Javassist to your classpath; if you have a Maven project, it’s incredibly easy to do so, just add this to your POM:

<dependency>
  <groupId>javassist</groupId>
  <artifactId>javassist</artifactId>
  <version>3.12.1.GA</version>
</dependency>

and annotate your JUnit test class with @RunWith(OrderedTestRunner.class), like this:

import wherever.you.put.OrderedTestRunner;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(OrderedTestRunner.class)
public class MyTestClass {

    @Test
    public void testZero() {
        System.out.println("test zero run");
    }

    @Test
    public void testOne() {
        System.out.println("test one run");
    }
}

That’s it!

About warning 2. above, in case you have something like MyTestClass extends BaseTestClass and you have methods annotated with @Test in BaseTestClass that aren’t overridden by MyTestClass, OrderedTestRunner will just fail printing the message you can see at line 50 above on System.err. I did this because I don’t think there’s a well-defined order in that case (should all methods from the super class run first? Should that go all the way up in the class hierarchy?), so you can adjust it to fit your particular needs.

Advertisements

Using reflection to unit-test private methods in Java

Everybody has a different opinion on whether you should write unit tests for private methods, and my own is that you should, unless said methods are very small procedures called by some public function, in which case you can treat the whole package as a black box.

Unfortunately, several classes contain code that is completely irrelevant to users of the class, or code that should never be called directly if not from within the class itself (because you’re proxying calls to control accesses to the class, or keeping stats, or demultiplexing calls from several classes, or whatever). The purpose of unit-testing is to help maintain the code, and to help spotting bugs more quickly; it shouldn’t become self-referential and it definitely should not change the way you structure your classes, just for the sake of testability. Hence, private methods should be used whenever you don’t want to expose the internal workings of a class to outsiders, but at the same time they shouldn’t be left behind when writing tests, or they quickly become a source of subtle errors.

I like PowerMock‘s mockPrivate feature, and I use it a lot, but what I want to do here is to actually test the private method, not replace it with a mock! (I know it should be obvious, but when using powermock very often you start to think of it as the solution to all your problems)

Reflection comes in handy: you set methods accessibility flag to true and you can call them! You can also change the value of private fields, or call private constructors. I mostly use these 3 functions, so I came up with a simple class that I use in all my Java JUnit-driven projects.

In the spirit of my previous post on some of my favorite Java utility methods, I’ll post here this class, hoping that some of you may find it useful:

/**
 * ReflectionUtils.java 
 * Created on Apr 30, 2012
 * Copyright 2012 mb
 * <https://somethingididnotknow.wordpress.com>
 * 
 * This program is free software; you can redistribute it and/or modify it under
 * the terms of the GNU General Public License as published by the Free Software
 * Foundation; either version 2 of the License, or (at your option) any later
 * version.
 * 
 * This program is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
 * details.
 * 
 * You should have received a copy of the GNU General Public License along with
 * this program; if not, write to the Free Software Foundation, Inc., 51
 * Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
 */

import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.Arrays;

/**
 * Some utility methods that can be used for unit tests to alter internal
 * objects states by exploiting features from <tt>java.lang.reflect</tt>
 * package.
 * 
 * @author mb
 */
public class ReflectionUtils {

    /**
     * Sets the value of the <tt>static</tt> field named <tt>fieldName</tt> in
     * class <tt>clazz</tt> to <tt>newValue</tt> and returns <code>true</code>
     * if the operation was successful.
     * 
     * @param clazz
     *            the class whose field is to be changed
     * @param fieldName
     *            the (case-sensitive) name of the field whose content is to be
     *            changed
     * @param newValue
     *            the new value that the field should store
     * @return <code>true</code> if the new value has been set,
     *         <code>false</code> otherwise
     */
    public static boolean changeStaticField(Class<?> clazz, String fieldName,
            Object newValue) {
        if (clazz == null)
            return false;
        return change(clazz, null, fieldName, newValue);
    }

    private static boolean change(Class<?> clazz, Object object,
            String fieldName, Object newValue) {
        boolean success = false;
        try {
            Field toChange = clazz.getDeclaredField(fieldName);
            toChange.setAccessible(true);
            toChange.set(object, newValue);
            success = true;
        } catch (NoSuchFieldException e) {
            e.printStackTrace();
        } catch (SecurityException e) {
            e.printStackTrace();
        } catch (IllegalArgumentException e) {
            e.printStackTrace();
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        }
        return success;
    }

    private static Object call(Class<?> clazz, Object object,
            String methodName, Object... args) throws IllegalArgumentException,
            InvocationTargetException {
        Method[] allMethods = clazz.getDeclaredMethods();
        for (Method m : allMethods) {
            // cycling through all methods, as we don't want users to specify
            // the list of argument types, we rely on the compiler
            if (m.getName().equals(methodName)) {
                m.setAccessible(true);
                try {
                    Object result = m.invoke(object, args);
                    return result;
                } catch (IllegalAccessException e) {
                } catch (IllegalArgumentException e) {
                }
            }
        }
        throw new IllegalArgumentException(
                String.format(
                        "No matching method has been found for method named %s and parameters %s",
                        methodName, Arrays.toString(args)));
    }

    /**
     * Invokes the method called <tt>methodName</tt> on <tt>object</tt> passing
     * the provided arguments to it, returning the result if the invocation was
     * successful, throwing an {@link IllegalArgumentException} otherwise.
     * 
     * @param object
     *            the object onto which the method is to be invoked
     * @param methodName
     *            the (case-sensitive) name of the method to be called
     * @param args
     *            the arguments to be passed to the method
     * @return the value returned by the method
     * @throws IllegalArgumentException
     *             in case a method called <tt>methodName</tt> accepting the
     *             provided list of arguments is not found, or <tt>object</tt>
     *             is <code>null</code>
     * @throws InvocationTargetException
     *             in case the method threw an exception (that can be retrieved
     *             calling {@link InvocationTargetException#getCause()} on the
     *             caught exception)
     */
    public static Object callMethod(Object object, String methodName,
            Object... args) throws IllegalArgumentException,
            InvocationTargetException {
        if (object == null || methodName == null)
            throw new IllegalArgumentException("null object or method name");
        return call(object.getClass(), object, methodName, args);
    }

    /**
     * Invokes the method called <tt>methodName</tt> of class <tt>clazz</tt>
     * passing the provided arguments to it, returning the result if the
     * invocation was successful, throwing an {@link IllegalArgumentException}
     * otherwise.
     * 
     * @param clazz
     *            the class whose method is to be invoked
     * @param methodName
     *            the (case-sensitive) name of the method to be called
     * @param args
     *            the arguments to be passed to the method
     * @return the value returned by the method
     * @throws IllegalArgumentException
     *             in case a method called <tt>methodName</tt> accepting the
     *             provided list of arguments is not found, or <tt>object</tt>
     *             is <code>null</code>
     * @throws InvocationTargetException
     *             in case the method threw an exception (that can be retrieved
     *             calling {@link InvocationTargetException#getCause()} on the
     *             caught exception)
     */
    public static Object callStaticMethod(Class<?> clazz, String methodName,
            Object... args) throws IllegalArgumentException,
            InvocationTargetException {
        if (clazz == null || methodName == null)
            throw new IllegalArgumentException("null class or method name");
        return call(clazz, null, methodName, args);
    }

    /**
     * Sets the value of the field named <tt>fieldName</tt> for object
     * <tt>object</tt> to <tt>newValue</tt> and returns <code>true</code> if the
     * operation was successful.
     * 
     * @param object
     *            the object whose field is to be changed
     * @param fieldName
     *            the (case-sensitive) name of the field whose content is to be
     *            changed
     * @param newValue
     *            the new value that the field should store
     * @return <code>true</code> if the new value has been set,
     *         <code>false</code> otherwise
     */
    public static boolean changeField(Object object, String fieldName,
            Object newValue) {
        if (object == null)
            return false;
        return change(object.getClass(), object, fieldName, newValue);
    }

    /**
     * Creates a new object of type <tt>T</tt> by calling a constructor of class
     * <tt>clazz</tt> accepting the provided list of <tt>args</tt>.
     * 
     * @param <T>
     *            the type of the object to be created
     * @param clazz
     *            the class of the object to be created (remember, it's
     *            <tt>Class&lt;T&gt;</tt>)
     * @param args
     *            the arguments to feed the constructor with
     * @return the created object or <code>null</code> if anything goes wrong
     * @throws InvocationTargetException
     *             in case the matching constructor throws an <tt>Exception</tt>
     *             (that can be retrieved calling
     *             {@link InvocationTargetException#getCause()} on the caught
     *             exception) when called
     */
    public static <T> T createNew(Class<T> clazz, Object... args)
            throws InvocationTargetException {
        if (clazz == null)
            return null;
        Constructor<?>[] allConstructors = clazz.getDeclaredConstructors();
        for (Constructor<?> c : allConstructors) {
            // cycling through all constructors, as we don't want users to
            // specify
            // the list of argument types, we rely on the compiler
            c.setAccessible(true);
            try {
                Object result = c.newInstance(args);
                return clazz.cast(result);
            } catch (IllegalAccessException e) {
            } catch (IllegalArgumentException e) {
            } catch (InstantiationException e) {
                e.printStackTrace();
            }
        }
        throw new IllegalArgumentException(
                String.format(
                        "No matching constructor has been found for class %s and parameters %s",
                        clazz, Arrays.toString(args)));
    }
}

This is the same class on pastebin, I think it’s more readable there.

The weird for loop in call() and createNew() is there because I don’t want to specify the type of every argument to be passed along to methods or constructors, so I can’t use getDeclaredMethod() or getDeclaredConstructor(). This is testing code, so usability should be favored over performance, I guess…

Testing a class with a non-zero argument constructor in Google Test

Of all the C++ testing libraries I googled for, the most powerful and easiest to use seems to be… er… Google’s 🙂

Me being mostly a Java/Python programmer (and proud to be!), I still can’t find it “easy” but… it’s mostly like using JUnit (well, mostly).

You get your SetUp() and TearDown() functions, your ::testing::Test class to extend and your TEST_F()s with all their EXPECT_whatever() to be run stuffing ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); into your main() function.

Ok, the first test I wrote actually shattered my “oh-my-it’s-just-like-JUnit!” dream.. I just wanted to write a test for a class that needs a std::string as argument. The easiest way I found to do that is to have your test class extend the class under test, and wrap the super class constructor with a no-args one.

Here’s what I did:

#include "gtest/gtest.h"
#include "my_class.h"

using namespace std;

class MyClassTest: public ::testing::Test, public MyClass {
  public:
    static string test_path_;
    string pathname() {
      return pathname_; // as it's a protected field in MyClass
    }
  protected:
    MyClassTest() :
      MyClass(test_path_) {
    };
    virtual void SetUp() {
      my_class_test_ = this;
    }
    MyClassTest* my_class_test_;
};

string MyClassTest::test_path_ = "test";

// Tests that MyClass constructor sets the correct path
TEST_F(MyClassTest, Constructor) {
  EXPECT_STREQ("test", my_class_test_->pathname().c_str());
}

Of course this is just a toy to understand the framework, but it’s a little example I couldn’t easily find on the web right now..
I’m no C++ expert at all, so if anyone reading this feels insulted by something I wrote, please, please post a comment and teach me how this should be done, I’m more than willing to learn! 🙂

[Update – July 2013]: Mark Abraham posted how it’s done, just scroll down to his comment!