JUnit Testing Notes

Russell Bateman
2008
last update:



JUnit testing

JUnit testing is pretty easy. Here's a really good tutorial: http://clarkware.com/articles/JUnitPrimer.html. I think there are some mistakes in the accompanying source code to this tutorial which I'll attempt to correct in these notes (by not making the same mistakes). Another good tutorial is http://www.onjava.com/pub/a/onjava/2004/02/04/juie.html. This latter tutorial doesn't really cover the TestSuite concept.

Some useful notions are:


Approaches...

Black box testing is testing without having access to the source code (or by not looking at it). This fits well in with test-driven development.

White box testing is examining the source code (of the subject under test) as tests are written. This ensures that tests collectively cover all of the paths any given action might take with an eye to watching boundaries of loops, respecting all logical conditions, etc.

Testing state refers to performing an action in a test case. then verifying, probably by assertion, that an expected result was achieved (or not achieved. Doing this sort of thing can certainly be part (an objective of) black- or white-box testing.

Writing JUnit tests amounts to getting everything set up for a test case (arranging), executing the action under test) (acting) and verifying that the action completed correctly (asserting).

Advice

  • Keep assertions to a minimum; create more test cases rather than "testing the fool" out of something. This makes the test cases more useful, Lego&tm;-like and less brittle (easier to maintain).
  • Assertionless test cases should probably be avoided or, at least, commented upon such that it's clear they only exist to demonstrate code functionality, approach taken, etc. as if documentary in nature.
  • Don't repeat like-assertions over and over again.
  • Don't collect assertion statements together in utility methods to be called from test cases themselves. This obfuscate the meaning of the test case.

Setting up JUnit tests in Eclipse...

Here's how to wire up a JUnit library to your Eclipse project, it's pretty simple to do:

  1. Right-click your project.
  2. Choose Build Path -> Configure Build Path...
  3. Click the Libraries tab.
  4. Click Add Library...
  5. Select JUnit.
  6. Click Next.
  7. Select version and click Finish.

At your project root, create a new folder named test.

  1. Right-click it, choose Build Path, then Use as Source Folder.
  2. Create a new package matching the package name in your project/src folder whose class(es) you want to test.

To Create a new JUnit test class:

  1. Right-click the new test/package and choose JUnit Test Case (probably only available if you added the JUnit library in the steps above).
  2. Follow the wizard as should be obvious.

Writing JUnit test code...

Is twofold: a) writing test cases and b) writing a test suite.

Automatic test generation

If using Eclipse, a test case for any class may be generated automatically with only the need to fill out stubs. The stubs are initially set up to generate failures, which concords with test-driven development (TDD) theory (everything should fail initially and implementation consists of eliminating the failures).

To set up a test case, locate the folder/package dominating the location the test source code will be place and right-click. Choose New->JUnit Test Case. Enter the name of the package (if it's wrong), the name of the test case (as the name of the class to test followed by “Test”), then click setUp() and tearDown() unless you don't want these for some reason, and last, the name of the class you're testing. Then, click next.

At this point, you're offered automatic code generation of those methods you select. Click Finish. You also have the option to create final method stubs and to create Eclipse tasks to remind you to finish implementing the test methods that are about to be generated.

Writing the test methods...

The test cases are written thus:

  package test-location;

  import junit.framework.TestCase;

  public class class-nameTest extends TestCase
  {
    protected void setUp()
    {
      // Set up data and objects required for use in any test case. Usually,
      // these would be instance variables.
    }

    protected void tearDown()
    {
      // Release objects set up above.
    }

    public void testmethod-name()        <------ this is a test case
    {
      assertEquals( expected, actual );
    }
  }

Let's make a useful if bogus instantiation of the naming concepts above in order for later, TestSuite sample code to be of any use.

  package com.myapp.utils;

  import junit.framework.TestCase;

  public class FunAndGamesTest extends TestCase
  {
    protected void setUp()
    {
      // Set up data and objects required for use in any test case. Usually,
      // these would be instance variables.
    }

    protected void tearDown()
    {
      // Release objects set up above.
    }

    public void testRobBank()
    {
      assertEquals( expected, actual );
    }

    public void testLivingTheVidaLoca()
    {
      assertEquals( expected, actual );
    }
  }

Let's make a second sample test class for the purposes of later demonstration. We'll even add just enough real code to demonstrate this undertaking further than FunAndGamesTest did.

  package com.myapp.utils;

  import junit.framework.TestCase;

  public class AllWorkAndNoPlayTest extends TestCase
  {
    protected void setUp()
    {
      AllWorkAndNoPlay  today = new AllWorkAndNoPlay();

      System.out.println( "AllWorkAndNoPlay set-up" );
    }

    protected void tearDown()
    {
      System.out.println( "AllWorkAndNoPlay tear-down" );
    }

    public void testMakeADullBoy()
    {
      today.MakeADullBoy( true );
      assertEquals( true, today.boyIsDull );
    }
  }

Invoke the tests for all classes in a package, or for all classes contributing to an important, identifiable functionality, using a TestSuite. In our sample template here, we're assuming that the package consists of classes of a common functionality all of which we'll test in this suite. Note that, unlike the previous code implementing tests of methods in a single class, the variable part of the TestSuite is ONLY the tests we add.

Assume that in com.myapp.utils, we create UtilsTestSuite to run all the individual class tests. We will use addTestSuite to do this.

  package com.myapp.utils;

  import junit.framework.Test;
  import junit.framework.TestSuite;


  public class UtilsTestSuite
  {
    public static Test suite()
    {
      TestSuite suite = new TestSuite( "Utilities tests" );

      suite.addTestSuite( FunAndGamesTest.class );
      suite.addTestSuite( AllWorkAndNoPlayTest.class );

      return suite;
    }

    public static void main( String[] args )
    {
      junit.textui.TestRunner.run( suite() );
    }
  }

Note the distinction between addTestSuite, which adds a test to the suite being defined in this class, and later, addTest, which adds test suites. This is a distinction that the first tutorial linked above did poorly in its test code. Indeed, none of the tutorials accurately addressed this distinction if they mentioned addTestSuite at all.

addTestSuite is a misnomer. You don't use it to add a TestSuite, but to add a single class as a TestSuite (see class UtilsTestSuite above). It's a shortcut for: addTest( new TestSuite( class-nameTest.class ) ). In our case here, the code is the following and we won't even use addTestSuite. I think it's less confusing to do it this way (and never use addTestSuite).

So, above the package leaf-node level collective test, UtilsTestSuite, we'll constitute successively higher collective tests. This one might, if the MyApp is simple enough, have been named AllTests.java. (In the discussion of hierarchy below, we show a more complex application and suggest the names of successively higher levels.)

  import junit.framework.Test;
  import junit.framework.TestSuite;

  public class MyAppTestSuite
  {
    public static Test suite()
    {
      TestSuite suite = new TestSuite( "Tests for all of MyApp" );

      suite.addTest( UtilsTestSuite.suite() );
      //suite.addTest( AnotherTestSuite.suite() ); *

      return suite;
    }

    public static void main(String[] args)
    {
      junit.textui.TestRunner.run( suite() );
    }
  }

* Now, I know this is the same cop-out in the existing tutorials, but here I have add the paragraph clarifying addTest and addTestSuite which they did not do. This oversight led to initial confusion about how best to add test suites to successively higher test suites (which is not done, at least by us, using addTestSuite since its name only misleads you to thinking this is what to use.


JUnit test code order of execution

It appears that under the Eclipse debugger, at least, you can add a new test method to the whole, but it will execute after the previous ones. To add one in front and get it to execute first, just suck out all the others, run the new method, then paste back in the old stuff you copied. The new one will then execute first.

(I haven't experimented overly with this.)


JUnit test code order of execution—SOLVED

Beginning in JUnit v4.11, the following annotations influence the order. Ordinarily, however, it's somewhat perverse to insist upon the order and, should test cases develop a behavioral dependency on run order, this would absolutely indict the wisdom of the developer coding the tests. Order should make no difference other, perhaps, than for aesthetics, for example, if you wanted to see an ordered list appear at the console for a set of test cases because it imparts a certain documentation order on a reduced number of test cases or is pleasing to the eye.

import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.FixMethodOrder;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestName;
import org.junit.runners.MethodSorters;

@FixMethodOrder( MethodSorters.JVM )
//@FixMethodOrder( MethodSorters.NAME_ASCENDING )
//@FixMethodOrder( MethodSorters.DEFAULT )
public class TestFixMethodOrder
{
  // @formatter:off
  @Rule        public TestName name = new TestName();
  @Before      public void setUp() { System.out.print( "  " + name.getMethodName() + "..." ); }
  @BeforeClass public static void beforeClass() { System.out.print( "MethodSorters.JVM--coded order:" ); }
//@BeforeClass public static void beforeClass() { System.out.print( "MethodSorters.NAME_ASCENDING--alphabetic order:" ); }
//@BeforeClass public static void beforeClass() { System.out.print( "MethodSorters.DEFAULT--random order:" );
}

  @Test public void testZoolander()   { }
  @Test public void testMightyMouse() { }
  @Test public void testSnagglepuss() { }
  @Test public void testAntMan()      { }
}

Output (consider that you run this test suite three times exchanging commented out lines above as you may imagine):

MethodSorters.JVM--coded order:
  testZoolander...
  testMightyMouse...
  testSnagglepuss...
  testAntMan...

MethodSorters.NAME_ASCENDING--alphabetic order:
  testAntMan...
  testMightyMouse...
  testSnagglepuss...
  testZoolander...

MethodSorters.DEFAULT--random order:
  testZoolander...
  testMightyMouse...
  testSnagglepuss...
  testAntMan...

Of course, this doesn't really work. It only appears some times to work.


JUnit test code tricks

If you don't want to set up and tear down between each test method, you can create methods to perform the set-up and tear-down, call the set-up from at the beginning of the first test and the tear-down at the end of the last test methods. In this case, make setUp and tearDown more or less stubs.


Testing hierarchy

For example, here is the proposed JUnit test hierarchy for a product called MyApp that consists of a server application, worker application and common library projects. Assume the developing company is named myapp.com. using packages such as com.myapp.archive, etc.

There are additional notions illustrated here, like the smoke test, that will be defined afterward. As there are identical package paths in common between MyAppServer, MyAppWorker and MyAppLib (at least), conventions such as ServerClassesTestSuite become necessary.

  (Application projects...)
    MyAppServer                      MyAppWorker
      AllTests (top-level for MyApp Server)          AllTests (top-level for MyApp Worker)
        SmokeTestSuite (structural integrity)          SmokeTestSuite (structural integrity)
          ServerClassesTestSuite                         WorkerClassesTestSuite
            AddressTestSuite                               FiltersTestSuite
              ABSearchTest                                   MainFilterTest
              ABTreeTest                                     QrtzInitializeServletTest
            ArchiveTestSuite                                 QrtzInitializeServletATest
              HcapArchiveTest                                .
            AuthenticateTestSuite                            .
              LDAPTestSuite                                  .
                BlindSSLSocketFactoryTest
                LDAPParamsTest
                LDAPUtilTest
              AbstractGroupWiseAuthenticatorTest
              .
              .
              .
              WorkerAuthenticationMethodTest
            CachingTestSuite
              ABCacheTest
              IndexHandlerCacheTest
              .
              .
              .
              WorkerPasswordCacheTest
            CfgTestSuite
            .
            .
            .
          ContractorClassesTestSuite
            ArchiveTestSuite
              ArchiveFileTest
              ArchiveFileBaseTest
              .
              .
              .
              XMLFileTest
            CfgTestSuite
            .
            .
            .

  (Oblique projects...)
    MyAppLib        EasySoap        SharedJava
      AllTests               AllTests                AllTests
        .                      .                       .
        .                      .                       .
        .                      .                       .

This stratagem adds 1) one test class for every implementation class written, 2) one test suite class for every package's leaf and intermediate nodes, and 3) one AllTests.java and one SmokeTestSuite.java for every project. However, there is nearly no work at all to any of these classes except for the individual class test.

Eclipse Maven support is reputedly less than full or ideal. Maven could be used to generate all these classes except for the individual class tests.


Smoke tests

Smoke is testing that ensures the basic, continued stability of the application code by revealing serious flaws in very short order. In theory, failing smoke testing is tantamount to rejecting a release candidate with no need to spend more time or other resources on validating it. “Smoke and build” refers to applying and passing smoke tests, then performing the full build of a product's ISO or other distributive means for the real validation testing performed by a Quality Assurance team.

I remain uncertain as to what extent AllTests.java and SmokeTestSuite.java differ.


JUnit: “The input type of the launch configuration...

...does not exist!” (an alert you get when you attempt to Run/Debug a JUnit test in a project whose test subdirectory has not been qualified as part of the source code. The easiest way to solve this is to right-click on the folder, then choose Build Path → Use as Source Folder.


How to set up a project for JUnit testing in Eclipse...
  1. Right-click on the project name. Choose New->Source Folder.
  2. Name the new folder test and confirm its creation.
  3. Click that folder and create a new package path choosing something that aligns with the packages in the src folder you are testing.
  4. Click on the new package just created and choose New->JUnit Test Case. If this is the first one you've created, there will be an opportunity to add the JUnit library JAR to the project. You must do this.

JUnit 4

Starting with Java 5, JUnit 4 is possible on account of the use of annotations. JUnit 4 test-writing is, I think, superior in flexibility and easier than JUnit 3. Without deep-ending on the differences of JUnit 4, here's a succinct demonstration:

package com.etretatlogiciels.junit;

import static org.junit.Assert.assertEquals;

import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

public class MyClassTest
{
  @Test
  public void testMultiply()
  {
    MyClass  tester = new MyClass();
    assertEquals( "Result", 50, tester.multiply( 10, 5 ) );
  }

  @Before public void beforeEach()
  {
    System.out.println( "beforeEach() test..." );
  }

  @After public void afterEach()
  {
    System.out.println( "afterEach() test..." );
  }

  @BeforeClass public static void beforeAll()
  {
    System.out.println( "beforeAll() tests..." );
  }

  @AfterClass public static void afterAll()
  {
    System.out.println( "afterAll() tests..." );
  }
}

These tests run yielding the following output at the console, adequately demonstrating the function of the annotations:

	beforeAll() tests...
	beforeEach() test...
	afterEach() test...
	afterAll() tests...

Notes from JUnit Recipes

(From Manning by J. B. Rainsberger: this is a very old book; some of the recipes are now out of favor, but on the whole, there's a great deal of useful material.)

"Stop debugging: write a test instead!"

JUnit test results yield:

	78 run, 2 failures, 1 error

Fix the error first and maybe the failures will disappear.

Why?

  • Throws error which is unchecked so not every bit of code needs to declare that it throws an error.
  • An error means that there's something wrong with the test code (not necessarily with the production code).
  • Failed assertion is a test failure and means that there is a logic error to fix in the production implementation.

Best practices


Enabling Java assertions...

See Enabling Java assertions.


JUnit print name of each test case as a ruler from setUp()
import org.junit.Test;
import org.junit.Before;
import org.junit.Rule;
import org.junit.rules.TestName;

public class SomeTest
{
  @Rule
  public TestName name = new TestName();

  @Before
  public void setUp() throws Exception
  {
    String testName = name.getMethodName();
    int PAD = 100;
    int nameWidth = testName.length();
    System.out.print( "Test: " + testName + " " );
    PAD -= nameWidth;
    while( PAD-- > 0 )
      System.out.print( "-" );
    System.out.println();
  }

  @After
  public void tearDown() throws Exception
  {
  }

  @Test
  public void test_sample()
  {
  }
}

JUnit 5

JUnit 5 was released at the end of 2017 after Junit 4's reign since 2006.

Annotations

Expect test to throw exception

@Test
void testThrowsException() throws Exception
{
  Assertions.assertThrows( Exception.class, () -> { ... } );
}

Assertions

import org.Junit.jupiter.api.Assertions;

{
  assertEquals( 1, 2, "Failed assertion message" );
}