Running Unit Tests for RCP and OSGi Applications

Eclipse provides great tools for testing RCP and OSGi applications using JUnit, but there a few areas that are problematic.

  • It’s not easy to run all the tests in a set of plug-ins. The test launcher allows you to run all the tests in a single project, but RCP and OSGi developers are usually working with a large set of test plug-ins. Sure it’s possible to create test suites, but keeping suites up-to-date is a real pain.
  • It’s not easy to use test fragments. To find out why you’d want to use test fragments instead of test plug-ins, check out my previous post on Testing Plug-ins with Fragments. The problem is that even the standard suite-based solution does not work with fragments. There are workarounds, but they’re not very pretty.
  • It’s not easy to run all of your tests during an automated build using the Eclipse Testing Framework. This is related to the first point above, and again you can use suites to handle this. 
So to sum things up, I want to run sets of tests across multiple plug-ins or fragments and I don’t want to use suites. My solutions has been to create a simple bundle test collection plug-in that harvests unit tests based on a set of filters. To use the plug-in, you need to do the following:
  1. Download the bundle test collector which is licensed under the standard EPL. The archive contains the test collector plug-in and also an example plug-in showing proper usage.
  2. Add the com.rcpquickstart.bundletestcollector plug-in to your workspace.
  3. Create a plug-in that will contain a suite or set of suites that will load tests based on filters. The tests making up the suites will be generated dynamically, so you won’t need to maintain them. This plug-in will need to depend on the com.rcpquickstart.bundletestcollector and junit plug-ins, but that’s it.
  4. In your suite, add the following method:
    	public static Test suite() {
    		BundleTestCollector testCollector = new BundleTestCollector();
    
    		TestSuite suite = new TestSuite("All Tests");
    
    		/*
    		 * assemble as many collections as you like based on bundle, package and
    		 * classname filters
    		 */
    		testCollector.collectTests(suite, "com.mycompany.", "com.mycompany.mypackage.",
    				"*Test");
    
    		return suite;
    
    	}

 

You can then run the test suite both inside of the Eclipse IDE and using the Eclipse Testing Framework. I should note that this works only for JUnit 3.x tests. JUnit 4 describes suites using annotations which makes it (as far as I can tell) impossible to dynamically generate a suite at runtime. If anyone has a solution to this, I’d love to hear it. 

As always, comments and fixes are much appreciated.

Advertisements

36 Responses to Running Unit Tests for RCP and OSGi Applications

  1. […] UPDATE: For an alternative approach to running tests in fragments, see my post on Running Unit Tests for RCP and OSGi Applications. […]

  2. Jan says:

    Wow, that is exactly what I was looking for for the test server of Xtext. Do you have it on a publicly available CVS, such that I could reference it from a map file?

  3. Patrick says:

    Hi Jan,

    Currently, the code is just available on this blog. Sorry about that.

    After writing this utility, I did find that the Pluginbuilder project has something called an autotestsuite plug-in that might do something similar. It looks like it might only work with Eclipse 3.3 though. You can find out more about it at http://www.pluginbuilder.org.

    — Patrick

  4. Jan says:

    Unfortunately, I cannot get your solution working in an eclipse build (Xtext is built on server hosted by the eclipse foundation). It keeps throwing NoClassDefFound errors, which is very nasty to debug. How did you manage to run it in the Eclipse test framework?

  5. Patrick says:

    Hi Jan,

    I’ve managed to duplicate the problem but haven’t figured out what’s going on yet. Hopefully, I’ll be able to track this down soon, and I’ll post a comment on this post when I do.

    — Patrick

  6. Eric Geordi says:

    Thank you for writing this! Eclipse Test Framework has been somewhat of a pain to use lately. Unfortunately, I am seeing the same problem that Jan was when running the bundle collector with the Eclipse Test Framework. I tried the autotestsuite plugin but wasn’t having much luck with that in 3.4.

  7. Patrick says:

    Hi Eric,

    It appears that the problem may be related to previously existing config info in the Eclipse SDK that is running the test framework. What works for me is to have a clean Eclipse SDK archive and unzip this during the build process.

    Can you tell me how you’re assembling your test environment? Are you copying the “eclipse” directory for an instance of Eclipse that has been run in the past?

    — Patrick

  8. Eric Geordi says:

    Yeah, I am currently just doing a copy from a previously run eclipse instance. I’ll try unzipping from a clean Eclipse SDK and see if that works better.

  9. Eric Geordi says:

    I am still getting stuck here… Here’s the process that I am following in our build. We are building three features, one of which is the unit test feature. All the unit tests are fragments for the corresponding plugins being tested. At the end of the build, three archives are created. I extract those archives into a test directory location. Then, I extract a fresh eclipse 3.4 SDK into that directory, as well as the eclipse test framework for 3.4.

    Is there a different route I should take for this ? It appears that as soon as I take a working unit test fragment and add the test bundle collector as a dependency, I get a “Class not found” error. If I take it back out, then the unit test works again. Any ideas on where I’m going wrong here ?

    Thanks,
    Eric

  10. Patrick says:

    Hi Eric,

    Your process sounds fine. I’m guessing that you include the bundlecollector plug-in in your test feature along with the plug-in containing your test suite. Is that right?

    I’m curious, are you getting a ClassNotFoundException or a NoClassDefFoundError? And in either case, which class is it looking for?

    The problems I had with the config area were causing NoClassDefFoundErrors to be thrown, and the offending class was usually something deep inside the Eclipse API.

    — Patrick

  11. Eric Geordi says:

    Ah! I just found out that I wasn’t including the bundlecollector as part of the test feature. I knew it had to be something simple! Anyways, thank you so much for your help.

  12. Martin says:

    I also tried this, and I could not get it running. I also got plenty of Class not found – Exceptions.
    What did work is to declare a “Buddy-Policy: dependent” in the Bundle-Test-Collector Plugin, then suddenly no problems.

  13. Patrick says:

    Hi Martin,

    Thanks! That helps a lot. I’ll update the manifest in the download to help others avoid this issue.

    — Patrick

  14. Martin says:

    Hi again,

    I´m not sure, and did not yet have the time to test it, but I found the following solution, which sounds, as if this could solve the issue with junit 4:
    You make your JUnit 4 test classes accessible to a TestRunner designed to work with earlier versions of JUnit, declare a static method suite that returns a test.

    public static junit.framework.Test suite() {
    return new JUnit4TestAdapter(Example.class);
    }

    So you could run all your junit4-tests with the standard junit3-runner?
    What do you mean about this? If I somehow get the time to test this, I´ll report again

  15. Patrick says:

    Hi Martin,

    I’d be interested to hear of any success you have with this. My understanding is that the ETF itself does not yet support JUnit 4. This issue seems to be that having separate plug-ins for JUnit 3 and JUnit 4 is causing classloading issues with the ETF. You can find out the details here:

    https://bugs.eclipse.org/bugs/show_bug.cgi?id=153429

    Again, if you find a solution to this I would be very interested. I have a few clients that are looking for an answer.

    — Patrick

  16. Martin says:

    So, once more;-),
    I now found the time to play around with this, and it seems to work, at least for me, if you make some minor changes.
    I did not try it together with the ETF, but was just interested in a possibility, to run all my junit4 fragment unit-tests inside the ide.

    Here are 2 possibilites how this could be done:

    my first approach was just to add all tests to a single suite, this alone would have been enough for me, because so all tests run.
    If you want to do this, you just have to change the following lines:

    for (Class clazz : testClasses) {
    suite.addTestSuite(clazz);
    }

    to

    for (Class clazz : testClasses) {
    suite.addTest(new JUnit4TestAdapter(clazz));
    }

    Well, I´m not so experienced in In-Depth-Junit-Facts, are there possible problems running all tests in just one suite? Thanks in advance

  17. Patrick says:

    Hi Martin,

    Yes, the JUnit 4 tests work fine inside the IDE. The only problem is with the ETF.

    — Patrick

  18. Martin says:

    Hi Patrick,

    I just found again some minutes to play around my special friend etf, and I think, it works with JUnit4 without any problems with my approach I submitted earlier.
    Did you also have success with it?

    Btw, you should give more Blog-Entries;)

  19. Patrick says:

    Hi Martin,

    I’m glad to hear that you’ve got this working. Did you apply the patch mentioned in the Bugzilla entry I mentioned in an earlier comment? Let me know exactly what you did and I’ll try to see if I can duplicate your success.

    And yes, I should have more blog entries… I’ve been traveling non-stop the last month or two, but I’m hoping to have time for posting in the near future.

    — Patrick

  20. […] den BundleTestCollector auszuführen müssen nur die in diesem Eintrag von Patrick beschriebenen Schritte beachtet […]

  21. Howard says:

    Hi Patrick,

    Did you get your JUnit4 tests working with ETF? I added “JUnit4TestAdapter” as Martin mentioned. However I got the following errors:

    “=”junit.framework.JUnit4TestAdapter” type=”java.lang.ClassCastException”>at org.eclipse.test.EclipseTestRunner.getTest(EclipseTestRunner.java:265) at org.eclipse.test.EclipseTestRunner.(EclipseTestRunner.java:220) at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:204) at org.eclipse.test.UITestApplication$3.run(UITestApplication.java:195) …”

    I use Eclipse 3.4 and same load of ETF. Now I wounder if ETF supports JUnit4 tests.

    Thansk and Regards,

    Howard

  22. Howard says:

    In addition to my above post, I tried to applied the patch from Bug153429 into my running eclipse\plugins ditectory, and could not make JUnit4 tests working. The worse thing was that JUnit3 tests failed if applying the patch. Does anyone have some ideas? Thanks.

    – Howard

  23. Patrick says:

    Hi Howard,

    In my experience, JUnit 4 tests will not run with the ETF. I have not applied the patch, but others I’ve worked with have and they have not yet been successful. They have asked for help on the Bugzilla entry, but no luck yet.

    I wish I could be more help, and I really hope they get this fixed for Eclipse 3.5. If it’s important to you, make your voice heard on the Bugzilla entry.

    — Patrick

  24. Howard says:

    Hi Patrick,

    Thank you very much for your quick response. I believe that you are correct. I did try the same test in 3.5M3, and got the same errors as 3.4. Will use Bugzilla to ask the fix.

    – Howard

  25. Martin Dilger says:

    Hi Patrick,

    one question, have you ever considered using the “Eclipse-ExtensibleAPI”-Header in your Manifests? so you don´t need your Bundletestcollector?

  26. Patrick says:

    Hi Martin,

    That’s a great suggestion, and it makes it much easier to assemble unit tests based on fragments. It certainly eliminates the need to use the reflection workaround that I’ve been using in the past. Thanks!

    I think a bundle test collector still makes sense in situations where you don’t want to maintain hard-wired test suites, though.

    — Patrick

  27. Will Horn says:

    You can dynamically generate a suite at runtime in JUnit4 with a custom runner:

    @RunWith(DynamicSuite.class)
    public class AllTestsSuite {
    }

    public class DynamicSuite extends Suite {
    public DynamicSuite(Class klass) throws InitializationError {
    super(klass, getTests());
    }
    static Class[] getTests() {
    //…collect tests
    }
    }

    Of course, this does not help with the ETF bug.

  28. Johannes says:

    Hi,

    just a Comment on this, the ETF works perfectly well, if you do it the hard way. Just remove all references to JUnit3. We did this, and this is the best approach until this whole thing is in place, the patch does not work really well.

  29. Patrick says:

    Hi Johannes,

    That’s great, but can you be more specific? Do you mean you removed the org.junit plug-in from the test environment and removed all references to it in other Eclipse SDK plug-ins? I’d love to have a detailed solution to this problem, and I’d be interested to hear what you did.

    — Patrick

  30. Markus says:

    Hi,

    I just would like to give a pointer to an alternative test framework to ETF which is called Autotestsuite and is part of Pluginbuilder.

    The Autotestsuite collects both JUnit3 and
    JUnit4 Test Cases and creates a test suite automatically. JUnit 3
    tests are determined by hierarchy (subclasses of TestCase) whereas
    JUnit 4 tests are determined by annotations.

    The advantage of the autotestsuite is that you neither need to create
    All* suites nor add any XML configuration. Every (except ignored JUnit
    4) test will be executed without additional tedious configuration. If
    you really want to exclude tests, you can do so with regular
    expressions. The regular expressions also provide an easy way to
    create conventions for your projects, e.g. *UITest* to execute all UI
    specific tests.

  31. Johannes says:

    Hi Patrick,

    no, its surprisingly easy to get this etf running on junit4, just replace all the references in the etf to org.junit with org.junit4. You don´t have to change anything within the SDK (why would you anyway?)

    Johannes

  32. […] his article Running Unit Tests for RCP and OSGi Applications, Patrick describes how sets of tests across multiple plug-ins or fragments can be run without using […]

  33. Martin says:

    Hi Johannes,

    I still run on the junit4 problem with etf. I checked out etf from cvs. Then I exported the etf feature and it seemed to be valid. Of course before I removed every dependency to org.junit and added org.junit4.
    But if i run the test target in build.xml I get the message:

    core-test:
    java-test:
    [echo] Running com.rcpquickstart.helloworld.test.collector.AllTests. Result file: E:\WS_AutomatedBuildExample\com.rcpquickstart.helloworld.build-and-test/buildDirectory/test/eclipse/results/com.rcpquickstart.helloworld.test.collector.AllTests.xml.
    [java] Java Result: 13
    collect-results:
    [junitreport] the file E:\WS_AutomatedBuildExample\com.rcpquickstart.helloworld.build-and-test\buildDirectory\test\eclipse\com.rcpquickstart.helloworld.test.collector.AllTests.xml is empty.
    [junitreport] This can be caused by the test JVM exiting unexpectedly
    [style] Warning: the task name is deprecated. Use instead.
    [style] Transforming into E:\WS_AutomatedBuildExample\com.rcpquickstart.helloworld.build-and-test\buildDirectory\test\eclipse\results
    collect:
    [junitreport] the file E:\WS_AutomatedBuildExample\com.rcpquickstart.helloworld.build-and-test\buildDirectory\test\eclipse\com.rcpquickstart.helloworld.test.collector.AllTests.xml is empty.
    [junitreport] This can be caused by the test JVM exiting unexpectedly

    Even with debugging I couldn’t find out, what exactly the problem is. Maybe since there is no actual build instruction for etf, can you explain more precisely how you made it.

    I would appreciate the help.

    Martin

  34. Gabe says:

    This is great. Thanks!

    2 questions:

    After creating an AllTests class that extends BundleTestCollector (like shown in the example provided in the zip archive) I cant get it to run as part of my build through ETF but all the tests are shown in a flat list. Can I provide a heirarchy so it is easier to see which tests belong to which package/class?

    How can I run directly from the Eclipse IDE?

  35. Gabe says:

    I am also having a problem where test classes in a fragment are showing a “No tests found” error. I can run the class as an ordinary Junit test just fine. Test classes in a plugin are run fine by BundleTestCollector.

  36. Patrick says:

    Hi Gabe,

    The collector is pretty simple and could easily be extended to organize tests in different ways. There are no output options right now, though.

    To run tests in the IDE, you can just run the test suite that extends the bundle test collector. The tests should run normally in the IDE, but make sure you choose Run As -> JUnit Plug-in Test.

    I’ve haven’t been able to run any fragment based tests under the ETF either, at least under 3.4. The fragment doesn’t get activated along with the bundle. Haven’t tried it on 3.5 yet. What version are you using?

    — Patrick

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: