Testing Plug-ins with Fragments

As Eclipse plug-in and Rich Client Platform developers, we face unique challenges in how we structure and execute our unit tests. In this article, I suggest an approach to unit testing based on Eclipse fragments that can help us overcome these challenges. If you find yourself frustrated with your current plug-in testing options, read on!

But before going into detail on a fragment based solution, let’s examine the current approaches and the pros and cons associated with each. The first approach is the one most of us start with as we learn the ropes of plug-in development: placing all code in a single plug-in.

Option 1: Place all code in a single plug-in

Placing all your code in a single plug-in is obviously the simplest approach. Usually, developers who go down this path choose to place their code in two different source directories: src and test.

Placing all code in a single plug-in

In addition to being simple, this approach has the benefit of allowing all of your code to be loaded by the same classloader (each plug-in has its own classloader). This means that if you maintain the same package structure in your source and test folders, your test classes will have access to the non-public methods of the classes under test.

The downside to this approach is that it seriously complicates your build process. If you wish to run your tests as part of your build process (and who doesn’t), then you’ll need to include your test code in your deployable plug-ins. Your plug-ins will also need to declare dependencies on JUnit and any other test libraries you use (EasyMock, etc.). It’s possible to manually remove the classes and manifest entries after the build, but this is far from ideal.

Option 2: Place test code in a separate plug-in

To get around these build-related issues, many of us choose to place unit test code in a separate plug-in. Usually, each testable plug-in will have a corresponding test plug-in. This is the approach taken by most Eclipse projects, and it has the advantage of cleanly separating out test code and dependencies during the build process.

Placing test code in a separate plug-in

The disadvantage of this approach, however, is that your test classes will be loaded by a separate classloader and will therefore not have access to the non-public methods of the classes under test. In addition, your tests will not have access to classes in packages not exported by the plug-in under test.

Plug-ins that are properly designed will typically export only a small set of API packages, keeping most of the implementation packages hidden from the outside world. But it’s often useful to have access to these implementation packages during testing. A common workaround is to expose these packages as friends of the test plug-in. While this approach works, it pollutes your plug-in manifest with reference to test plug-ins. It also does nothing to solve the other issue of accessing non-public methods.

Option 3: Place test code in a fragment

Wouldn’t it be great if we could combine the classloading benefits of having a single plug-in with the clean separation of test code achieved with a separate plug-in? Well luckily for us, this is possible using Eclipse fragments.

A fragments looks much like a plug-in from the outside. It is represented as a separate project in your workspace, it contains a manifest that describes its contents, it is built and deployed as a jar. What makes a fragment different is that it contributes it’s resources at runtime to a single host plug-in. The classes in the fragment are therefore loaded by the host plug-in’s classloader.

Placing test code in a fragment

By placing our unit tests in fragments, we can provide our tests access to the non-public methods of the classes under test. Also, because the source code and test code are actually part of the same plug-in, there are no issues related to non-exported packages. Test classes will have access to all packages in the plug-in, whether they are exported or not.

Workaround: Creating a master test suite

The main disadvantage to this fragment based approach is that it is difficult to aggregate unit tests into a master test suite. While it’s easy to create a test suite that includes the tests within a fragment, it’s not so easy to create a suite that includes the tests in multiple fragments.

The basic problem is that classes in fragments are not visible outside of the fragment itself. Any attempt to reference fragment classes in another plug-in will result in a compile error, so it’s not possible to assemble a master test suite in the standard way. But because the fragment classes are accessible via the host plug-in’s classloader at runtime, it is possible to create a master test suite using reflection. The code below shows how this can be done.

public class AllTests {

	public static Test suite() throws ClassNotFoundException {
		TestSuite suite = new TestSuite(
				"Master test suite.");

		suite.addTest(getTest("com.mycompany.myplugin1.AllTests"));
		suite.addTest(getTest("com.mycompany.myplugin2.AllTests"));
		return suite;
	}

	private static Test getTest(String suiteClassName) {
		try {
			Class clazz = Class.forName(suiteClassName);
			Method suiteMethod = clazz.getMethod("suite", new Class[0]);
			return (Test) suiteMethod.invoke(null, new Object[0]);
		} catch (Exception e) {
			throw new RuntimeException("Error", e);
		}
	}
}

Simply place this suite in a plug-in that declares dependencies on all testable plug-ins. You should then be able to run all of your tests with one click. While this code does add some complexity to the test suite, it is only necessary for the single master test suite class. The individual test suites in the fragments can be written normally.

UPDATE: For an alternative approach to running tests in fragments, see my post on Running Unit Tests for RCP and OSGi Applications.

UPDATE 01/12/2009: Another alternative is to add a Eclipse-ExtensibleAPI: true entry to the manifest of the plug-in being tested. This header makes your fragment classes available to other plug-ins during development, and eliminates the need for the reflection workaround above. Thanks to Martin Dilger for the suggestion.

Final thoughts

There are some other minor issues with a fragment based approach. One of the most annoying is that the JUnit view in Eclipse does not allow you to double-click on a test case to open the class. For some reason, though, clicking on a line in the Failure Trace section does work. Go figure.

Overall, however, I’ve found that the benefits of fragment based unit testing far outweigh the disadvantages. If you are satisfied with the more traditional approaches to unit testing plug-ins, then by all means keep doing what you’re doing. But if you find yourself frustrated with the limitations of these approaches, give fragments a try. You’ll be glad you did!

About these ads

32 Responses to Testing Plug-ins with Fragments

  1. Peter Nehrer says:

    Patrick, thanks for the great post, this is a really useful trick indeed. Actually, you can even get rid of the annoyance of having to use reflection to aggregate your unit tests from different fragments. You’re correct in that packages exported from a fragment are not visible at build time; however, you can work around this:

    1. export the fragment’s package (in its manifest)
    2. add the fragment project as a reference to your master suite’s Java build path (can’t do this in the manifest)
    3. add the exported fragment package into the Import-Package header of your master suite’s manifest (for runtime visibility)

    This way, the compiler is happy because it has the fragment project’s exports in the master suite’s classpath. The runtime is happy too because it can wire the fragment-exported package into your master suite’s classpath.

    I think this is simply a shortcoming of PDE — if you were to define a target platform that includes your test fragments (with exported packages) and use it to compile your master suite, then you wouldn’t even have to jump through these hoops, because those packages would be visible at build time, too (just like all SWT’s win32 packages are visible when win32 is the target platform).

  2. Thomas says:

    Thanks for sharing this useful idea.

    What I fail to see is why one would need such a master test suite in the first place? Wouldn’t you pack all test code into one fragment? Perhaps my (first) plugin I’m currently writing is so tiny that I cannot imagine splitting its test code… ;-)

  3. I also prefer the fragment approach.

    Thomas, I think Patrick meant a master suite that includes tests for many plug-ins.

  4. pjpaulin says:

    Hi Thomas,

    Yes, if you’re developing a single plug-in, then there’s no need for a master test suite. I suppose this is more of an issue for RCP developers who write applications made up of many plug-ins.

  5. Lukasz Bobowiec says:

    That’s great article. I investigated a half year ago the best approach. I chose fragments for unit tests. All advantages were described in details but there are also some troubles. I am using Declarative Services for my plugins so I do not write any Activator for my bundles. So the first problem I have encountered was how to get a bundle context? (resolved by getting it from component context and exposing some static method – I know that is ugly but it works), the second problem was how to start automatically a fragment? I know that fragment can be only in RESOLVED state (can not be started) but how to invoke unit tests on demand (e.g. from OSGi console)?

  6. pjpaulin says:

    Hii Lukasz,

    Thanks for the tip about declarative services. As far as starting the fragment from the OSGi console, I would imagine you would need to start the host plug-in itself. I’ve never tried to run fragment unit tests from the console, though, so I’m not sure if this would work or not. Let me know if you get it working.

  7. Tony says:

    I know this is off-topic, but can anyone point me to some resources on fragments? I’ve been successfully using fragments for my JUnit tests (though I haven’t taken the time to put together a master test suite using the techniques described herein just yet).

    As it turns out, I also have some optional functionality to implement, and thought I would implement it as a plugin fragment (which I’ve read is one of the intended uses of fragments).

    Now to me, supplying optional functionality means providing code (IE classes – in a fragment) that code in the host plugin can optionally make use of if it is available. You confirmed what I’d already learned the hard way when you said that “classes in fragments are not visible outside of the fragment itself”. My trial-and-error testing shows that this is indeed the case, and holds even for the fragment’s host. What I don’t understand is how it can be said that a fragment contributes its resources to its host if the host can’t access those resources because of visibility restrictions.

    I’m glad to see at least two possible workarounds, one in the main article and another in the comments, that I will surely consider using in my project. But before I resort to a workaround, I would prefer to understand the proper and intended use of the fragment facility, and for some reason I can’t figure out how to interpret anything I’ve seen so far as other than self-contradictory. Thanks.

  8. pjpaulin says:

    Hi Tony,

    Originally, fragments were created to support internationalization (resource bundles) and supplying platform specific libraries. In general, I think of fragments as useful only in situations that require code to run in the same classloader. JUnits are a good example of this, as sharing a classloader provides you access to non-public method.

    I don’t think fragments have anything to do with “optional” code. A plug-in would work just as well. But in either case, the only way for the optional code to make itself known is through the extension point mechanism. A fragment would support the use of reflection, of course, but that’s a bit ugly.

    There’s not a lot of good doc on fragments, but I did find this thread from a few years ago that discusses some of these issues.

    http://dev.eclipse.org/newslists/news.eclipse.platform.rcp/msg04165.html

    Hope it helps.

  9. Tony says:

    Paul,

    Thanks very much, it actually helps quite a bit. The link you supplied gave me exactly the information I was looking for, if not the answer I wanted. I had been putting off learning all about the extension mechanism, but I guess it’s time to jump right in…

  10. Dan says:

    Thanks Patrick to this document. I think is interesting use fragments but I think if is possible access from “com.mycompany.myplugin1.AllTests” to resources other IProjects in the same workspace.

    Thanks

  11. pjpaulin says:

    Hi Dan,

    I’m not exactly sure what your question is. If you’re trying to access other plug-ins from your AllTests class, you should be able to do this if you have the other plug-ins declared as dependencies for either the fragment or the host plug-in.

    You can’t access an AllTests in a fragment from anywhere outside the fragment.

  12. Nix says:

    Sorry if this is a bonehead question, but my Java skills are rather rusty and I just started with Eclipse…
    When you say “non-public” as in “This means that if you maintain the same package structure in your source and test folders, your test classes will have access to the non-public methods of the classes under test.”; do you mean “package and protected, but of course not private”?
    Or am I missing the point entirely?

  13. Agata Przybyszewska says:

    Thank you for this excellent post – I have been working with RCP applications for about 6 months, but have still not found an elegant way to deal with tests.

    I wonder what you mean by “one click away” – I tried to copy the code you propose for the master test suite – when I try to run it as a plug-in test the response is “no tests found” … so what is the magic click combination? I would just love to be able to run all my junit tests at once.

    Do you need to change anything in your build process if you move your tests into fragments (as opposed in an extra test source folder in the old plugin).

    BTW will you add the master test suite as a fragment to the main product plugin?

    Cheers,
    Agata

  14. pjpaulin says:

    Hi Agata,

    Sorry for the delay in getting back to you. I’ve been on vacation for the last week.

    As for the master test suite, it should exist in a separate plug-in, not a fragment. Also, make sure you declare dependencies in the master test plug-in for the plug-ins containing the code under test. In other words, you need your master test plug-in to depend on the plug-ins that serve as the hosts for the test fragments. It’s easy to forget to do this because you’re using reflection to get to the plug-in level test suites and so you don’t get a compile error.

    If you follow these steps and things still don’t work, it might be best to create a simple example and email it to me. I can try to help figure out what’s going on.

    Hope this helps,

    — Patrick

  15. pjpaulin says:

    Hi Nix,

    You’ve got it exactly right. If you keep the same package structure, you can access protected and default class members but obviously not private ones. I should probably make this more clear in the future!

    — Patrick

  16. Joerg Bullmann says:

    Hi Patrick,

    First of all thanks a lot for the article. I have started using fragments for my tests and it works like a charm.

    There is one thing (which does not seem related to the the fragment approach here) that does not work for me: Eclipse JUnit Plug-in Test launch configurations. The problem here seems that none of my test classes are seen and executed. I use JUnit 4 and Europa Fall 2 on Sun Java 1.6.

    If I just use a normal JUnit launch configuration I can test without a problem. This helps me doing smaller scale testing. But if I want to test the proper setup of my plug-ins and their interaction, I’m stuck.

    What might I be missing?

    Cheers,
    Joerg

  17. Patrick says:

    Hi Joerg,

    How are you creating your launch configurations?

    I usually right-click on a unit test or test suite and select Run As -> JUnit Plug-in Test. The test runs and a valid launch configuration is created.

    If you’re creating a launch configuration inside of the Run Dialog itself, there may be some config information missing (checked plug-ins, etc.)

    Hope this helps,

    — Patrick

  18. chandra says:

    Hi Patrick
    First I will explain all the things that I have done to implement multi-language support in my RCP application.
    (1)–> plugin_.properties files for English and French, where I have kept view titles, tool tip text and etc for each language.
    (2)–> message_.properties files for the multi language text support in Java files. Here I need to set the RCP application from ApplicationWorkbenchWindowAdvisor class, the title text will also be part of i18n text content.
    (3)–> Now I have created the plugin fragment project , where I have moved all the properties files in a proper manner.
    (4)–> Now I started the RCP application from product configuration file from eclipse IDE itself. In this case, though I have properties files in different projects, I am able to test the i18n implemention by switching the language settings in system
    (5)–> This is where the problem occurs, I am exporting the product as RCP product and I started the RCP application, Now I could see something strange. ie. the RCP uses proper plugin_.properties files from fragments and Throws error when it accesses the message_.properties files used in Java classes.
    (6)–> I checked the build files and everything, the plug-in Jar file and Fragment Jar contain all properties files in place properly.

    Please guide me why the ResourceBundle message properties files are not taking in it from fragments Jar file.

    Thanks,
    Chandra M

  19. Joerg Bullmann says:

    Hi Patrick,

    Sorry for coming back so very late. I had tried many places for info on this over the course of a few days and got no answer so I moved on to different tasks of the project. And — a classic — I then missed your answer which you posted so quickly.

    Only today (with a fresh mind) I came back to the problem. And I have solved it.

    It had to do with the plug-ins the test was launched with. Somehow — and there is almost no need to say it — not all necessary plug-ins were selected. All sorts of launch configuration settings and plug-in combinations I then tried failed. I must have carefully negotiated *around* the working setups — which I so often do.

    My project consists of UI and non-UI plug-ins. I want to unit-test only the underlying infrastructure, i.e. the non-UI plug-ins. I use JUnit 4 and Europa Fall 2 on Sun Java 1.6.

    Here now my recipe:

    0) Create a Run configuration of the type “JUnit Plug-in Test”; choose the JUnit 4 test runner
    1) Go to tab “Main”
    2) Select “Run as application” and select “[No Application] – Headless Mode”
    3) Go to tab “Plug-ins”
    4) Select “Launch with plug-ins selected below only”
    5) Select all plug-ins of the development workspace
    6) De-select everything in the target platform.
    7) In the target platform plug-in list, select “org.eclipse.jdt.junit4.runtime”
    8) Add all required plug-ins by pressing the button.
    9) Now pressing “Validate” should not find any problem.
    10) Save this.

    When I run this thing now, I get a headless Eclipse (which fires up much faster than the ide which makes the heck of a difference when running tests) with all my plug-ins (and the necessary targety platform plug-ins) in it and I can test all sorts of plug-in interaction, extension point initialization, etc. nicely in JUnit4 tests. I am *very* happy with this.

    I guess one of the confusing bits for me was this: in the run configuration managament dialog I select “JUnit Plug-in Test” and when I then go to the Plug-ins pane and press the “Add Required Plug-ins” button I somehow thought it would take into account that I am JUnit-testing.

    Also, I was not able to find out *why* my tests were not running. I wonder whether it would be possible for Eclipse to give some more ‘in the face’ diagnostics which might point out the way.

    Anyway — am a happy camper now.

    Thanks a lot for your article which helped me a lot in the first place and also for being so responsive and helpful.

    Cheers,
    Joerg

  20. Patrick says:

    HI Chandra,

    Sorry I missed your question earlier. Normally, if something is working in the IDE but not after export it is either an issue with the build.properties files (not including the resource bundles) or in the product configuration (not including the fragments in the config). It sounds like you’ve checked both of those already.

    I can’t really tell more without looking at some code. If you can come up with a simple case that fails, feel free to email me the plug-ins and I’ll take a look.

    — Patrick

  21. Joerg Bullmann says:

    Hi Patrick,

    To be able to run all my tests I have followed the basic idea of your master test suite. The code snippet you have posted seems JUnit 3 based so I have gone a slightly different route (no TestSuite class as such in JUnit 4 any more): I have written a small plug-in with a TestRunner class implementing IApplication. In here I collect and run the test classes using below code:

    for(String tn : allTestNames) {
    // some trying
    Class c = Class.forName(tn);
    BasicConfigurator.resetConfiguration(); // Reset Log4J before each test class
    Result res = JUnitCore.runClasses(c);
    // now collect some statistics from res for overall stats at the end.
    // some catching
    }

    // At the end: spit out statistics and failures.

    This seems to work fine. I use Log4J and each test class has a @BeforeClass method configuring the logging (necessary if I run the test class with a run configuration). To ensure things are clean before each test class is executed (after the previous one) I have the resetConfiguration() in the code above. I guess, I could put this also in a @AfterClass method in each test. With this setup, the test classes can be run from a run configuration and also from my TestRunner.

    How I see it is that in JUnit 4 many things seem pretty much annotation based and if I go and manually collect my test classes (as I had to) I have to manually execute them. But I might be missing some JUnit 4 bells and whistles here.

    If in Eclipse the JUnit Plug-in Test run configuration allowed to specify a list of projects to collect the test classes from (instead of a single project, package or folder only as it is now), then I could throw away my code and use this run configuration instead. And there would also be no need for the last section of your article. Or am I missing something?

    Cheers,
    Joerg

  22. Patrick says:

    Hi Joerg,

    I’m haven’t switched over to using JUnit 4 yet, so I don’t have any easy answers. I know there are @Suite, @SuiteClasses and @RunWith annotations that can be used to create suite-like test aggregations, but I’m not sure how they would work with fragments. If you have to specify the test classes in an annotation, there isn’t a good place to hook the reflection code to reach into the fragments.

    I’m sorry I don’t have a better answer. If you figure this out (without creating your own test runner), I’d be happy to hear about it.

    — Patrick

  23. Joerg Bullmann says:

    Hi Patrick,

    Having this test runner is not a problem, it’s straightforward and works fine. I just thought I’d share it here as it took me a little while to work out the difference between JUnit 3 and 4 (I have never used JUnit 3 myself and started with 4 just recently). Should I find more elegant ways of dealing with this, I’ll come back here and post.

    Cheers,
    Joerg

  24. Alan Bram says:

    I tried this, and it works, and I love having things separate like this — I think fragments are the ideal mechanism for this.

    However, I’m noticing that my tests now take much longer to run. Before I moved the tests to a separate fragment, they used to take about 22 seconds to run. Now they take around 33 seconds.

    Anyone have any ideas why?

    Cheers,
    – arb

  25. Patrick says:

    Hi Alan,

    Can you tell me how your tests were structured before? Were they in the same plug-in as the code under test or in a separate test plug-in? If the tests were in the same plug-in, the extra time could be related to loading the fragment into memory, but I’m not sure.

    It would be interesting to run those tests using a profiling tool to see where the extra time is going. I’d also be interested to know whether there is a scaling 50% increase in time for tests or whether there is simply a 10 second penalty.

    I’ll try to look into this at some point but I’m pretty buried at the moment :-) If you find out anything, please let me know.

    — Patrick

  26. Alan Bram says:

    Hi Patrick,

    Thank you for your reply.

    I found out why my tests had started to take longer, and it had nothing to do with the fragment change. When I made the big change, I moved everything into a new Eclipse workspace. I cloned all the project stuff from the old workspace, but I was not clever enough to also copy all of the Eclipse metadata. I mean, of course I could have copied all of it, but I was afraid that if I had done that I might have ended up retaining confusing, conflicting settings from the old project organization. In other words, I would have liked to clone only part of the metadata, but the prospect of trying to figure out which parts are which was daunting.

    So of course I had to redo a bunch of settings from scratch. I forgot that in the old days I had customized the VM parameters for my test run, to give it much more memory than the default. When I duplicated that config setting in my new workspace the tests once again started running about as fast as they used to.

    So now everything’s all great, and I’m really happy.

    Thanks again for the cool tip. And I apologize for the “crying wolf”.

    Cheers,
    – arb

  27. Patrick says:

    Hi Alan,

    No problem. I’m glad you figured it out and that the fragments weren’t increasing your test times.

    — Patrick

  28. […] why you’d want to use test fragments instead of test plug-ins, check out my previous post on Testing Plug-ins with Fragments. The problem is that even the standard suite-based solution does not work with fragments. There are […]

  29. Nihal says:

    Hey Patrick,

    We’ve been using the fragment option for almost a year now. As you may recall you helped us to get our RCP app up and running. Anywho, in using fragments as containers for tests, I’ve found that it’s useful to have “Eclipse-ExtensibleAPI: true” in the manifest for the host plugin. By default this value is false. This allows fragments to contribute API to the host plugin and is useful if you want to provide utility classes and methods for testing that other test fragments can use.

    For example:

    ModelPlugin:
    -SomeModel class

    ModelTestPlugin:
    -ModelTests class — extends TestCase and provides helper methods to create SomeModel objects
    -SomeModelTest — extends ModelTests; just for completeness :]

    PresentationPlugin:
    -SomePresenter class

    PresentationTestPlugin:
    -SomePresenterTest class — extends ModelTests*

    * Without specifying the “Eclipse-ExtensibleAPI: true” in the manifest for ModelPlugin, I would not be able to extend ModelTests in PresentationTestPlugin (would get compilation errors). Of course the disadvantage to this is that now PresentationPlugin is going to have visiblity to ModelTests, but that’s something I’m willing to live with. Perhaps there’s still a better way yet?

    Cheers,
    Nihal

    Reference:

    http://help.eclipse.org/help33/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/bundle_manifest.html

  30. Patrick says:

    Hi Nihal,

    Your approach makes sense. The only alternative I see is to create some common test plug-in that could contain reusable test classes. This would eliminate the visibility leakage issue you have, but as you say it’s not really a big deal.

    Thanks for posting this tip. I think it will help others trying to do fragment-based testing.

    — Patrick

  31. […] know that the osgi classloading it a bit different as usual. In Equinox there are also Buddies and Fragments which effect the […]

  32. […] plug-ins or fragments can be run without using test suites. In one of his previous articles (Testing Plug-ins with Fragments) he explains how fragments can be used to separate the test code from the business […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: