Selectively inherit xunit tests

Inheriting selective Xunit tests from another assembly

Use case: I have a single assembly which implements several xunit tests, but want to add Visual Studio projects with new tests but also selectively add tests from the "base assembly".

To do this we'll support an enumeration and assembly level attributes to specify which tests to run in the "super" tests. E.g.

   public enum GeneralFacts
   {
      GenericSearch,
      GenericPagination
   }

And an attribute such as:

   [AttributeUsage(AttributeTargets.Assembly)]

   public class IncludeGeneralFactAttribute : Attribute
   {
      public readonly GeneralFacts Facts;
 
      public IncludeGeneralFactAttribute(GeneralFacts facts)
      {
         Facts = facts;
      }
   }

So the idea is, you define several tests in your base assembly, e.g.

      [GeneralFact(GeneralFacts.GenericSearch)]
      public void GenericSearchTest1()
      {
         Assert.True(true);
      }
 
      [GeneralFact(GeneralFacts.GenericPagination)]
      public void GenericPaginationTest2()
      {
         Assert.True(false);
      }

And in your super assembly, one that references the base assembly, you include assembly level attributes like

[assembly: IncludeGeneralFact(GeneralFacts.GenericSearch)]

By which we indicate: we support the "GenericSearch" feature and its tests but not the GenericPagination one.

Attempt 1

Simple, override Xunit's FactAttribute as we see above and override its EnumerateTestCommands method to see which general facts were included on the assembly by setting the IncludeGeneralFact attribute on the test assembly.
Two problems with this: Xunit does not look for the FactAttribute with inheritance of attributes, second, we need to get at the actual assembly being tested to be able to get those IncludeGeneralFact attributes.

Attempt 2

Ok, this needs more work. Doing some serious Googling I got no further so one has to resort to reading Xunit source code, which, thankfully, is easy to read. Diving into the source we see that ultimately there is a method GetCustomAttributes on IMethodInfo that is used by Xunit to find all [Fact] methods. So we need to find a way to get our own wrapper in there.
In turns out that one way to do this is to use a custom "runner" and add a RunWithAttribute to a class, something like this:

[RunWith(typeof(InheritTestClassCommands))]
public class InheritTests : AttributeExampleBase {
}

Here AttributeExampleBase is the class that contains the tests above. Of course, another way to do all this is to simply inherit from the class whose tests you want to inherit (e.g. a SearchGenericTests class etc.)

So we have to implement InheritTestClassCommands. This class will just be a wrapper where every interface call we'll simply forward to an instance of TestClassCommand, which is the Xunit native implementation. The only real change lies in its TypeUnderTest method which will look like this:

      public Xunit.Sdk.ITypeInfo TypeUnderTest
      {
         get
         {
            return _mWrappedCommand.TypeUnderTest;
         }
         set
         {
            _mWrappedCommand.TypeUnderTest = value is WrappedTypeInfo ? value : new WrappedTyeInfo(value);
         }
      }

The WrappedTypeInfo wraps the ITypeInfo implementation provided by Xunit. The only change there is that we want to wrap IMethodInfo, so the only methods with any beef are:

      public IMethodInfo GetMethod(string methodName)
      {
         var innerMethod = _mInner.GetMethod(methodName);
         if (innerMethod == null) return null;
         return new WrappedMethodInfo(innerMethod, this);
      }
 
      public IEnumerable<IMethodInfo> GetMethods()
      {
         var innerMethods = _mInner.GetMethods();
         if (innerMethods == null) return innerMethods;
         return innerMethods.Select(m => new WrappedMethodInfo(m, this));
      }

Thus we're wrapping IMethodInfo with WrappedMethodInfo, where the only change is the fact we want *inherited* attributes, thus:

      public IEnumerable<IAttributeInfo> GetCustomAttributes(Type attributeType)
      {
         return MethodInfo.GetCustomAttributes(attributeType, inherit: true).Select(t => Reflector.Wrap((Attribute)t));
      }
 
      // Change: inherit attributes.
      public bool HasAttribute(Type attributeType)
      {
         return MethodInfo.IsDefined(attributeType, inherit: true);
      }

So, that's it. Well, almost. Xunit's implementation depends on the Equals and GetHashCode methods being implement on IMethodInfo (which is strange and asymmetrical if looking at ITypeInfo in my opinion). All they need to do is say sure I'm equal to the other fella if my method is the same, i.e.:

      public override bool Equals(Object obj)
      {
         WrappedMethodInfo other = obj as WrappedMethodInfo;
         if (other == null)
            return false;
 
         return this.MethodInfo == other.MethodInfo;
      }
 
      public override int GetHashCode()
      {
         return this.MethodInfo.GetHashCode();
      }

It is probably also a good idea to ensure that ToString is forwarded in all cases.
Finally, we need to communicate the ITypeInfo to the methodinfo so that we can get at it in the actual customized fact attribute. This we can simply do passing it to the constructor of WrappedMethodInfo. The custom Fact attribute then looks like this:

   public sealed class GeneralFactAttribute : Xunit.FactAttribute
   {
      private readonly GeneralFacts _mFact;
 
      public GeneralFactAttribute(GeneralFacts fact)
      {
         _mFact = fact;
      }
 
      protected override IEnumerable<Xunit.Sdk.ITestCommand> EnumerateTestCommands(Xunit.Sdk.IMethodInfo methodInfo)
      {
         var method = methodInfo as WrappedMethodInfo;
         if (method == null) return base.EnumerateTestCommands(methodInfo);
 
         // Get the assembly under test.
         var asm = method.TypeUnderTest.Type.Assembly;
 
         if (asm.GetCustomAttributes(typeof(IncludeGeneralFactAttribute), inherit: true)
                .Cast<IncludeGeneralFactAttribute>()
                .Any(att => att.Facts == _mFact))
            return base.EnumerateTestCommands(method);
 
         return Enumerable.Empty<Xunit.Sdk.ITestCommand>();
      }
   }

That's it, we can now specify which tests we want inherited in our special projects.

If there's an easier way to do this or if you want more info I can always be found on twitter @marcusvanhoudt.