NUnit result file result classification

Now that we are moving away from Launchpad, it's better to ask
questions in our nunit-discuss google group. I've modified the project
page on Launchpad to indicate that.

Historically, NUnit only had the attributes executed and success.
Tests with executed=true were successful or failed depending on the
value of the success attribute. Those with executed=true didn't have a
success attrbute at all.

The result attribute was added at a later time, in order to cope with
new results beyond not run, success and failure. This gives the most
complete information on the result of a test and should be used by new
programs. The executed and success attribtes are kept for backward
compatibility.

Additionally, many report programs don't look at either of these. For
example, to list failures they simply find all tests with a <failure>
element.

To answer your conceptual questions:
  * No, NUnit does not consider ignored or inconclusive tests as successful.
  * A test suite is not considered successful unless it contains no
errors, failures or ignored items. Inconclusive tests don't affect the
result of a normal suite at all but may affect the result of a suite
representing a Theory - that is, if _all_ tests are inconclusive, the
theory is considered to fail. Essentially, each type of suite gets to
decide how it aggregates the results of its child tests.

NUnit 3.0 is using an entirely new result file format, by the way.

Charlie

On Thu, Nov 7, 2013 at 7:01 AM, Peter Brightman
<email address hidden> wrote:
> New question #238839 on NUnit Framework:
> https://answers.launchpad.net/nunit-3.0/+question/238839
>
> Hello all,
>
> i recently tried to count the results of a NUnit resultfile using xpath expressions. I have used version 2.5.10. What confuses me are the attributes of the test-case nodes. I have noticed the following attributes:
>
> executed
> result
> success
> time
> asserts
>
> So i have created some variables that count the results that way:
>
> <xsl:variable name="TestsAllCount" select="count(/test-results//test-case)"/>
> <xsl:variable name="FixturesFailed" select="count(/test-results//test-suite[failure][@type='TestFixture' and @result='Failure'])"/>
> <xsl:variable name="TestsFailed" select="count(/test-results//test-case[failure][@executed='True' and @success='False'])"/>
> <xsl:variable name="TestsPassed" select="count(/test-results//test-case[@executed='True' and @result='Success'])"/>
> <xsl:variable name="FixturesIgnored" select="count(/test-results//test-suite[@type='TestFixture' and @result='Ignored'])"/>
> <xsl:variable name="TestsIgnored" select="count(/test-results//test-case[@executed='False' and @result='Ignored'])"/>
> <xsl:variable name="FixturesInconclusive" select="count(/test-results//test-case[not(failure)][@executed='True'][@result='Inconclusive'])"/>
> <xsl:variable name="TestsInconclusive" select="count(/test-results//test-case[not(failure)][@executed='True'][@result='Inconclusive'])"/>
> <xsl:variable name="TestsNotExecuted" select="count(//test-case[@executed='False'])"/>
>
> Is this correct or not?
>
> I dig that ignored and inconclusive test have the executed-attribute set to 'False' but what for is the attribute "success"? Isn't a test successful even if it is ignored or inconclusive? As long as the attribute "success" just takes True|False as values, i wonder why i just can't use the attribute "result" that seems not to have just True|False as value but friendly names like "success", "failure" or "error" or "inconclusive" or "ignored". Is there any documentation about what values are set in what cases? What if a test-suite has 4 test-cases, one test is "Successful", another "Error", another "Ignored" and the last one is "Inconclusive". What value holds the attribute "result" of the appropriate test-suite of the mentioned test-cases? Does it contain all of the values of the test-cases in a comma-separated list? Please help. I am confused. Thank you.
>
> Peter
>
>
>
>
>
>
>
> --
> You received this question notification because you are a member of
> NUnit Core Developers, which is an answer contact for NUnit Framework.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~nunit-core
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~nunit-core
> More help : https://help.launchpad.net/ListHelp

Thanks for explaining. You wrote

"Additionally, many report programs don't look at either of these. For
example, to list failures they simply find all tests with a <failure>
element."

I do this as well if you look at my xpath expressions, but since a test-fixture can also fail and so will also have a <failure> childtag, not just test-case tags have to be examined but also test-suite tags (those of type TextFixture).

But now my xpath seems to not work anymore, because i have the case where a test-suite failed, and so of course all the test-case tags have also a result="Failure" set. But the attribute result of the test-suite tag has not the value "Failure" but "Error". This means that this xpath expression doesn't work:

<xsl:variable name="FixturesFailed" select="count(/test-results//test-suite[failure][@type='TestFixture' and @result='Failure'])"/>

This relates to my initial question, what values can the attribute "result" take and what are the cases. When does it take the value "Failure" and when does it take the value "Error"? What's the difference between "Failure" and "Error"? Are there any other values that can occurr?

Peter

The result attribute can take on any of the values defined for the
ResultState enum. See
http://bazaar.launchpad.net/~nunit-core/nunitv2/trunk/view/head:/src/NUnitCore/interfaces/ResultState.cs

Charlie

On Fri, Nov 8, 2013 at 5:51 AM, Peter Brightman
<email address hidden> wrote:
> Question #238839 on NUnit Framework changed:
> https://answers.launchpad.net/nunit-3.0/+question/238839
>
> Status: Answered => Open
>
> Peter Brightman is still having a problem:
> Thanks for explaining. You wrote
>
> "Additionally, many report programs don't look at either of these. For
> example, to list failures they simply find all tests with a <failure>
> element."
>
> I do this as well if you look at my xpath expressions, but since a test-
> fixture can also fail and so will also have a <failure> childtag, not
> just test-case tags have to be examined but also test-suite tags (those
> of type TextFixture).
>
> But now my xpath seems to not work anymore, because i have the case
> where a test-suite failed, and so of course all the test-case tags have
> also a result="Failure" set. But the attribute result of the test-suite
> tag has not the value "Failure" but "Error". This means that this xpath
> expression doesn't work:
>
> <xsl:variable name="FixturesFailed" select="count(/test-results//test-
> suite[failure][@type='TestFixture' and @result='Failure'])"/>
>
> This relates to my initial question, what values can the attribute
> "result" take and what are the cases. When does it take the value
> "Failure" and when does it take the value "Error"? What's the difference
> between "Failure" and "Error"? Are there any other values that can
> occurr?
>
> Peter
>
> --
> You received this question notification because you are a member of
> NUnit Core Developers, which is an answer contact for NUnit Framework.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~nunit-core
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~nunit-core
> More help : https://help.launchpad.net/ListHelp