Re: EXTERNAL: Re: List of Known Issues for a particular release

[Elijah Newren <newren@gmail.com>]

Hi Mark,

On Thu, Jul 11, 2019 at 12:02 PM Mark T. Ortell
<mtortell@ra.rockwell.com> wrote:
>
> Elijah,
>
> Thanks for the response. I am not clear whether the test_expect_failure means that the test is trying to do something that should fail and so it is a valid test case or if it is a test case that is failing, but should succeed and has only been temporarily disabled until it is fixed. I'm guessing the former. In this case, if it successfully did whatever it were testing, that would be an issue. A simple example of this would be a test to try to login with an invalid username and password. That is expected to fail and if it passed, it would be an issue. If this is the case, then it doesn't look like it provides a list of issues. Please clarify what the test_expect_failure indicates.

Please don't top-post on this list.

test_expect_failure (as well as other helper functions in the test
harness library, such as test_expect_success, test_must_fail,
test_might_fail, etc.) are explained in t/README.  By its definition,
it technically satisfies "list of known issues" as you asked for.
However, most software products that publish a list of known issues
has probably curated problems that users are likely to see or be
curious about, and which they want to inform users of both to reduce
support load and help users avoid problems.

This list is not curated in any such way.  It's just a list of issues
developers thought to document for themselves and/or other developers.
It is thus way different than what you might want:

(1) There is evidence that some have used it for "In an ideal world,
this thing should support this feature too in which case I'd expect it
to behave a certain way that it doesn't yet."  The line between
feature (what works is fine but we could make it better) and bug (it's
not really correct if it doesn't do it this way) gets really blurry at
times, and you'd pick a much different tradeoff in communication
between developers than you would in communication from developers to
users; with other developers you spend a lot more time talking about
internals and goals and direction we'd like to move the software in.

(2) Also, some of these "known breakages" could be in corner cases
that are very unlikely to be hit be users, and perhaps not only likely
to be hit by individual users, but unlikely that anyone anywhere will
ever hit that error (some of the merge recursive tests I added might
fall into that category).

(3) There may also be cases where someone once thought that optimal
behavior would be a little different and that they were planning to
implement more features, and then later changed their mind but forgot
to clean up the testcases.

(4) ...and that's just a few off the top of my head.  I'm sure the
list has several other things that make it not quite match what you
want.

As such, Brian's answer to your question elsewhere in this thread is
probably better than mine, but if by chance you are just being forced
to go through a box checking exercise and there's no reason for
needing these results other than that someone asked that they be
provided (I sometimes had to go through such exercises when I worked
at Sandia National Labs years ago), then technically the command I
gave you could be used to satisfy it.

> Below is the output from the provided command:
<snip>

Not sure why you included this.

> Regards,
> Mark


Best Wishes,
Elijah