Revisiting the migration path to Kyua

[Julio Merino]

Hello all,

Almost a year ago already (wow... time flies), I imported Kyua into the src tree as a replacement for the old ATF tools. At the same time, I also imported some Kyua-based alternatives for atf-run and atf-report, which are compatible on the UI side of things but don't behave in exactly the same manner.

All of the above is protected by a MKKYUA knob, which is still set to no by default.

In order to make progress on this project, I'd like to change the approach to the migration to make it easier for developers and users to test Kyua.

Proposal: make MKKYUA=yes NOT replace atf-run and atf-report with compatibility wrappers. Instead, allow both kyua and atf-run/atf-report to coexist in the same installation, and then set MKKYUA=yes by default. (We'd probably kill the import of kyua-atf-compat along the way as well; yay, less code.)

This would allow 1) the continuous testing machines to work just fine without any changes to them and 2) it would also allow end users to start playing with the new tools without the need to rebuild the system. We'd then more easily and progressively evaluate the change.

Thoughts? Objections?

Thanks.

[Paul Goyette]

On Fri, 14 Feb 2014, Julio Merino wrote:

8< snip >8

Post by Julio Merino
Proposal: make MKKYUA=yes NOT replace atf-run and atf-report with
compatibility wrappers. Instead, allow both kyua and
atf-run/atf-report to coexist in the same installation, and then set
MKKYUA=yes by default. (We'd probably kill the import of
kyua-atf-compat along the way as well; yay, less code.)
This would allow 1) the continuous testing machines to work just fine
without any changes to them and 2) it would also allow end users to
start playing with the new tools without the need to rebuild the
system. We'd then more easily and progressively evaluate the change.
Thoughts? Objections?

Sounds like a good idea to me.


-------------------------------------------------------------------------
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com |
| Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net |
| Kernel Developer | | pgoyette at netbsd.org |
-------------------------------------------------------------------------

[Justin Cormack]

Post by Julio Merino
Hello all,
Almost a year ago already (wow... time flies), I imported Kyua into the src tree as a replacement for the old ATF tools. At the same time, I also imported some Kyua-based alternatives for atf-run and atf-report, which are compatible on the UI side of things but don't behave in exactly the same manner.
All of the above is protected by a MKKYUA knob, which is still set to no by default.
In order to make progress on this project, I'd like to change the approach to the migration to make it easier for developers and users to test Kyua.
Proposal: make MKKYUA=yes NOT replace atf-run and atf-report with compatibility wrappers. Instead, allow both kyua and atf-run/atf-report to coexist in the same installation, and then set MKKYUA=yes by default. (We'd probably kill the import of kyua-atf-compat along the way as well; yay, less code.)
This would allow 1) the continuous testing machines to work just fine without any changes to them and 2) it would also allow end users to start playing with the new tools without the need to rebuild the system. We'd then more easily and progressively evaluate the change.
Thoughts? Objections?

Sounds like a good idea to me.

Justin

[Andreas Gustafsson]

Julio,

Post by Julio Merino
Proposal: make MKKYUA=yes NOT replace atf-run and atf-report with
compatibility wrappers. Instead, allow both kyua and
atf-run/atf-report to coexist in the same installation, and then set
MKKYUA=yes by default. (We'd probably kill the import of
kyua-atf-compat along the way as well; yay, less code.)
This would allow 1) the continuous testing machines to work just
fine without any changes to them

I don't think that's quite true. I followed your suggestion last year
of making the "anita test" command run kyua if the system was built
with MKKYUA=yes or atf-run + atf-report otherwise, and if I understand
your new proposal correctly, because it involves setting MKKYUA to
"yes", it would cause the continuous testing machines to run the kyua
tests but not the ATF tests. Would we not want them to either continue
running the ATF tests, or to run both of them?

Post by Julio Merino
and 2) it would also allow end users to start playing with the new
tools without the need to rebuild the system. We'd then more easily
and progressively evaluate the change.

Has there been any progress on the issue of the kyua output needing
much more disk space than the ATF output does?

How much disk space and how many inodes per test run will it take to
store the kyua output initially, if your proposal is implemented but
assuming no additional kyua tests have yet been created or ported over
from the ATF side?

[Julio Merino]

Post by Andreas Gustafsson
Julio,

Post by Julio Merino
Proposal: make MKKYUA=yes NOT replace atf-run and atf-report with
compatibility wrappers. Instead, allow both kyua and
atf-run/atf-report to coexist in the same installation, and then set
MKKYUA=yes by default. (We'd probably kill the import of
kyua-atf-compat along the way as well; yay, less code.)
This would allow 1) the continuous testing machines to work just
fine without any changes to them

I don't think that's quite true. I followed your suggestion last year
of making the "anita test" command run kyua if the system was built
with MKKYUA=yes or atf-run + atf-report otherwise, and if I understand
your new proposal correctly, because it involves setting MKKYUA to
"yes", it would cause the continuous testing machines to run the kyua
tests but not the ATF tests. Would we not want them to either continue
running the ATF tests, or to run both of them?

The way things are now (and the way things were last year), when you set MKKYUA=yes, atf-run and atf-report get replaced by shell scripts that invoke kyua underneath. The CLI of these scripts is compatible with the real ATF tools, but the semantics of their execution differ. This is why flipping the flag on your system showed some differences.

With my new proposal, setting MKKYUA=yes would have no effect on atf-run nor atf-report at all. These tools would continue to be the "real ATF tools" and, to invoke Kyua, you'd need to execute kyua explicitly. Users would then need to manually switch their invocations to use kyua(1) instead, and we'd work on cleaning up any rough edges in the tool without affecting atf-* nor rushing to replace them.

[Andreas Gustafsson]

Post by Julio Merino
The way things are now (and the way things were last year), when you
set MKKYUA=yes, atf-run and atf-report get replaced by shell scripts
that invoke kyua underneath. The CLI of these scripts is compatible
with the real ATF tools, but the semantics of their execution
differ. This is why flipping the flag on your system showed some
differences.
With my new proposal, setting MKKYUA=yes would have no effect on
atf-run nor atf-report at all.

It may have no effect on atf-run or atf-report, but it will have an
effect on anita which greps for MKKUYA=yes in /etc/release and behaves
differently depending on whether or not it's there, as you requested.

[Julio Merino]

Post by Andreas Gustafsson

Post by Julio Merino
The way things are now (and the way things were last year), when you
set MKKYUA=yes, atf-run and atf-report get replaced by shell scripts
that invoke kyua underneath. The CLI of these scripts is compatible
with the real ATF tools, but the semantics of their execution
differ. This is why flipping the flag on your system showed some
differences.
With my new proposal, setting MKKYUA=yes would have no effect on
atf-run nor atf-report at all.

It may have no effect on atf-run or atf-report, but it will have an
effect on anita which greps for MKKUYA=yes in /etc/release and behaves
differently depending on whether or not it's there, as you requested.

Oh, that's a good point that I forgot about!

If we go with this proposal, we need to change Anita to decouple the enabling of Kyua in the base system by default with the selection of the tool used in your test machines.

I think it'd make sense to add a command-line flag to "anita test" to let the caller specify whether atf or kyua is to be used -- and to ignore MKKYUA altogether. Something along the lines of:

anita test --engine=atf ...
anita test --engine=kyua ...

the former being the default for now, and the latter being offered for experimentation purposes. The default could change at a later point once you are happy with what Kyua offers (not yet, I know).

I think the changes should be easy. Should I try to come up with a diff?

Thanks!

[Andreas Gustafsson]

Post by Julio Merino
If we go with this proposal, we need to change Anita to decouple the
enabling of Kyua in the base system by default with the selection of
the tool used in your test machines.
I think it'd make sense to add a command-line flag to "anita test"
to let the caller specify whether atf or kyua is to be used -- and
to ignore MKKYUA altogether.

Would we not also want the ability to run both? Will there be a
transition period when some tests are only in ATF and others are
only in Kyua?

Also, the command line flag approach is awkward for testing historical
versions, as you would have to manually figure out the appropriate
setting for the particular version you want to test. It might be
better for anita to try running both ATF and Kyua, and for the
higher-level test harnesses that generate HTML reports to publish the
results from whichever one worked, or both.

Post by Julio Merino
I think the changes should be easy. Should I try to come up with a diff?

I can make the changes to anita once we have decided what they
should be.

But I'm afraid I still don't quite understand why this change in
transition strategy is needed. The main problem I found with
turning on MKKYUA in the current setup was a dramatic increase in
disk space usage for the test results, and I don't think this change
will solve that problem but only postpone the growth until the
existing tests are migrated, so what problems will it solve?

[Julio Merino]

Post by Andreas Gustafsson

Post by Julio Merino
If we go with this proposal, we need to change Anita to decouple the
enabling of Kyua in the base system by default with the selection of
the tool used in your test machines.
I think it'd make sense to add a command-line flag to "anita test"
to let the caller specify whether atf or kyua is to be used -- and
to ignore MKKYUA altogether.

Would we not also want the ability to run both? Will there be a
transition period when some tests are only in ATF and others are
only in Kyua?

No, I'd rather not do that because maintaining two different test sets will become ugly very very quickly. Both tools should run the exact same tests.

Post by Andreas Gustafsson
Also, the command line flag approach is awkward for testing historical
versions, as you would have to manually figure out the appropriate
setting for the particular version you want to test. It might be
better for anita to try running both ATF and Kyua, and for the
higher-level test harnesses that generate HTML reports to publish the
results from whichever one worked, or both.

Maybe, but as you say there are problems that prevent you from doing that right now. Also, doing that would double the time required to run tests, and we know that qemu is super-slow for this... so I'm not sure it's a good idea.

We could reassess this later though because running the tests through both tools would provide a mechanism to compare results and ensure that they match.

The idea of the flag is only so that developers (I suspect only me though) using anita can choose to test the new tool iff they want to. Your test machine should never have to use that flag: it should use the default behavior in anita and, when we decide that kyua is good enough, flip the default which would then switch the behavior of your machines.

Post by Andreas Gustafsson

Post by Julio Merino
I think the changes should be easy. Should I try to come up with a diff?

I can make the changes to anita once we have decided what they
should be.
But I'm afraid I still don't quite understand why this change in
transition strategy is needed. The main problem I found with
turning on MKKYUA in the current setup was a dramatic increase in
disk space usage for the test results, and I don't think this change
will solve that problem

Right, it won't, and the whole point of this change is to hide the problem from you so that progress can be made _on other fronts_. (Don't get me wrong: I do have plans to address your concerns, but they involve quite a bit of work and I haven't been able to commit the necessary time.)

I'm hoping that once people have an easier way to start testing kyua they will, and I'm sure that _more_ problems will be discovered once that happens. I'd rather uncover what these issues are sooner rather than later.

Post by Andreas Gustafsson
but only postpone the growth until the
existing tests are migrated, so what problems will it solve?

With "tests" you mean "machines running tests" here?

[Andreas Gustafsson]

Post by Julio Merino

Post by Andreas Gustafsson
Would we not also want the ability to run both? Will there be a
transition period when some tests are only in ATF and others are
only in Kyua?

No, I'd rather not do that because maintaining two different test
sets will become ugly very very quickly. Both tools should run the
exact same tests.

OK, it wasn't clear to me before that both tools would run the same
tests; that sounds good.

Post by Julio Merino

Post by Andreas Gustafsson
It might be
better for anita to try running both ATF and Kyua, and for the
higher-level test harnesses that generate HTML reports to publish the
results from whichever one worked, or both.

Maybe, but as you say there are problems that prevent you from doing
that right now. Also, doing that would double the time required to
run tests, and we know that qemu is super-slow for this... so I'm
not sure it's a good idea.

Given that both tools run the same tests, I agree that running both
would be wasteful. Maybe we could add a command line option like you
suggested, but make it fall back to ATF when Kyua is selected but not
available, as the case may be when testing a historic version?

Post by Julio Merino
I'm hoping that once people have an easier way to start testing kyua
they will, and I'm sure that _more_ problems will be discovered once
that happens. I'd rather uncover what these issues are sooner
rather than later.

Fair enough.

Post by Julio Merino

Post by Andreas Gustafsson
but only postpone the growth until the
existing tests are migrated, so what problems will it solve?

With "tests" you mean "machines running tests" here?

I meant "test cases", but never mind, I was incorrectly assuming that
the two tools would run different tests.

[Julio Merino]

Post by Andreas Gustafsson
Given that both tools run the same tests, I agree that running both
would be wasteful. Maybe we could add a command line option like you
suggested, but make it fall back to ATF when Kyua is selected but not
available, as the case may be when testing a historic version?

I don't have a strong preference (it's your software after all!), but _personally_ I'm not a big fan of such automatic fallbacks. If the user says --test-engine=kyua, then I'd expect the software to do as I said or fail if it's not possible. (However if instead the user said --test-engine=auto, then it'd be just fine to detect Kyua and fall back to ATF if not present.) Up to you really. Maybe a tristate (--test-engine=[atf|kyua|auto]) is nicest?

Do you want me to come up with a draft diff for this?

Thanks.

[Andreas Gustafsson]

Post by Julio Merino

Post by Andreas Gustafsson
Maybe we could add a command line option like you
suggested, but make it fall back to ATF when Kyua is selected but not
available, as the case may be when testing a historic version?

I don't have a strong preference (it's your software after all!),
but _personally_ I'm not a big fan of such automatic fallbacks.
If the user says --test-engine=kyua, then I'd expect the software to do
as I said or fail if it's not possible.

Having some kind of automatic selection is unavoidable, because
running "anita test" without any special options needs to work for any
version of NetBSD. If the NetBSD version in case supports only ATF,
it needs to run ATF, and if it supports only Kyua, it needs to run
Kyua.

I would think of the command line option not as a way of selecting a
test engine, but as a way of specifying a preference for which one to
use in the case where the version being tested happens to support
both.

Post by Julio Merino
Do you want me to come up with a draft diff for this?

I don't think that's needed.