Regular primitive bats uses assertions like '[ $foo = something ]'.
These are worthless for debugging: when they fail, all you know
is that foo is not "something" but you don't know what foo _is_.
Find and replace those assertions with 'assert', which is
more informative. Instances found via:
$ ack '^ *\[' tests/*.bats
There are many matches for 'test' (instead of '[') but those
mostly look like file-existence ones, which are less evil
than string-check tests. I'm leaving those be for now.
Signed-off-by: Ed Santiago <santiago@redhat.com>
This is much scarier than I had intended. The intention is to
start migrating from 'expect_output' and to use 'assert'
instead; the reason is that 'assert' allows much more
flexibility, particularly negative tests.
We've long wanted something like "assert that output does
not contain 'foo'". I've been too lazy to implement it,
but last week I noticed code in bud.bats that does:
! expect_output "sdfsdfsdf"
This is a really super bad idea: although it works fine when
things are good and tests pass, it's a disaster when tests
fail because the poor person debugging test output now sees
a test failure log, and starts tracking it down, and then
much later realizes that it was an intentional failure but
the '!' was negating it, and all that track-down work was
wasted.
To keep this PR reasonable, I'm keeping all positive uses
of expect_output untouched. We can migrate those (if desired)
over time. All negative uses are now assertions.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Show of hands: who here loves submitting a PR, then coming back
hours later to find one job failed, then spending time poring
over logs and finding a network error? Anyone? Anyone?
This is a lame attempt to minimize such flakes by caching
commonly-used images and restoring them on demand. We
introduce a new helper, _prefetch(), which podman-pulls
an image the first time, podman-saves it, then on
subsequent calls (for the same image) podman-loads it:
@test foo {
_prefetch alpine busybox
...tests that run buildah-from either
}
This is an imperfect solution: it is incomplete and will
grow more so over time as new tests are added. It is
difficult to verify its coverage. I'm really unhappy
with it but if it works, the Total Sum Of Unhappiness
might decrease overall thanks to fewer flakes. If it
doesn't work, it's trivial to remove _prefetch calls
using a sed script. Shall we give it a chance?
Signed-off-by: Ed Santiago <santiago@redhat.com>
The SELinux spc test is breaking under some CI circumstances in
Fedora 30; most notably, a CI-built buildah (with type != bin_t)
is failing with:
FAIL: container context matches our own
expected: 'system_u:system_r:unconfined_service_t:s0'
actual: 'system_u:system_r:spc_t:s0'
It is not clear to me what the initial purpose of this test
is, since (I believe) the container context should always
be spc_t regardless of the calling process context; likewise,
role should always be system_r.
This commit tweaks the test to check role/type/range
individually.
Needed for #2115
Signed-off-by: Ed Santiago <santiago@redhat.com>
Closes: #2125
Approved by: rhatdan
Replace this pattern:
cid=$(buildah ...)
with:
run_buildah ...
cid=$output
As a special case, if the buildah command is 'from' and the
string 'scratch' does not appear in the arguments, add '--quiet'.
Otherwise we'll get Pulling messages as part of $output. This
is an overkill solution.
For ease of review, this commit was entirely machine-generated via:
$ perl -pi -e 's{^(\s+)(\S+)=\$\(buildah (.*)\)$}{$indent=$1;$var=$2;$cmd=$3; $cmd =~ s/(\bfrom)\s(?!.*scratch)/$1 --quiet $2/; "${indent}run_buildah $cmd\n$indent$var=\$output"}e' *.bats
Signed-off-by: Ed Santiago <santiago@redhat.com>
Closes: #2029
Approved by: rhatdan
PR #1935 removed the default --debug logging; so all run_buildah
calls now use the default (error). It is safe to remove unnecessary
instances of --log-level=error .
For ease of review, this commit was entirely machine-generated via:
$ perl -pi -e 's/ --log-level=error / /' *.bats
Signed-off-by: Ed Santiago <santiago@redhat.com>
Closes: #2029
Approved by: rhatdan
The previous log-level implementation does not seem to work, which is
now fixed and aligns to other projects like podman and CRI-O. Therefore
the `--debug` flag has been deprecated and is now hidden.
Added documentation as well as integration tests.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Closes: #1811
Approved by: rhatdan
...and make the "$output" argument implicit, as well as the
description text. This greatly simplifies its invocation.
Also: make it test for exact matches unless --substring
option is passed; this eases my mind about a few potentially
ambiguous situations such as the one in #1464.
Also: add comments to two truth-table check functions
Also: reorder some config checks alphabetically, for ease
of reading.
Thanks to Tom Sweeney for the suggestions and encouragement.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Closes: #1492
Approved by: rhatdan
This started off as bug fixes necessary to get BATS tests
actually working on RHEL8 (they weren't). It grew. I will
defend my actions in the first comment post.
Primary change: import some helpers from podman BATS tests,
most importantly 'run_buildah' and 'is'. The vast majority
of the changes you'll see here are of the form:
- run buildah ...
- [ $status = 0 ]
- [ check $output ]
+ run_buildah ... ! automatically checks status
+ is "$output" "..."
Also: chmod'ed some files -x. Necessary because rpmbuild tries
to be oh-so-clever about requirements, and when it sees an
executable file with a shebang line like '#!env bats' it
helpfully adds 'Requires: /usr/bin/bats' to the rpm, which
then fails to install because RHEL8 does not have bats.
Also: refactored duplicate code in a few places, by writing
and invoking module-specific helper functions.
Also: changed a handful of 'buildah's to run_buildah, in
order to get error checking and debug logging.
Also: added descriptive reasons to many "skip"s.
Also: selinux test: some tweakery to make it run on production
system (context is different if /usr/bin/buildah is chcon'ed
appropriately). I can't get this test to pass on Fedora from
a build dir, and I'm actually not convinced that this test
has ever passed, but let's see what CI shows.
Also: selinux test: skip broken test (#1465).
Also: version test: skip parts of it if running w/o sources.
Tests are now passing as root on RHEL8; rootless has numerous
failures which I don't believe are related to this PR.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Closes: #1472
Approved by: TomSweeneyRedHat
When we're testing with rootless isolation, skip tests that check that
we were able to influence namespace and control group settings, since we
hardwire namespace settings and don't try to configure control groups.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #873
Approved by: rhatdan
Add an IsolationChroot that trades flexibility and isolation for being
able to do what it does in a host environment that's already isolated to
the point where we're not allowed to set up some of that isolation,
producing a result that leans more toward chroot(1) than runc(1) does.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #836
Approved by: rhatdan
When testing volumes as part of checking that "run" is working
correctly, use the ":z" flag if we detect SELinux support to avoid false
negatives.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #876
Approved by: nalind
