From The Linux Foundation
Assorted comments and experiences with LSB 4.0 testing. Note that comments about possible waivers are to be considered informal; the way to get a waiver granted is to file a problem report through the certification system. For instructions on installing and running the tests, please see these instructions from ISPRAS.
Note on maintaining this page: start by pasting test results from a given system which has an unknown failure. As analysis proceeds, migrate those raw results into an explanation; don't remove the problems. The idea is to have an in informal log of problems people might run into.
If you need more of a scratch page for collecting specific items from this page for items that need resolving, please use TestPilot40/Unresolved. However, please keep a record here in TestPilot40 of final actions taken.
Contents |
System resource requirements
The distribution test suite has grown in size considerably since LSB 3.1. Currently, a full install of the distribution tests under /opt/lsb/test takes about 1.2Gb. Application battery packages will add to this, and multiple runs under dtk-manager will accumulate test data rather quickly. Free space of 3-5Gb is probably a reasonable expectation for the system under test before starting LSB testing.
In addition, at this time, lsb-test-olver-core may require about 1Gb of system memory, as well as 300Mb of free disk space to complete a test run.
Distribution packages that may be required for successful testing
- lsb-test-perl - perl-devel (libperl-dev, provides libperls.so), gcc, termcap, perl-Test-Harness, perl-Test-Simple, perl-ExtUtils-MakeMaker, perl-ExtUtils-Embed (Fedora seems to split these off)
- lsb-test-olver-core - perl-XML-Parser
- lsb-test-desktop - qt4-database-plugin-sqlite-lib or libqt4-sql-sqlite or equivalent (provides libqsqlite.so)
- lsb-test-azov-qt3 - libqt3-mt-sqlite or qt3-sqlite or equivalent (provides libqsqlite.so)
- lsb-test-azov-qt4 - libqt4-sql or equivalent
Locales related issues
One might see a large number of locale related failures, particularly with lsb-test-libstdcpp (also xts5). The tests need a number of locales setup that may not be part of a standard install.
- Ubuntu: locale-gen it_IT en_US de_DE en_HK de_DE@euro es_MX fr_FR en_PH fr_FR@euro (are more needed here?)
- Debian: dpkg-reconfigure locales (select all-locales or it_IT en_US de_DE en_HK de_DE@euro es_MX fr_FR en_PH fr_FR@euro)
- Many rpm based systems: install locales-de, locales-en, locales-es, locales-fr, locales-it
Beta Process Links
Core test
Newer glibc implementations are showing a number of failures in the exec* tests. A typical journal will look something like this:
/tset/POSIX.os/procenv/sysconf/T.sysconf 1 FAIL /tset/POSIX.os/procprim/exec/T.execl 15 FAIL /tset/POSIX.os/procprim/exec/T.execl 16 UNRESOLVED /tset/POSIX.os/procprim/exec/T.execle 15 FAIL /tset/POSIX.os/procprim/exec/T.execle 16 UNRESOLVED /tset/POSIX.os/procprim/exec/T.execlp 15 FAIL /tset/POSIX.os/procprim/exec/T.execlp 16 UNRESOLVED /tset/POSIX.os/procprim/exec/T.execv 15 FAIL /tset/POSIX.os/procprim/exec/T.execv 16 UNRESOLVED /tset/POSIX.os/procprim/exec/T.execve 15 FAIL /tset/POSIX.os/procprim/exec/T.execve 16 UNRESOLVED /tset/POSIX.os/procprim/exec/T.execvp 15 FAIL /tset/POSIX.os/procprim/exec/T.execvp 16 UNRESOLVED
It would appear that the values for _SC_ARG_MAX(glibc) and ARG_MAX(kernel) no longer agree, plus it would seem that the value returned by _SC_ARG_MAX is not actually honored when an application tries to use it. More detail can be found in bug 2288.
XTS5
In the 4.0 beta release, 64bit platforms will tend to show an excessive number of failures, around 75 or so.
This is tracked in bug 2342, which we believe has been fixed in the snapshot builds.
The following failure shows up in a number of distributions:
/tset/Xlib3/clsdsply/Test 6 FAIL
Journal detail doesn't tell a lot:
Assertion XCloseDisplay-6.(A) When the client has grabbed the server, then a call to XCloseDisplay performs a XUngrabServer. METH: Create a connection for client1 using XOpenDisplay. METH: Create a connection for client2 using XOpenDisplay. METH: Create a window using XCreateWindow. METH: Grab the server for the default display using XGrabServer. METH: Create a process using tet_fork. METH: In child process: METH: Generate a ChangeProperty request using XChangeProperty. METH: Wait sufficient time for the ChangeProperty request to be processed. METH: Verify that no PropertyChange event has been generated by client2 using XCheckWindowEvent. METH: Close the client1 connection using XCloseDisplay. METH: In child process: METH: Wait suffient time for the ChangeProperty request to be processed. METH: Verify that a ChangeProperty request was generated by client2. TRACE: --- Running test with visual class TrueColor, depth 16 CHECK: XCloseDisplay-6 1, line 220 REPORT: XCloseDisplay() did not ungrab the server. CHECK: XCloseDisplay-6 1, line 843
Issue is tracked in bug 2405
Printing
- An additional test was added to lsb-test-printing to test for the directory /usr/share/ppd, which may not be present on all distributions (bug 2157).
- A couple of the foomaticrip tests seem to be failing on a number of distributions (FC9 listed, foomatic-3.0.2-67.fc9.i386):
/testfoomaticrip/testfoomaticrip 3 FAIL /testfoomaticrip/testfoomaticrip 11 FAIL
Journal Detail:
/testfoomaticrip/testfoomaticrip 3 FAIL Reference III.8.2 foomatic-rip inserts standard PPD options into the correct sections of a DSC-compliant PostScript file and withing the sections also in the correct order. Executing /usr/bin/foomatic-rip --ppd /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test.ppd -o FilterPath=/opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/ /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test-input-ps.ps Checking: JCLSetup: Option6 Checking: Prolog: Option5 Checking: Setup: PageSize Option1 Option7 Option8 Option2 Option3 Checking: PageSetup: Page 1: Option4, Page 2: Option4, Page 3: Option4, Page 4: Option4 Output written to stdout did not contain "\%\%Page:\s*1\s+1.*\%\%BeginFeature:\s*\*Option4\s+Choice\d.*gsave.*\%\%Page:\s*2\s+2.*\%\%BeginFeature:\s*\*Option4\s+Choice\d.*gsave.*\%\%Page:\s*3\s+3.*\%\%BeginFeature:\s*\*Option4\s+Choice\d.*gsave.*\%\%Page:\s*4\s+4.*\%\%BeginFeature:\s*\*Option4\s+Choice\d.*gsave" URL: http://developer.linux-foundation.org/lsbchk?suite=lsb-test-printing&arch=ia32&testcase=/testfoomaticrip/testfoomaticrip&tpnum=3&result=FAIL /testfoomaticrip/testfoomaticrip 11 FAIL Reference III.8.2 Option settings can be applied to selected pages (page overrides) Executing /usr/bin/foomatic-rip --ppd /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test.ppd -o FilterPath=/opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/ -o 3:Option4=Choice3 /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test-input-ps.ps Checking: PageSetup: Page 1: Option4=Choice1, Page 2: Option4=Choice1, Page 3: Option4=Choice3, Page 4: Option4=Choice1 Output written to stdout did not contain "\%\%Page:\s*1\s+1.*\%\%BeginFeature:\s*\*Option4\s+Choice1.*gsave.*\%\%Page:\s*2\s+2.*\%\%BeginFeature:\s*\*Option4\s+Choice1.*gsave.*\%\%Page:\s*3\s+3.*\%\%BeginFeature:\s*\*Option4\s+Choice3.*gsave.*\%\%Page:\s*4\s+4.*\%\%BeginFeature:\s*\*Option4\s+Choice1.*gsave" Executing /usr/bin/foomatic-rip --ppd /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test.ppd -o FilterPath=/opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/ -o 2-4:FoomaticOption1=Choice3 /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test-input-ps.ps Checking: FoomaticOption1=Choice3 for pages 2-4 Executing /usr/bin/foomatic-rip --ppd /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test.ppd -o FilterPath=/opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/ -o odd:FoomaticOption1=Choice3 /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test-input-ps.ps Checking: FoomaticOption1=Choice3 for odd pages Executing /usr/bin/foomatic-rip --ppd /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test.ppd -o FilterPath=/opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/ -o 1-2,4:FoomaticOption1=Choice3 /opt/lsb/test/printing/tet_tmp_dir/26243aa/testfoomaticrip/foomatic-test-input-ps.ps Checking: FoomaticOption1=Choice3 for pages 1-2 and 4 URL: http://developer.linux-foundation.org/lsbchk?suite=lsb-test-printing&arch=ia32&testcase=/testfoomaticrip/testfoomaticrip&tpnum=11&result=FAIL
Mail list threads (with no real resolution, perl issue?): thread start, continuation
And the bug - local testing seems to point to a regression in foomatic-rip itself. This was confirmed by upstream, the bug being introduced with this commit:
revno: 127 committer: Linux Foundation Patch Queue Manager <pqm@linux-foundation.org> branch nick: foomatic-filters timestamp: Sun 2008-04-20 12:20:20 -0700 message: Fixed problem of enumerated choice options with settings \"True\" and \"False\" not handled correctly. Added &rbinumcopies; placeholder.
Distributions shipping snapshots of 3.x should update to the 20081119 commit:
revno: 137 committer: Linux Foundation Patch Queue Manager <pqm@linux-foundation.org> branch nick: foomatic-filters timestamp: Wed 2008-11-19 03:20:10 -0800 message: PageSetup sections were inserted after the first PostScript code line of the p age and not directly after \"%%Page ...\".
- Distributions using the 4.0 branch of foomatic filter exhibit a different set of FAILS, with one test consuming the CPU until it's killed:
/testfoomaticrip/testfoomaticrip 7 FAIL /testfoomaticrip/testfoomaticrip 8 FAIL /testfoomaticrip/testfoomaticrip 10 FAIL /testfoomaticrip/testfoomaticrip 11 FAIL /testfoomaticrip/testfoomaticrip 13 FAIL
from top:
6824 root 20 0 15884 4284 2092 R 99.2 0.4 0:53.85 foomatic-rip
Tracked in bug 2423
Python
Python 2.6 is beginning to show up in some distributions (OpenSuSE 11.1 beta). Our scheme with a dual 2.4/2.5 setup falls apart with a 2.6 version - bug 2363. Changes were pushed to snapshots to partially address this so the tests at least run, but there are still a number of failures:
95 tests OK. 13 tests failed: test_deque test_enumerate test_eof test_format test_gc test_generators test_genexps test_iterlen test_itertools test_list test_syntax test_unicode test_unicodedata
The remaining 13 tests have now been reworked to run correctly under Python-2.6 in post-beta snapshot builds and should not fail in RCs and the final 4.0 test suite.
Desktop Test
xml
We're seeing xml failures against more modern libxml (bug 2274). We went through this once before when 2.6.27 came about. Rather than continue down the path of parallel test setups, a thought is to disable the non-portable regression tests. Typical results (Fedora 10 Beta, x86_64, libxml2-2.7.1):
XML_10 5 FAIL XML_10 6 FAIL XML_10 8 FAIL XML_10 9 FAIL XML_10 14 FAIL XML_10 58 FAIL XML_10 59 FAIL XML_10 63 FAIL XML_10 70 FAIL XML_10 78 FAIL XML_10 81 FAIL XML_10 82 FAIL XML_10 98 FAIL XML_10 100 FAIL XML_30 4 FAIL XML_31 4 FAIL XML_4 8 FAIL XML_5 8 FAIL Test was run: 20081021 16:34:06 Test Suite Version: unset Test Suite Architecture: unset Total Tests Passed: 2449 Total Tests Failed (including waived): 18 Total Tests Failed (excluding waived): 18
There is also an journal issue exposed in these failures (bug 2347), which we believe has been fixed in the snapshot builds.
A modification for bug 2274 to disable the non-portable regression tests has been pushed to both the 3.2 branch and development and should be reflected in snapshot builds and RCs, as well as the next 3.2.X refresh.
cairo
Some initial discussion/results for the new Cairo tests is here: CairoTesting40. I (Stew) decided to treat the image variations as warnings for LSB testing.
There are 2 tests that fail almost everywhere:
path-data-image 0 UNREPORTED
Error: CAIRO_STATUS_NULL_POINTER (cairo_append_path(cr, &path) \
path.data = NULL, path.num_data = 0, path.status = CAIRO_STATUS_SUCCESS)
path-data-xlib 1 UNREPORTED
Error: CAIRO_STATUS_NULL_POINTER (cairo_append_path(cr, &path) \
path.data = NULL, path.num_data = 0, path.status = CAIRO_STATUS_SUCCESS)
These tests do not appear in newer cairo versions and the thought is to disable them (bug 2353).
Two additional failures are showing up in the newest distributions (OpenSuSE 11.1 (cairo-1.7.4), Fedora 10 (cairo-1.7.4)):
surface-finish-twice-image 0 FAIL Error: Function under test failed surface-finish-twice-xlib 1 FAIL Error: Function under test failed
I'm not finding 1.7.4 source, but a diff of the 1.0.2 test code with 1.6.4 only shows test API changes. A diff with 1.8.0 does show:
- if (cairo_surface_status (surface) != CAIRO_STATUS_SURFACE_FINISHED)
+ if (cairo_surface_status (surface) != CAIRO_STATUS_SUCCESS)
+ return CAIRO_TEST_FAILURE;
+
+ cairo_surface_finish (surface);
+ if (cairo_surface_status (surface) != CAIRO_STATUS_SUCCESS)
return CAIRO_TEST_FAILURE;
Checking for only CAIRO_STATUS_SUCCESS fails on older versions of cairo. It looks like this is simply a regression test by the comments, so perhaps it should be disabled too:
* Subject: [Bug 2950] New: *** glibc detected *** double free or corruption * URL: https://bugs.freedesktop.org/show_bug.cgi?id=2950 * * The following short program gives the error message: * * *** glibc detected *** double free or corruption: 0x082a7268 *** * Aborted * * 2005-04-13 Carl Worth <cworth@cworth.org> * * Looks like surface->finished was never being set. Now fixed.
The 4 tests mentioned above have been disabled in post-beta snapshot builds and should not show up the the RCs or final test suites.
gtk
Seeing a number of GtkTreeSelection failures on some distributions. This appears to be from a patch added to upstream and is tracked in bug 2170:
/tests/functions/GtkTreeSelection/GtkTreeSelection 45 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 46 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 47 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 48 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 49 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 50 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 51 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 52 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 53 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 54 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 55 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 56 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 57 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 58 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 59 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 60 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 61 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 62 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 63 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 64 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 65 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 66 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 67 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 68 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 69 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 170 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 171 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 172 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 173 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 174 FAIL /tests/functions/GtkTreeSelection/GtkTreeSelection 175 FAIL
Seeing a number of failures with GtkAdjustment, GtkSpinButton, MainLoop on distribuitions shipping gtk-2.14.0. This is tracked in bug 2380 and would appear to be a regression in 2.14.x with x < 3, as 2.14.3 system do not show the failures. Typical results (F10 beta):
/tests/functions/GtkAdjustment/GtkAdjustment 61 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 62 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 63 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 69 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 72 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 73 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 74 FAIL /tests/functions/GtkAdjustment/GtkAdjustment 75 FAIL /tests/functions/GtkSpinButton/GtkSpinButton 48 FAIL /tests/functions/MainLoop/MainLoop 6 FAIL /tests/functions/MainLoop/MainLoop 7 FAIL
Seeing tests return UNRESOLVED with SIGSEGV on 2.14.0+ based systems. This is tracked in bug 2379. Root cause has not yet been determined. A modification has been applied to the tests to let the tests run and a bug has been filed with upstream, but it's not yet clear whether it's a test or gtk issue. Results are not consistent as to which subtest may fail, but a typical run (Mandriva 2009.0) might look like this:
/tests/functions/GtkFileSelection/GtkFileSelection 2 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 5 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 8 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 9 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 10 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 12 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 13 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 15 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 16 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 17 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 18 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 19 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 20 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 21 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 22 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 23 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 24 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 25 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 26 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 27 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 28 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 29 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 30 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 31 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 32 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 33 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 34 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 35 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 36 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 37 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 38 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 39 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 40 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 41 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 42 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 43 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 44 UNRESOLVED /tests/functions/GtkFileSelection/GtkFileSelection 45 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 4 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 8 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 9 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 10 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 11 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 12 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 13 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 14 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 15 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 16 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 17 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 18 UNRESOLVED /tests/functions/GtkFontSelectionDialog/GtkFontSelectionDialog 19 UNRESOLVED