The Linux Foundation is a non-profit consortium dedicated to fostering the growth of Linux.
CairoTesting40
From The Linux Foundation
Contents |
Distribution Testing Status for Cairo
New test planning for libcairo aren't in the schedule until 2009. For the short term, cairo ships with a number of unit tests which can be adapted and patched for TET journal reporting. Although not of the typical LSB assertion -> test style tests, they can at least check for the presence of a number of libcairo interfaces.
Build Time Issues
There is a bug in cairo-1.0.2 that prevents building, I'm using the patch included in freedesktop.org bugzilla to fix this issue.
Patching for TET reporting proved to be fairly easy. The build of the test reporting is in cairo-test.c, although pthread-show-text.c, and xlib-surface.c use a slightly different model than the rest of the tests. user-data is just a bunch of asserts, did not try to tet'ify it. (tetj patch)
We were lacking some png functions to build with lsbcc. These have now been added, with png_jmpbuf to go in soon.
Also a couple of issues with freetype code in the cairo source. Use of deprecated ft_glyph_format_bitmap vs FT_GLYPH_FORMAT_BITMAP, and for some reason fterrors.h needs to be include explicitly to get Ft_Error_OK defined. This is handled with another patch.
Build Time Test Issues
Some tests fail at build time (make check builds and runs the tests against the just built libcairo). To avoid running the xlib tests, I just unset DISPLAY (we're not really interesting in running all the tests at build time).
Even so, some tests still fail on SLES10 (our build host):
FAILS on ia32 at build time (disabled):
text-antialias-none
FAILS on ppc64 at build time:
text-antialias-subpixel-image
FAILS on ia64, s390 at build time:
clip-operator-image operator-clear-image operator-source-image show-text-current-point-image text-antialias-gray-image text-antialias-subpixel-image text-pattern-image unbounded-operator-image
Since we're not really interesting in testing the just-built libcairo, the thing to do here is seperate the build and check functions at build time.
Distribution Testing
Running the resulting tests (packaged as part of desktop test) also show a number of failures, the failures increasing with the respective cairo version installed on the distribution in question. We enable the xlib tests at runtime, which required making sure the render extension is enabled for lsb-xvfb (disabled for the other desktop tests). Fortunately the failures from the lesser versions of cairo are subsets of those from the greater versions, so disabling the superset could allow for clean runs on a variety of systems.
Failed test Images (all)
SLES10 (ia32, cairo 1.0.2)
clip-operator-xlib 1 FAIL unbounded-operator-xlib 1 FAIL Test was run: 20080808 14:16:21 Test Suite Version: unset Test Suite Architecture: unset Total Tests Passed: 112 Total Tests Failed (including waived): 2 Total Tests Failed (excluding waived): 2
Info from test log:
Error: 30897 pixels differ from reference image ./clip-operator-ref.png Error: 11890 pixels differ from reference image ./unbounded-operator-ref.png
Failed test Images
RHEL5 (ia32, cairo-1.2.4)
clip-operator-image 0 FAIL Error: 112333 pixels differ from reference image ./clip-operator-ref.png clip-operator-xlib 1 FAIL Error: 127843 pixels differ from reference image ./clip-operator-ref.png dash-caps-joins-image 0 FAIL Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png dash-caps-joins-xlib 1 FAIL Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png ft-font-create-for-ft-face-image 0 FAIL Error: Function under test failed ft-font-create-for-ft-face-xlib 1 FAIL Error: Function under test failed gradient-alpha-image 0 FAIL Error: 100 pixels differ from reference image ./gradient-alpha-ref.png gradient-alpha-xlib 1 FAIL Error: 100 pixels differ from reference image ./gradient-alpha-ref.png linear-gradient-image 0 FAIL Error: 38287 pixels differ from reference image ./linear-gradient-ref.png linear-gradient-xlib 1 FAIL Error: 38287 pixels differ from reference image ./linear-gradient-ref.png mask-image 0 FAIL Error: 50476 pixels differ from reference image ./mask-ref.png mask-xlib 1 FAIL Error: 50476 pixels differ from reference image ./mask-ref.png operator-clear-image 0 FAIL Error: 10358 pixels differ from reference image ./operator-clear-ref.png operator-clear-xlib 1 FAIL Error: 10358 pixels differ from reference image ./operator-clear-ref.png operator-source-image 0 FAIL Error: 27117 pixels differ from reference image ./operator-source-ref.png operator-source-xlib 1 FAIL Error: 27117 pixels differ from reference image ./operator-source-ref.png text-antialias-gray-image 0 FAIL Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png text-antialias-gray-xlib 1 FAIL Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png text-antialias-subpixel-image 0 FAIL Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png text-antialias-subpixel-xlib 1 FAIL Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png text-pattern-image 0 FAIL Error: 1017 pixels differ from reference image ./text-pattern-ref.png text-pattern-xlib 1 FAIL Error: 1017 pixels differ from reference image ./text-pattern-ref.png trap-clip-image 0 FAIL Error: 22269 pixels differ from reference image ./trap-clip-ref.png trap-clip-xlib 1 FAIL Error: 22269 pixels differ from reference image ./trap-clip-ref.png unbounded-operator-image 0 FAIL Error: 19441 pixels differ from reference image ./unbounded-operator-ref.png unbounded-operator-xlib 1 FAIL Error: 28203 pixels differ from reference image ./unbounded-operator-ref.png Test was run: 20080808 14:32:20 Test Suite Version: unset Test Suite Architecture: unset Total Tests Passed: 88 Total Tests Failed (including waived): 26 Total Tests Failed (excluding waived): 26
Failed test Images
Debian4 (ia32, cairo-1.2.4)
same 26 as RHEL5
OpenSuSE-10.3 (cairo-1.4.10)
clip-operator-image 0 FAIL Error: 58949 pixels differ from reference image ./clip-operator-ref.png clip-operator-xlib 1 FAIL Error: 80770 pixels differ from reference image ./clip-operator-ref.png dash-caps-joins-image 0 FAIL Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png dash-caps-joins-xlib 1 FAIL Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png ft-font-create-for-ft-face-image 0 FAIL Error: Function under test failed ft-font-create-for-ft-face-xlib 1 FAIL Error: Function under test failed gradient-alpha-image 0 FAIL Error: 100 pixels differ from reference image ./gradient-alpha-ref.png gradient-alpha-xlib 1 FAIL Error: 100 pixels differ from reference image ./gradient-alpha-ref.png linear-gradient-image 0 FAIL Error: 55738 pixels differ from reference image ./linear-gradient-ref.png linear-gradient-xlib 1 FAIL Error: 55738 pixels differ from reference image ./linear-gradient-ref.png mask-image 0 FAIL Error: 90916 pixels differ from reference image ./mask-ref.png mask-xlib 1 FAIL Error: 90916 pixels differ from reference image ./mask-ref.png operator-clear-image 0 FAIL Error: 5894 pixels differ from reference image ./operator-clear-ref.png operator-clear-xlib 1 FAIL Error: 5894 pixels differ from reference image ./operator-clear-ref.png operator-source-image 0 FAIL Error: 25366 pixels differ from reference image ./operator-source-ref.png operator-source-xlib 1 FAIL Error: 25366 pixels differ from reference image ./operator-source-ref.png paint-with-alpha-image 0 FAIL Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png paint-with-alpha-xlib 1 FAIL Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png set-source-image 0 FAIL Error: 10 pixels differ from reference image ./set-source-ref.png set-source-xlib 1 FAIL Error: 10 pixels differ from reference image ./set-source-ref.png text-antialias-gray-image 0 FAIL Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png text-antialias-gray-xlib 1 FAIL Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png text-antialias-subpixel-image 0 FAIL Error: 361 pixels differ from reference image ./text-antialias-subpixel-ref.png text-antialias-subpixel-xlib 1 FAIL Error: 361 pixels differ from reference image ./text-antialias-subpixel-ref.png text-pattern-image 0 FAIL Error: 1024 pixels differ from reference image ./text-pattern-ref.png text-pattern-xlib 1 FAIL Error: 1024 pixels differ from reference image ./text-pattern-ref.png trap-clip-image 0 FAIL Error: 44592 pixels differ from reference image ./trap-clip-ref.png trap-clip-xlib 1 FAIL Error: 44592 pixels differ from reference image ./trap-clip-ref.png unbounded-operator-image 0 FAIL Error: 11137 pixels differ from reference image ./unbounded-operator-ref.png unbounded-operator-xlib 1 FAIL Error: 21978 pixels differ from reference image ./unbounded-operator-ref.png Test was run: 20080814 16:39:54 Test Suite Version: unset Test Suite Architecture: unset Total Tests Passed: 84 Total Tests Failed (including waived): 30 Total Tests Failed (excluding waived): 30
Failed test Images
FC8 (ia32, cairo-1.4.14)
same 30 FAILS as OpenSuSE, plus "Journal indicates expected 114 tests, found 112"
FC9 (ia32, cairo-1.6.4)
caps-joins-image 0 FAIL Error: 285 pixels differ from reference image ./caps-joins-ref.png caps-joins-xlib 1 FAIL Error: 285 pixels differ from reference image ./caps-joins-ref.png caps-sub-paths-image 0 FAIL Error: 12 pixels differ from reference image ./caps-sub-paths-ref.png caps-sub-paths-xlib 1 FAIL Error: 12 pixels differ from reference image ./caps-sub-paths-ref.png clip-operator-image 0 FAIL Error: 59315 pixels differ from reference image ./clip-operator-ref.png clip-operator-xlib 1 FAIL Error: 81047 pixels differ from reference image ./clip-operator-ref.png clip-twice-image 0 FAIL Error: 96 pixels differ from reference image ./clip-twice-ref.png clip-twice-xlib 1 FAIL Error: 96 pixels differ from reference image ./clip-twice-ref.png dash-caps-joins-image 0 FAIL Error: 1059 pixels differ from reference image ./dash-caps-joins-ref.png dash-caps-joins-xlib 1 FAIL Error: 1059 pixels differ from reference image ./dash-caps-joins-ref.png fill-and-stroke-image 0 FAIL Error: 38 pixels differ from reference image ./fill-and-stroke-ref.png fill-and-stroke-xlib 1 FAIL Error: 38 pixels differ from reference image ./fill-and-stroke-ref.png ft-font-create-for-ft-face-image 0 FAIL Error: Function under test failed ft-font-create-for-ft-face-xlib 1 FAIL Error: Function under test failed gradient-alpha-image 0 FAIL Error: 100 pixels differ from reference image ./gradient-alpha-ref.png gradient-alpha-xlib 1 FAIL Error: 100 pixels differ from reference image ./gradient-alpha-ref.png leaky-polygon-image 0 FAIL Error: 10 pixels differ from reference image ./leaky-polygon-ref.png leaky-polygon-xlib 1 FAIL Error: 10 pixels differ from reference image ./leaky-polygon-ref.png linear-gradient-image 0 FAIL Error: 56719 pixels differ from reference image ./linear-gradient-ref.png linear-gradient-xlib 1 FAIL Error: 56719 pixels differ from reference image ./linear-gradient-ref.png mask-image 0 FAIL Error: 91308 pixels differ from reference image ./mask-ref.png mask-xlib 1 FAIL Error: 91308 pixels differ from reference image ./mask-ref.png operator-clear-image 0 FAIL Error: 6020 pixels differ from reference image ./operator-clear-ref.png operator-clear-xlib 1 FAIL Error: 6020 pixels differ from reference image ./operator-clear-ref.png operator-source-image 0 FAIL Error: 25741 pixels differ from reference image ./operator-source-ref.png operator-source-xlib 1 FAIL Error: 25741 pixels differ from reference image ./operator-source-ref.png paint-with-alpha-image 0 FAIL Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png paint-with-alpha-xlib 1 FAIL Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png rectangle-rounding-error-image 0 FAIL Error: 53 pixels differ from reference image ./rectangle-rounding-error-ref.png rel-path-image 0 FAIL Error: 3 pixels differ from reference image ./rel-path-ref.png rel-path-xlib 1 FAIL Error: 3 pixels differ from reference image ./rel-path-ref.png set-source-image 0 FAIL Error: 10 pixels differ from reference image ./set-source-ref.png set-source-xlib 1 FAIL Error: 10 pixels differ from reference image ./set-source-ref.png text-antialias-gray-image 0 FAIL Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png text-antialias-gray-xlib 1 FAIL Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png text-antialias-subpixel-image 0 FAIL Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png text-antialias-subpixel-xlib 1 FAIL Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png text-pattern-image 0 FAIL Error: 1024 pixels differ from reference image ./text-pattern-ref.png text-pattern-xlib 1 FAIL Error: 1024 pixels differ from reference image ./text-pattern-ref.png transforms-image 0 FAIL Error: 2 pixels differ from reference image ./transforms-ref.png transforms-xlib 1 FAIL Error: 2 pixels differ from reference image ./transforms-ref.png trap-clip-image 0 FAIL Error: 45107 pixels differ from reference image ./trap-clip-ref.png trap-clip-xlib 1 FAIL Error: 45107 pixels differ from reference image ./trap-clip-ref.png unantialiased-shapes-image 0 FAIL Error: 1427 pixels differ from reference image ./unantialiased-shapes-ref.png unantialiased-shapes-xlib 1 FAIL Error: 19 pixels differ from reference image ./unantialiased-shapes-ref.png unbounded-operator-image 0 FAIL Error: 11376 pixels differ from reference image ./unbounded-operator-ref.png unbounded-operator-xlib 1 FAIL Error: 22129 pixels differ from reference image ./unbounded-operator-ref.png Test was run: 20080808 10:40:49 Test Suite Version: unset Test Suite Architecture: unset Total Tests Passed: 65 Total Tests Failed (including waived): 47 Total Tests Failed (excluding waived): 47 Journal indicates expected 114 tests, found 112
The missing tests look to be:
path-data-image 0 PASS path-data-xlib 1 PASS
Which when run on these platforms exits with status 1 and no log or journal. The test states it is "Tests calls to path_data functions: cairo_copy_path_data, cairo_copy_path_data_flat, and cairo_append_path_data". Looking at the code, it too, varies from the other test code in that it's coded to be able to exit with status 1 before and logging or journal reporting.
Failed test Images
MDV2008.1 (ia32, cairo-1.6.4)
same as FC9
Ubuntu-8.04 (ia32, cairo-1.6.0)
same as FC9
Test Updates
ft-font-create-for-ft-face
Patched in a partial uplift to this test using the testing method used in cairo-1.6.4 - now passes on SLES10, RHEL5, Mdv2008.1, FC9, OpenSuse10.3 (08/19/08)
path-data
Patched the test to not do a "return 1" until the intermediate setup code has all run, capturing the state as "setup_stat". If this variable is "1", then pass UNREPORTED (with the CAIRO error code) for both the image and xlib tests.
Results (FC9):
path-data-image 0 UNREPORTED
Error: CAIRO_STATUS_NULL_POINTER (cairo_append_path(cr, &path) \
path.data = NULL, path.num_data = 0, path.status = CAIRO_STATUS_SUCCESS)
path-data-xlib 1 UNREPORTED
Error: CAIRO_STATUS_NULL_POINTER (cairo_append_path(cr, &path) \
path.data = NULL, path.num_data = 0, path.status = CAIRO_STATUS_SUCCESS)
Note: This test is not present in 1.4.14 or 1.6.4. Candidate to drop?
image differences
Thinking about the tests in an LSB context, perhaps we're not that interested in subtle differences in the reference and generated image, which can be driven by the system's font or X server differences. What we're primarily interested in is whether the interface being tested exists, and that it returns as expected. With this in mind, I've updated (8/22/08) the tetj patch to return TETJ_WARNING for these cases.
This (and the other changes above) reduces the FAIL count considerably:
- SLES10 - 0 FAIL, 2 WARNING, 1 UNTESTED
- RHEL5 - 0 FAIL, 24 WARNING, 1 UNTESTED
- Debian4 - 0 FAIL, 24 WARNING, 1 UNTESTED
- OpenSuSE-10.3 - 0 FAIL, 28 WARNING, 1 UNTESTED
- FC8 - 0 FAIL, 28 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}
- FC9 - 0 FAIL, 45 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}
- Mandriva-2008.1 - 0 FAIL, 45 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}
- Ubuntu-8.04 - 0 FAIL, 45 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}