Tag Archive for 'phpt'

PHP TestFest 2008

PHP.net has posted a nice overview of the PHP TestFest 2008 initiative, which aims to increase the PHP source code coverage through PHPT tests. Worldwide, the TestFest was carried out with different user groups, and of course the PHPBelgium and phpGG were also present. We arranged a meetup in Roosendaal earlier this year and with a group of 10 people, we managed to write about 40 tests! All efforts combined, there was about 10 % increase in code coverage for PHP, which was great to hear!

Actually, I found it so much fun and rewarding to help PHP as a whole in this new way, I decided to continue contributing tests to the submission system. You can imagine I was quite excited when the PHP-QA team actually asked me if I was interested in commit access to the PHP source tree! It’s a great way to give back to the community, and apparently I got mentioned for this in the php.net article, which of course made me somewhat proud ;-)

It was very cool to see the collaborative work around the world being rewarded with our favourite pets, and no less than 4 people from Belgium and the Netherlands (Michelangelo van Dam, Stefaan Koopmanschap, Rein Velt, Marc Veldman) were given an elePHPhant.

Actually I think initiatives like the PHP TestFest make this thriving and exciting community what it is right now, and it’s because of that I very much like the Emphpower initiative from Lucas Smith. To quote the mission statement: “emPHPower is a mediator and catalyst that empowers members of the community to follow their own ideas”. For example, things like the PHP TestFest could fall under the organisation of Emphpower.

I’m pretty much sure I’ll organise the next year PHP TestFest for Belgium (and the Netherlands), under the PHPBelgium / phpGG umbrella of course ;-) Testing is one of many great ways to engage in the PHP community, and the only thing you need to know is how to write PHP code. After all, most of you who are reading this probably know how :-) More information can be found on the PHP-QA website.

PHP TestFest 2008 on PHP.net” screenshot on flickr.

Generating PHP test coverage and ccache

Today I had a weird error while generating PHP code coverage for PHPT tests. If you’re writing PHP extensions, it’s quite common to use compiler caches like ccache. They cache compile results in a way that speeds up multiple compilation of the same source, something which happens quite a lot if you’re working on coding some C extension. So I have cc, gcc and g++ dynamically linked to the ccache binary, which does the caching after proxying the compile request to the actual compiler. This works great, but when generating GCC code coverage (gcov), things don’t work as expected. When running make lcov, the following happened at the end:

Generating php_lcov.info
Capturing coverage data from lcov_data/
Found gcov version: 4.1.2
Scanning lcov_data/ for .gcda files ...
ERROR: no .gcda files found in lcov_data/!
make: *** [php_lcov.info] Error 255

The .gcda files are the ones which contain the useful coverage data used by lcov, but they just weren’t there! It took a while to figure out that disabling ccache fixed the problem. Actually the problem itself isn’t fixed, so if anyone has any idea why ccache and gcov data don’t like eachother, let me know in the comments! But for now, at least there’s a way around it, just disable ccache :-)

Testing Databases, streams and files with PHPT

Writing PHPT tests is pretty easy to learn. Basically, you execute php code, and define what output is expected from the code. That’s fine for testing string manipulating functions or some other basic function, but what do you do if databases or streams need to be tested? What if you need files or directories to execute code upon?

For databases, things can either be a bit difficult or not. Testing MySQL functionality is still a bit difficult, currently this is done mostly by fetching environment variables. You can set variables for MYSQL_TEST_HOST, MYSQL_TEST_PORT, MYSQL_TEST_USER, MYSQL_TEST_PASSWD, MYSQL_TEST_ENGINE, MYSQL_TEST_SOCKET to define connection parameters for your MySQL tests (for PDO_MYSQL they’re named a bit different, but you get the idea). An example of getting the hostname is:

$host = getenv("MYSQL_TEST_HOST") ? getenv("MYSQL_TEST_HOST") : "localhost";

Testing sqlite on the other hand is a whole lot easier. If you know that sqlite supports in-memory databases, you can pretty much get everything tested without a database dependency or files on the filesystem. Just open the database handle like this:

// procedural
$db = sqlite_open(':memory:');
// or OO-style
$db = new SQLiteDatabase(':memory:');

PHP streams are a similar subject when it comes to PHPT testing. If you want to test functions that interact with a stream handler, just use the php://memory built-in stream, like this:

$fp = fopen("php://memory", "r+");
// Now just use it as a regular PHP stream
fwrite($fp, "foobar");

If you do need to create files or directories for testing (for example for testing the SPL DirectoryIterator classes), best is to create them in the directory where the tests are located. Give them a descriptive name, and be sure to clean up after you’re done! This is done with a --CLEAN-- section in a PHPT test. Remember that the --FILE-- and --CLEAN-- parts of a PHPT test are called totally separate, so any variables that live in the --FILE-- part are lost in the --CLEAN-- part. This is an example to make it all a bit more clear:

Test is_file() function: basic functionality
$file_path = dirname(__FILE__);
$file_name = $file_path . DIRECTORY_SEPARATOR . "is_file_basic.tmp";
$handle = fopen($file_name, "w");
var_dump( is_file($file_name) );
// Rebuild file path, because we can't use the $file_name variable anymore.
$file_path = dirname(__FILE__);
$file_name = $file_path . DIRECTORY_SEPARATOR . "is_file_basic.tmp";

PHPT (testfest) writing resources

The PHP TestFest month is almost over and we can already probably say it was quite a success. Of course it would be great if some of the test writers continue to occasionally write a new test or two, and for this occasion, I thought it would be handy to have a resource listing of presentations, websites and pdf’s that cover PHPT testing. Although I believe this is a decent list of resources I’ve actually used myself, I’m probably missing some, so this will be a work in progress and updated once in a while. :-)

First, some of the official PHP resources on PHPT testing (pretty much all of qa.php.net is relevant here):

GCC code coverage page for PHP, which shows test code coverage for the most popular CVS branches. Some graphs and other compiling, testing and analysing information is also available:

This is the place you’ll need to go to acually submit tests if you don’t have CVS commit karma yet (it’s good for overall testing statistics too):

Next is a blog post on what Josie Messa has learned while writing tests. Interesting article which shows a lot of real code examples and issues that she dealt with while writing PHPT tests:

Slides by Sebastian Bergmann, useful to prepare yourself for writing tests. This presentation tells about what PHPT testing is all about, how to properly write tests, and how you should prepare for testfest:

This is an article I wrote, explaining a way to set up Mac OS X for writing PHPT tests. This doesn’t use the built in PHP distribution but instead prepares the OS for manually compiling and running PHP from CVS, with code coverage and all the goodies:

Presentation by Marcus Börger with pretty extensive overview on testing (and PHPT testing in specific). This is one of the most complete presentations on the subject I could find:

The blog of Zoe Slattery, which has 2 posts (and much more to come, undoubtly!) about testing PHP and PHP TestFest:

This is the official website of the PHPT testing framework:

Of course you’re best off grabbing a CVS checkout of the PHP_5_3 (or 6_dev) source tree:

In the CVS checkout of the PHP source code, there is a interesting README file called README.TESTING. It provides useful information on how to test, testing conventions, how to handle failed tests and more:

If you have questions regarding writing PHPT tests, there are several ways to go. First off, you can ask away at the IRC channel #phptestfest on irc.freenode.net. For any kind of QA or testing related question, you can also mail php-qa@lists.php.net mailing list, and subscribe to the mailing list on http://www.php.net/mailing-lists.php.

Standard core PHP 5.3 extensions

When writing PHPT tests for PHP, it’s good practice to have a conditional --SKIPIF-- block to see if the test can be skipped when some extension you’re testing isn’t available. With the PHP_5_3 CVS branch (the one you should be testing against), some new extensions can’t be disabled, so the --SKIPIF-- block isn’t needed anymore. This applies to the following extensions: PCRE, Reflection and SPL.

In fact, the following extensions can’t be disabled so you can safely test against them without the --SKIPIF-- block:

  • date
  • ereg
  • pcre
  • Reflection
  • SPL
  • standard

For completeness, a short example for a Reflection test:

ReflectionClass::isAbstract() method
// REMOVE THEM! <?php extension_loaded('reflection') or die('skip'); ?>
abstract class TestAbstractClass {}
$abstractClass = new ReflectionClass('TestAbstractClass');

Writing PHPT tests on your mac

This is a short tutorial on how to set up a Mac with Mac OS X to prepare for writing PHPT tests. This preparation includes installing some extra software that enables you to checkout and compile the PHP_5_3 branch of the PHP source without problems. Furthermore we’ll install some tools that make it possible to get an easy overview of untested PHP code with code coverage reports.

This overview assumes you’re working on the latest version of Mac OS X, being 10.5 (Leopard). Some terminal experience also comes in handy!

1. Installation of XCode

Apple’s XCode is used to create a development environment on your mac. For example, the GCC compiler, automake and other build tools are installed with XCode. Notice: you do need XCode version 3 or higher. The files needed for the installation can be found on the Mac OS X installation DVD or on the Apple Developer Connection website (http://developer.apple.com)

2. Installation of MacPorts

MacPorts is a package manager that will ease the installation, configuration and updating of open-source software on your mac. We’ll use MacPorts to install some tools that are needed in our development environment. You can find the MacPorts installation files on http://www.macports.org. By default, the MacPorts binaries are installed in /opt/local/bin. To make working with the terminal easier, we’ll add the directory to our PATH variable. Let’s open Apple’s Terminal.app and type:

# echo 'export PATH=/opt/local/bin:$PATH' >> ~/.profile
# source ~/.profile

Trough MacPorts, we’ll install a lexer used by PHP (re2c) and a tool to download files (wget):

# sudo port install re2c
# sudo port install wget

3. Lcov installation

Next up, we’ll install lcov. Lcov is a graphical front-end for gcov, a code coverate testing tool for the GCC compiler. Lcov isn’t available in the MacPorts repository, so we’ll need to install this manually:

# sudo mkdir -p /usr/local/src; cd /usr/local/src
# sudo wget http://downloads.sourceforge.net/ltp/lcov-1.6.tar.gz
# sudo tar -xzvf lcov-1.6.tar.gz
# cd lcov-1.6

Due to a non-compatible ‘install’ binary on Mac OS X, the installation can’t be completed with the -D switch. This has no implications on the actual resulting installation, but it does need to be changed to suppress the error messages:

# sudo nano /usr/local/src/lcov-1.6/bin/install.sh

Go to line 34 (install -D $SOURCE $TARGET) and remove the -D switch. Afterwards you can install lcov with the following command:

# sudo make install

4. Installing the right flex version

Now we need to install a alternate flex version. By default, Mac OS X has the flex lexical parser installed, but the installed version is not compatible with PHP. Thus we’ll install an older flex on an alternate location, which then can be used for compiling PHP:

# cd /usr/local/src
# sudo wget http://dfn.dl.sourceforge.net/sourceforge/flex/flex-2.5.4a.tar.gz
# sudo tar -xzvf flex-2.5.4a.tar.gz
# cd flex-2.5.4
# ./configure && make
# sudo make install

5. Installing the right autoconf version

If you compile PHP, you’re advised to use autoconf version 2.13. Since Mac OS X comes with autoconf version 2.61 by default, we’ll install an older autoconf version, which again can be used when we’re compiling PHP:

# cd /usr/local/src
# sudo wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.13.tar.gz
# sudo tar -xzvf autoconf-2.13.tar.gz
# cd autoconf-2.13
# ./configure && make
# sudo make install

Let’s change our PATH variable. That way, the right lcov, flex and autoconf binaries will be found by PHP:

# echo 'export PATH=/usr/local/bin:/opt/local/bin:$PATH' >> ~/.profile
# source ~/.profile

6. Downloading the PHP source

We create a directory to put all our sources and other related files:

# mkdir ~/phptest; cd ~/phptest/

Now we can check out the PHP source code using anonymous SVN:

# svn checkout http://svn.php.net/repository/php/php-src/branches/PHP_5_3 php-src-5.3 
# cd ~/phptest/php-src-5.3
# ./buildconf

7. Compiling and testing of PHP

Everything is now set up to write PHPT tests or just explore and hack the PHP source code. If you want to configure and compile PHP, use the following commands:

# ./configure && make && make test

If you also want to enable code coverage results, use the following command:

# ./configure --enable-gcov && make && make lcov

Optional, you can also pass the TESTS variabele to the make lcov command to only run a specific test-case (of course, this speeds up compiling the PHP source):

# make lcov TESTS=ext/foo/tests/bar.phpt

Next up, have a look at the lcov_html/ directory, where you can find the HTML code coverage results. Open your browser to view the code coverage, and write some PHPT tests to increase the overall coverage!

8. Some links to get you started on testing

PHP-QA: http://qa.php.net
PHP TestFest: http://wiki.php.net/qa/testfest
PHPT: http://phpt.info/

Update (09/07/2010): I’ve updated the article to reflect the change from CVS to SVN as version control system of the PHP project.