How to Locate GCC Regressions

A regression is a bug that did not exist in a previous release. Problem reports for GCC regressions have a very high priority, and we make every effort to fix them before the next release. Knowing which change caused a regression is valuable information to the developer who is fixing the problem, even if that patch merely exposed an existing bug.

People who are familiar with building GCC but who don't have the knowledge of GCC internals to fix bugs can help a lot by identifying patches that caused regressions to occur. The same techniques can be used to identify the patch that unknowingly fixed a particular bug on the mainline when that bug also exists as a regression on a release branch, allowing someone to port the fix to the branch.

These instructions assume that you are already familiar with building GCC on your platform.

Search strategies

If you've got sufficient disk space available, keep old install tree around for use in finding small windows in which regressions occur. Some people who do this regularly add information to our bug tracker about particular problem reports for regressions.

Before you start your search, verify that you can reproduce the problem with GCC built from the current sources. If not, the bug might have been fixed, or it might not be relevant for your platform, or the failure might only happen with particular options. Next, verify that you get the expected behavior for the start and end dates of the range.

The basic search strategy is to iterate through the following steps while the range is too large to investigate by hand:

• Get GCC sources for that date.
• Build GCC, or specific components that are needed for testing.
• Run the test.
• Based on the outcome of the test, find the midpoint of the new search range.

The first three steps are described below. They can be automated, as can the framework for the binary search. The directory contrib/reghunt in the GCC repository includes scripts to do this work.

There are several short cuts that can be used to shorten the elapsed time of the search.

Eventually you'll need to identify the patch and verify that it causes the behavior of the test to change.

There are a variety of problems you might encounter, but many of them are simple to work around.

Get GCC sources

Check Out a Working Copy

Check out a working copy using Git. Use a new working copy that is separate from what you use for development or other testing, since it is easy to end up with files in strange states.

Branch and release dates

If no one has provided a range of dates for when a particular mainline regression appeared, you can narrow the search by knowing in which release it first appeared and then testing the mainline between the branchpoint for that release and the branchpoint for the previous release that does not have the bug. To find out the revision/date at which a branch or tag was created, use the command svn log --stop-on-copy.

Build GCC

The kind of bug you are investigating will determine what kind of build is required for testing GCC on a particular date. In almost all cases you can do a simple make rather than make bootstrap, provided that you start with a recent version of gcc as the build compiler. When building a full compiler, enable only the language you'll need to test. If you're testing a bug in a library, you'll only need to build that library, provided you've already got a compatible version of the compiler to test it with. If there are dependencies between components, or if you don't know which component(s) affect the bug, you'll need to update and rebuild everything for the language.

If you're chasing bugs that are known to be in cc1plus you can do the following after a normal configure:

    cd objdir
    make all-build-libiberty || true
    make all-libiberty
    make all-libcpp || true
    make all-intl || true
    make all-libbanshee || true
    make configure-gcc || true
    cd gcc
    make cc1plus

This will build libiberty, libcpp, libbanshee, intl and cc1plus (make configure-gcc is required since December 2002, make all-intl since July 2003, make all-libbanshee from May until September 2004, make all-libcpp since May 2004, and make all-build-libiberty since September 2004). Alternatively, you can do

    cd objdir
    make all-gcc TARGET-cc1plus

This works since October 2004. When you have built cc1plus, you can feed your source code snippet to it:

    cc1plus -quiet testcase.ii

Run the test

Assuming that there is a self-contained test for the bug, write a small script to run it and to report whether it passed or failed. If you're automating your search then the script should tell you whether the next compiler build should use earlier or later GCC sources.

Hints for coming up with a self-contained test is beyond the scope of this document.

Identify the patch

The following commands are particularly useful to help you identify changes from one version of a file to another:

• git log --help
• git diff --help
• git show --help
• git annotate --help

When you've identified the likely patch out of a set of patches between the current low and high dates of the range, test a source tree from just before or just after that patch was added and then add or remove the patch by updating only the affected files. You can do this by identifying the revision of each file when the patch was added and then using svn update -rrev file to get the desired version of each of those files. Build and test to verify that this patch changes the behavior of the test.

Short cuts

• If you've narrowed down the dates sufficiently, it might be faster to give up on the binary search and start doing forward updates by small increments and then incremental builds rather than full builds. Whether this is worthwhile depends on the relative time it takes to update the sources, to do a build from scratch, and to do an incremental build.
• Similarly, you can do incremental builds when going forward a small amount from the previous build, and go back to builds in clean object directories when building from earlier sources. When moving forward and doing incremental builds, use contrib/gcc_update.
• Before building a compiler after updating the sources, check that the new sources are different from the sources for the current ends of the range. If not, make this new date one of the endpoints without doing the build and running the test.

Work around problems

• If one of the test builds fails, try a date or time slightly earlier or later and see if that works. Usually all files in a patch are checked in at the same time, but if there was a gap you might have hit it.
• Sometimes regressions are introduced during a period when bootstraps are broken on the platform, particularly if that platform is not tested regularly. Your best bet here is to find out whether the regression also occurs on a platform where bootstraps were working at that time.
• If a regression occurs at the time of a large merge from a branch, search the branch.
• If a test causes the compiler or the compiled test program to hang, run it from a csh or tcsh script using limit cputime mm:ss so it will fail if it requires more than the amount of time you specified. The same technique can be used to limit other resources, including memory.
• Latent bugs can become apparent due to to small changes in code sizes or data layout. Test failures for these bugs can be intermittent, leading to randomness in a binary search for the patch that introduced the bug. This makes it important to see if the patch resulting from a regression hunt looks as if it's actually related to the bug. For example, if a search on i686-pc-linux-gnu comes up with a change to an unrelated target, you're probably looking for such a bug.