regex: re_search etc. should return -2 when memory exhausted

This bug was uncovered when testing 'grep'.  Without the fix,
re_search and friends return -1 when memory is exhausted, but -1
means no match, and this causes grep to falsely report no-match
instead of memory-exhaustion.  See
<http://sources.redhat.com/bugzilla/show_bug.cgi?id=13762>.
* lib/regexec.c (re_search_stub): Return -2 (not -1) if there is
trouble; this can occur if re_search_internal ran out of memory.

diff --git a/ChangeLog b/ChangeLog
index ed9c384..433609a 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,14 @@
+2012-02-26  Paul Eggert  <eggert@cs.ucla.edu>
+
+       regex: re_search etc. should return -2 when memory exhausted
+       This bug was uncovered when testing 'grep'.  Without the fix,
+       re_search and friends return -1 when memory is exhausted, but -1
+       means no match, and this causes grep to falsely report no-match
+       instead of memory-exhaustion.  See
+       <http://sources.redhat.com/bugzilla/show_bug.cgi?id=13762>.
+       * lib/regexec.c (re_search_stub): Return -2 (not -1) if there is
+       trouble; this can occur if re_search_internal ran out of memory.
+
 2012-02-26  Bruno Haible  <bruno@clisp.org>
 
        modfl-ieee: Work around test failures on IRIX, OSF/1, mingw.

diff --git a/lib/regexec.c b/lib/regexec.c
index 660d25b..5cbfa91 100644 (file)
--- a/lib/regexec.c
+++ b/lib/regexec.c
@@ -475,7 +475,7 @@ re_search_stub (struct re_pattern_buffer *bufp,
 
   /* I hope we needn't fill ther regs with -1's when no match was found.  */
   if (result != REG_NOERROR)
-    rval = -1;
+    rval = result == REG_NOMATCH ? -1 : -2;
   else if (regs != NULL)
     {
       /* If caller wants register contents data back, copy them.  */