regex: avoid glibc deadlock during configure

glibc has a known bug where certain corruptions of the heap can
cause malloc to default to printing a debug message that includes
a backtrace, but the act of getting the backtrace uses dlopen
which in turn calls into malloc, causing a recursive lock ending
in deadlock.  Thus, when configure is probing for a known glibc
heap corruption bug, the overall configure would hang.

The solution suggested by glibc developers is to force malloc
to quit printing debug messages, which avoids recursive malloc.

* m4/regex.m4 (gl_REGEX): Avoid recursive malloc deadlock when
glibc bug 15078 in turn triggers bug 16159.
Reported by Michal Privoznik.

Signed-off-by: Eric Blake <eblake@redhat.com>

diff --git a/ChangeLog b/ChangeLog
index 2bc45fc..4e7893f 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2013-12-03  Eric Blake  <eblake@redhat.com>
+
+       regex: avoid glibc deadlock during configure
+       * m4/regex.m4 (gl_REGEX): Avoid recursive malloc deadlock when
+       glibc bug 15078 in turn triggers bug 16159.
+       Reported by Michal Privoznik.
+
 2013-12-02  Pádraig Brady <P@draigBrady.com>
 
        md5, sha1, sha256, sha512: use openssl routines if available.

diff --git a/m4/regex.m4 b/m4/regex.m4
index 424ae33..0089c08 100644 (file)
--- a/m4/regex.m4
+++ b/m4/regex.m4
@@ -1,4 +1,4 @@
-# serial 64
+# serial 65
 
 # Copyright (C) 1996-2001, 2003-2013 Free Software Foundation, Inc.
 #
@@ -28,6 +28,7 @@ AC_DEFUN([gl_REGEX],
     # If cross compiling, assume the test would fail and use the included
     # regex.c.
     AC_CHECK_DECLS_ONCE([alarm])
+    AC_CHECK_HEADERS_ONCE([malloc.h])
     AC_CACHE_CHECK([for working re_compile_pattern],
                    [gl_cv_func_re_compile_pattern_working],
       [AC_RUN_IFELSE(
@@ -41,6 +42,9 @@ AC_DEFUN([gl_REGEX],
             # include <unistd.h>
             # include <signal.h>
             #endif
+            #if HAVE_MALLOC_H
+            # include <malloc.h>
+            #endif
           ]],
           [[int result = 0;
             static struct re_pattern_buffer regex;
@@ -49,11 +53,17 @@ AC_DEFUN([gl_REGEX],
             const char *s;
             struct re_registers regs;
 
+            /* Some builds of glibc go into an infinite loop on this
+               test.  Use alarm to force death, and mallopt to avoid
+               malloc recursion in diagnosing the corrupted heap. */
 #if HAVE_DECL_ALARM
-            /* Some builds of glibc go into an infinite loop on this test.  */
             signal (SIGALRM, SIG_DFL);
             alarm (2);
 #endif
+#ifdef M_CHECK_ACTION
+            mallopt(M_CHECK_ACTION, 2);
+#endif
+
             if (setlocale (LC_ALL, "en_US.UTF-8"))
               {
                 {