Disabling executable stack on linux

[Niels Möller]

As far as I know, GMP does not use any code that relies on the stack
being executable. It would be nice if it also told that to the linker.

https://www.redhat.com/archives/fedora-devel-list/2003-November/msg00838.html
explains what to put into the assembler files, and how it is
propagated to the executable.

In Nettle, I just added the following configure test,

  ASM_MARK_NOEXEC_STACK=''

  AC_CACHE_CHECK([if we should use a .note.GNU-stack section],
  nettle_cv_asm_gnu_stack,
  [ # Default
    nettle_cv_asm_gnu_stack=no

    cat >conftest.c <<EOF
int foo() { return 0; }
EOF
    nettle_compile="$CC $CFLAGS $CPPFLAGS -c conftest.c >conftest.out 2>&1"
    if AC_TRY_EVAL(nettle_compile); then
      cat conftest.out >&AC_FD_CC
      objdump -x conftest.o | grep '\.note\.GNU-stack' > /dev/null \
      && nettle_cv_asm_gnu_stack=yes
    else
      cat conftest.out >&AC_FD_CC
      echo "configure: failed program was:" >&AC_FD_CC
      cat conftest.s >&AC_FD_CC
    fi
    rm -f conftest.*])
  if test x$nettle_cv_asm_gnu_stack = xyes ; then
    ASM_MARK_NOEXEC_STACK='.section .note.GNU-stack,"", at progbits'
  fi

  AC_SUBST(ASM_MARK_NOEXEC_STACK)

and then in my config.m4.in I put

  divert(1)
  @ASM_MARK_NOEXEC_STACK@
  divert

Seems to work. Perhaps GMP could do something similar?

To check if an object is marked as using a non-executable stack, use
objdump -x foo.o and check of the section .note.GNU-stack is
executable. To check an executable or a shared library, use objdump -p
foo.so, and look at the STACK entry in the program header.

And if anybody has advice on how to handle other systems than
linux/glibc, I'd like to hear it. There are other systems that let you
mark the stack as not executable, right?

At last, improvements to the configure test above is of course also
appreciated.

Regards,
/Niels