Avoid undefined behavior in extract_integer

-fsanitize=undefined showed that extract_integer could left-shift a
negative value, which is undefined.  This patch fixes the problem by
doing all the work in an unsigned type.  This relies on
implementation-defined behavior, but I tend to think we are on safe
ground there.  (Also, if need be, violations of this could probably be
detected, either by configure or by a static_assert.)

gdb/ChangeLog
2018-10-03  Tom Tromey  <tom@tromey.com>

	* findvar.c (extract_integer): Do work in an unsigned type.

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 36c44932743cbea75401801e9e244528abc9b4c7..5787d44424facdd6d1b25a8c6bd831385fd52256 100644 (file)
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,7 @@
+2018-10-03  Tom Tromey  <tom@tromey.com>
+
+       * findvar.c (extract_integer): Do work in an unsigned type.
+
 2018-10-03  Tom Tromey  <tom@tromey.com>
 
        * common/enum-flags.h (enum_flags::operator~): Add static assert.

diff --git a/gdb/findvar.c b/gdb/findvar.c
index 9256833ab601976b2f36f03aa241150f599ec10b..be6c9d6f60b89f36d1760634f2902b19b4d1509e 100644 (file)
--- a/gdb/findvar.c
+++ b/gdb/findvar.c
@@ -50,7 +50,7 @@ template<typename T, typename>
 T
 extract_integer (const gdb_byte *addr, int len, enum bfd_endian byte_order)
 {
-  T retval = 0;
+  typename std::make_unsigned<T>::type retval = 0;
   const unsigned char *p;
   const unsigned char *startaddr = addr;
   const unsigned char *endaddr = startaddr + len;