Richard Cooper has uploaded this change for review. ( https://gem5-review.googlesource.com/c/public/gem5/+/70237?usp=email ) Change subject: scons: Fix gem5 Python3.11 build. ...................................................................... scons: Fix gem5 Python3.11 build. The code generation in gem5's build system requires the use of Regular Expression flags when defining the regular expressions used for tokenization. However, the Python Lex-Yacc (PLY) [1] library used by gem5 does not allow the user sufficient control of the flags for RE compilation. Previously, gem5 used inline RE flags to control RE compilation. However, from Python 3.11, inline RE flags must be at the start of the RE string. Because PLY wraps the user supplied RE strings before compilation, there is no way for the user to supply a RE string with the inline flag at the start. This makes gem5 incompatible with Python 3.11 when using PLY. This change modifies gem5's build files to patch `re.compile` with a wrapped version that can handle embedded flags anywhere in the RE string, for all current versions of Python. The patched version re-formats the user supplied RE string to convert inline RE flags to explicit RE flags. This patch is intended as a temporary stop-gap until PLY can be fixed upstream. See the gem5 Issue Tracker [2] for more details. [1] https://github.com/dabeaz/ply [2] https://gem5.atlassian.net/browse/GEM5-1321 Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef --- M SConstruct M site_scons/gem5_scons/__init__.py 2 files changed, 47 insertions(+), 0 deletions(-) diff --git a/SConstruct b/SConstruct index b784a04..b1cb05b 100755 --- a/SConstruct +++ b/SConstruct @@ -171,6 +171,10 @@ Export('MakeAction') +# Patch re.compile to support inline flags anywhere within a RE +# string. Required to use PLY with Python 3.11+. +gem5_scons.patch_re_compile_for_inline_flags() + ######################################################################## # # Set up the main build environment. diff --git a/site_scons/gem5_scons/__init__.py b/site_scons/gem5_scons/__init__.py index 7214876..65aae5d 100644 --- a/site_scons/gem5_scons/__init__.py +++ b/site_scons/gem5_scons/__init__.py @@ -302,6 +302,48 @@ return pickle.loads(node.read()) +def patch_re_compile_for_inline_flags(): + """Patch `re.compile` with a version that can handle RE strings with + inline flags anywhere within them. This is required to use PLY + with Python 3.11+. + + """ + + import re + from functools import partial + + def _inline_flag_aware_re_compile(re_compile, re_str, flags=0x0): + """Provide an alternative implementation of `re.compile` that allows + inline flags that are not at the start of the regular + expression string. + + From Python 3.11, the `re` module only supports inline flags + at the start of the RE string. This makes it impossible to add + flags to the Lexer strings when using PLY, because PLY embeds + the user supplied token REs, and does not provide sufficient + control of the `flags` argument. + + """ + _flags_map = { + ("(?a)", b"(?a)"): re.ASCII, + ("(?i)", b"(?i)"): re.IGNORECASE, + ("(?L)", b"(?L)"): re.LOCALE, + ("(?m)", b"(?m)"): re.MULTILINE, + ("(?s)", b"(?s)"): re.DOTALL, + ("(?x)", b"(?x)"): re.VERBOSE, + } + for (pattern_s, pattern_b), flag in _flags_map.items(): + pattern = pattern_b if isinstance(re_str, bytes) else pattern_s + replacement = b"" if isinstance(re_str, bytes) else "" + if pattern in re_str: + flags |= flag + re_str = re_str.replace(pattern, replacement) + return re_compile(re_str, flags) + + # Patch the default `re.compile` + re.compile = partial(_inline_flag_aware_re_compile, re.compile) + + __all__ = [ "Configure", "EnvDefaults", @@ -312,4 +354,5 @@ "MakeActionTool", "ToValue", "FromValue", + "patch_re_compile_for_inline_flags", ] -- To view, visit https://gem5-review.googlesource.com/c/public/gem5/+/70237?usp=email To unsubscribe, or for help writing mail filters, visit https://gem5-review.googlesource.com/settings Gerrit-MessageType: newchange Gerrit-Project: public/gem5 Gerrit-Branch: develop Gerrit-Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef Gerrit-Change-Number: 70237 Gerrit-PatchSet: 1 Gerrit-Owner: Richard Cooper <richard.cooper@arm.com>