stack_env.py: 'include:' compiler- and lmod/tcl-specific files instead of convolving yaml's

[AlexanderRichert-NOAA]

Description

This PR proposes to refactor the way we include yaml configs with spack stack create env. Currently, the yaml configs get convolved into one packages.yaml for common/ and site/ (and similarly for modules.yaml). This PR modifies spack_env.py so that the files specific to compilers or to lmod vs. tcl module systems are pointed to explicitly in the newly created spack.yaml. For example:
instead of having the following where the common packages_oneapi.yaml and site-specific packages_oneapi-2024.2.1.yaml are convolved into common/packages.yaml and site/packages.yaml, respectively:

  include:
  - site
  - common

spack stack create env would yield:

  include:
  - site/packages_oneapi-2024.2.1.yaml
  - site
  - common/packages_oneapi.yaml
  - common/modules_lmod.yaml
  - common

Note that putting the file before their parent directories is done to retain the current precedence, where the compiler/module sys-specific customizations override the general ones.

Advantages of this approach:

• It makes it much easier to propagate changes back to $SPACK_STACK_DIR/config because the config files are no longer modified.
• spack config blame makes it very easy to know where a problematic setting came from without having to do a second layer of investigation into which exact file it was pulled from, what the precedence was, etc.
• It (arguably) keeps the configurations in a clearer structure/order, especially when it comes to comments, again because the individual files don't get modified. If a user is unclear on what the net result is of, say, having an entry for a given package in each of 3 or 4 config files, the net result is easily determined using spack config get or spack config blame

Dependencies

none

Issues addressed

none

Applications affected

n/a

Systems affected

all

Testing

• CI: Note whether the automatic tests (GitHub actions tests that run automatically for every commit) pass or not
  • GitHub actions CI tests pass
• Additional testing: Add information on any additional tests conducted
  • Tested config validity on local machine (i.e., ran spack config get/blame)
  • Tested on Acorn (concretization & validation; spack config blame to verify config file usage)

Checklist

• This PR addresses one issue/problem/enhancement or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.
• All necessary updates to the documentation (spack-stack wiki) will be made when this PR is merged

[mathomp4]

Trust in Alex!

[climbfuji]

This is a great improvement, thanks for working on this.