Fix #4122: Move CALENDAR to env_run.xml to allow keepexe with calenda…

[johnpaulalex]

Description of changes

Moves the CALENDAR variable definition in cime_config/config_component.xml from env_build.xml (group build_def) to env_run.xml (group run_begin_stop_restart).
This resolves an issue where cloning a case with --keepexe and applying usermods that alter the CALENDAR variable (such as during PLUMBER2 or NEON site spinups) failed with:
ERROR: env_build.xml cannot be changed via usermods if keepexe is an option

Specific notes

• The CALENDAR variable is strictly a runtime configuration parameter used by Python setup and namelist-building scripts (buildnml). It is never referenced in preprocessor macros or build system logic.
• Moving CALENDAR to env_run.xml accurately reflects its runtime nature and allows CIME's case_clone.py script to accept calendar modifications under --keepexe without requiring a re-compilation.

Contributors other than yourself, if any: Antigravity using Gemini LLMs came up with the fix, looked for any other CALENDAR references, ran the tests below, and wrote this PR (except this one line :))

CMEPS Issues Fixed (include github issue #): Fixes ESCOMP/CTSM#4122

Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial)
BFB (Bit-for-bit). This change only alters the XML file location (env_run.xml instead of env_build.xml) where the CALENDAR variable is stored during case creation.

Any User Interface Changes (namelist or namelist defaults changes)?
No namelist changes. The XML variable CALENDAR is now stored in env_run.xml instead of env_build.xml.

Testing performed

Please describe the tests along with the target model and machine(s)
If possible, please also added hashes that were used in the testing

1. Created a base case via create_newcase (--compset IHistClm50BgcCrop, --mach ubuntu-latest) and ran ./case.setup.
2. Created a usermod directory containing ./xmlchange CALENDAR=GREGORIAN in shell_commands.
3. Tested create_clone --keepexe --user-mods-dir <usermod_path> against the base case.
4. Result: Before this change, create_clone failed with an env_build.xml validation error. With this change, create_clone completes successfully with --keepexe.

[ekluzek]

This is the correct change. And it would be helpful to have this come in.

I wondered if that was the right group for it to be in, but that is the best one based on the other choices. It does make sense, since it's the calendar that will be used to defie what the beging, stop and restart dates refer to.

Since, this is a simple change the simple testing done is good enough to me. Once, it comes in we should also try it in CTSM for some of those cases that were failing in the PLUMBER2 PR coming in.

[ekluzek]

Pinging @billsacks or @fischer-ncar to approve and make the merge.

[ekluzek]

@johnpaulalex be sure to credit any AI that you used to do this work.

[johnpaulalex]

Thanks Erik! Gemini is now credited :)

[billsacks]

I tend to be nervous about making changes like this without understanding the reason why things were set up as they originally were. I spent some time trying to dig through history for why CALENDAR was put in env_build rather than env_run but gave up. I don't see any problems with this myself.

@mvertens do you know of any reasons why CALENDAR would need to be in env_build? I added you as a reviewer to mark that we should wait to merge until you've had a chance to comment, but if you don't feel a need to look at this, we can merge without your review.

I do see one other minor change needed: In namelist_definition_drv.xml, there is text "set by CALENDAR in env_build.xml", which should be updated to env_run.xml.

[ekluzek]

I tend to be nervous about making changes like this without understanding the reason why things were set up as they originally were. I spent some time trying to dig through history for why CALENDAR was put in env_build rather than env_run but gave up. I don't see any problems with this myself.

Good point. The one thing I wondered about is that if you change it midrun -- maybe it causes problems? And it seems likely it would. I'm not sure it would be easy to disallow changing if CONTINUE_RUN==TRUE either. So that might be why it was added to env_build? Having it in env_build restricts changes more than is needed, but just being in env_run might not restrict it enough?

@mvertens and @billsacks what do you think about that?

[johnpaulalex]

I fixed the comment Bill mentioned.

AI traced the initial placement back through several repos back to 2015:

What the Git & Code Data Actually Shows (Empirical Facts)

1. Undocumented Placement: In the initial 2015 CIME creation commit (bb4ff6191), CALENDAR was placed into <file>env_build.xml</file> under <group>build_def</group> without any commit comment explaining the decision.
2. --keepexe Didn't Exist Yet: The --keepexe option for create_clone was not added until 2016 (commit bccef6ae), and the --user-mods-dir option with --keepexe was added in late 2017 (PR #1866).
3. The Conflict Came Later: Because CALENDAR was placed in env_build.xml in 2015—before --keepexe and usermod cloning were created in 2016–2017—no error was raised at the time because the strict env_build.xml immutability check during cloning didn't exist yet.

---

What is Historical Context / Inference (Domain Knowledge)

• Standard Global Model Setup: In climate modeling (such as CMIP5/CMIP6), standard coupled global simulations traditionally used a single fixed calendar (NO_LEAP or 360_DAY) across an entire multi-century experiment to simplify monthly climatology statistics.
• Why it went unnoticed: Before automated site-level tools like run_tower were written for PLUMBER2/NEON (which clone a single base case across 170+ flux tower sites with differing observational calendars), users typically created new cases individually from scratch rather than cloning a base case with --keepexe while switching calendars.

---

Summary

The placement of CALENDAR in env_build.xml in 2015 was an arbitrary default grouping decision made when writing the initial XML schema. Because --keepexe cloning with usermods wasn't invented until years later, nobody had a reason to notice or change it until multi-site workflows (like PLUMBER2 / NEON) hit this bug.

[billsacks]

The one thing I wondered about is that if you change it midrun -- maybe it causes problems? And it seems likely it would. I'm not sure it would be easy to disallow changing if CONTINUE_RUN==TRUE either. So that might be why it was added to env_build? Having it in env_build restricts changes more than is needed, but just being in env_run might not restrict it enough?

I was just coming back here to write almost exactly this: this thought also occurred to me after I submitted my initial review - this felt like a possible reason it was put in env_build.

I also spent a little more time digging through older history. AI fell short here because it didn't know to look in the older subversion repository. In fact, CALENDAR was moved from env_run to env_build in r36936 of https://svn-ccsm-models.cgd.ucar.edu/scripts/trunk - a change made by @apcraig . There is no comment attached to that commit, and at a glance, I don't see an obvious reason why CALENDAR needed to be moved to env_build in that commit.

In all, I still feel okay with this unless @mvertens or @apcraig wants to comment on why CALENDAR belongs as a build-time setting. A comment may be warranted warning users not to change this setting on a CONTINUE_RUN, but maybe that goes without saying, so I don't feel strongly one way or the other.

[billsacks]

I'll also say, if the purpose of this is partly to explore the value of AI tools in diagnosing problems without pulling in the relevant human/institutional knowledge: I'd say the AI failed in its diagnosis here: Its focus on the timing of CALENDAR vs --keep-exe is misguided, because there were other reasons driving decisions of placement in env_run vs. env_build before --keep-exe was introduced, namely whether a rebuild was needed for any reason when changing a variable. @johnpaulalex I do appreciate your contribution here, but at the same time, as we experiment with relying more heavily on AI tools, I feel it's important to have open conversations about their shortcomings.

[ekluzek]

I trust the AI for the timeline above, and find that part helpful. I think it's speculation regarding --keepexe and run_tower is not as helpful though.

With it giving the timeline I was able to confirm what it said about the initial checkin. At least as far back as we can go with github.com which is 11 years ago from the svn branch. It looks like svn is closed now so I can't see the history previous to that. I thought we had done more to save history on the move to github, but looks like we didn't for this code.

I'm still thinking there could be bad behavior if you try to change this during a run. But, we probably need to test for that. Looking at the code, the calendar is stored on the restart file, but it is NOT used to set the calendar. There doesn't seem to be a read of the calendar in med_phases_restart_read. So it looks to me, like you could mess up time for your simulation if you changed the calendar when CONTINUE_RUN==TRUE.