Add workflow to automatically generate and commit site HTML

[nchammas]

As proposed here, I believe it would be best to overhaul the workflow for this project by splitting the source into a master branch, and having asf-site be just for the generated HTML. We could even have per-branch staging sites generated and published automatically for us by ASF.

However, this PR takes a more conservative approach as suggested by @cloud-fan and reworks the GitHub Actions workflows as follows:

• html-build.yml: Added a new workflow that runs on PRs and simply generates the docs. This confirms the Jekyll build is not obviously broken.
• html-push.yml: Added a new workflow that runs on new commits to asf-site. It generates the docs and pushes the generated HTML automatically as a new commit to asf-site.
• doc_gen.yml: Deleted since it is subsumed by the new workflows.

This PR also removes the content/ symlink to site/, which doesn't appear to be needed. ASF understands Jekyll configs and looks here to know where to serve the site from:

spark-website/_config.yml

Line 6 in fd110d9

destination: site

TODO:

• Remove current branch from push workflow.
• Confirm ASF destination config with committer.

[nchammas]

I need a maintainer to approve the workflow runs so I can test my changes.

[nchammas]

Thank you to whoever approved the previous workflows. I need another approval.

Could I perhaps be added to some workflow whitelist so maintainers don't have to manually approve my workflows every time?

[nchammas]

Looks like everything is working. New [html] commits are not being pushed anymore because there are no changes to the HTML output. If I make a change to the source that changes the output HTML, we will see a new [html] commit pushed to my branch.

This is just test behavior. Before merging, I will remove it so that [html] commits are only pushed to asf-site when necessary.

[cloud-fan]

so PR only need to modify md files, once PR is merged, a post-merge job will regenerate html and push? how do we handle concurrency? like 2 PRs merged almost at the same time?

[nchammas]

so PR only need to modify md files, once PR is merged, a post-merge job will regenerate html and push?

Correct.

how do we handle concurrency? like 2 PRs merged almost at the same time?

There should be no problem.

1. PR A is merged and pushes commit C1 to asf-site.
2. Immediately after, PR B is merged and pushes commit C2 to asf-site.
3. The HTML workflow for commit C1 fails when it tries to push an HTML commit because its checkout is stale. C2 has already been pushed.
4. The HTML workflow for commit C2 succeeds and includes the HTML for both commits C1 and C2.

To be a bit cleaner and avoid the spurious workflow failure in this scenario, I've added a concurrency group so that the workflow for commit C1 would be canceled immediately once the workflow for commit C2 starts.

[nchammas]

Could I perhaps be added to some workflow whitelist so maintainers don't have to manually approve my workflows every time?

@cloud-fan - Checking in again about this. Not absolutely required, obviously, but it would make it easier for me to contribute here.

I understand there is a repo setting to require workflow approval only for first-time contributors to a repo. You can also add me specifically as a read-only repo collaborator.

[cloud-fan]

to be clear, the post-merge job will generate HTML for the current codebase, so even if something goes wrong, a later post-merge job will make md files and HTML fully in sync again?

BTW I don't think Apache allow us to add a workflow approver, so it has to be approved every single time...

[nchammas]

to be clear, the post-merge job will generate HTML for the current codebase, so even if something goes wrong, a later post-merge job will make md files and HTML fully in sync again?

Correct.

Every commit to asf-site will trigger a build of the HTML. A new commit will be pushed with [html] prepended to the title of the latest commit on the branch. The [html] string in the title is used as a marker to prevent the workflow from building and pushing the HTML again.

spark-website/.github/workflows/html-push.yml

Lines 13 to 15 in 6eab236

	# This condition is important. We don't want to trigger this job if the last
	# commit was created _by_ this job!
	if: "!contains(github.event.head_commit.message, '[html]')"

Whenever a new commit is pushed to asf-site, as long as it does not include this [html] marker, that will trigger a new build + push.

[cloud-fan]

as long as it does not include this [html] marker

Can we use a more internal marker that no one will accidentally use in their own PR title?

[nchammas]

I kept the [html] marker but updated the condition to also check the commit author:

spark-website/.github/workflows/html-push.yml

Lines 15 to 19 in bf80b5e

	if: >-
	!(
	contains(github.event.head_commit.message, '[html]') &&
	github.event.head_commit.author.name == 'github-actions[bot]'
	)

That way, even if contributors accidentally use [html] in their commit message, it won't have any impact.

[cloud-fan]

LGTM if CI is green

[nchammas]

Just wanted to double check one thing with you before you merge:

Confirm ASF destination config with committer.

Is my analysis about site correct? We don't have a staging deployment, so I can't test this out.

In any case, I'll be watching closely here after you merge to make sure everything builds and deploys correctly.

[cloud-fan]

yea there is no staging site AFAIK.

[nchammas]

Looks like something went wrong! Perhaps content/ is needed after all? I'm not sure. This is where a staging site would help.

@cloud-fan - Better revert for now and I'll look into this.

[nchammas]

I think the problem is content/, even though I cannot find any documentation from ASF Infra about it.

I will open a new PR (after we first get the site back up) with all the same changes here, minus the deletion of the content/ symlink and its mention in _config.yml. I will also add some documentation for it to our README so that nobody else assumes it is no longer needed. 😅