Revert "Add workflow to automatically generate and commit site HTML (…

.github/CONTRIBUTING.md

@@ -1,5 +1 @@
-# Contributing to this repo
-
-Build and test your changes locally according to the instructions in [README](../README.md).
-
-Once you've done that, submit a pull request with your changes. You only need to commit your changes to the Markdown source. A GitHub Actions workflow will [generate the corresponding HTML and push it for you](./workflows/html-push.yml).
+Make sure that you generate site HTML with `jekyll build`, and include the changes to the HTML in your pull request also. See README.md for more information.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1 @@
+<!-- *Make sure that you generate site HTML with `bundle exec jekyll build`, and include the changes to the HTML in your pull request. See README.md for more information.* -->

.github/workflows/doc_gen.yml

@@ -0,0 +1,76 @@
+name: Check document generation
+
+on:
+  push:
+    branches:
+    - asf-site
+  pull_request:
+    branches:
+    - asf-site
+
+jobs:
+  lint:
+    name: check whether all documentation was generated with the right Jekyll version
+    runs-on: ubuntu-24.04
+    steps:
+    - name: Free up disk space
+      shell: 'script -q -e -c "bash {0}"'
+      run: |
+        echo "=================================="
+        echo "Free up disk space on CI system"
+        echo "=================================="
+
+        echo "Listing top 100 largest packages (from large to small)"
+        printf "Installed-Size\tPackage\n"
+        dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n -r | head -n 100
+        df -h
+
+        echo "Removing large packages"
+        rm -rf /__t/CodeQL
+        rm -rf /__t/go
+        rm -rf /__t/node
+
+        apt-get remove --purge -y '^aspnet.*' || true
+        apt-get remove --purge -y '^dotnet-.*' || true
+        apt-get remove --purge -y '^llvm-.*' || true
+        apt-get remove --purge -y 'php.*' || true
+        apt-get remove --purge -y '^mongodb-.*' || true
+        apt-get remove --purge -y 'gfortran-11' || true
+        apt-get remove --purge -y 'humanity-icon-theme' || true
+        apt-get remove --purge -y 'nodejs-doc' || true
+        apt-get remove --purge -y snapd google-chrome-stable microsoft-edge-stable firefox || true
+        apt-get remove --purge -y azure-cli google-cloud-sdk mono-devel powershell libgl1-mesa-dri || true
+        apt-get autoremove --purge -y
+        apt-get clean
+
+        df -h
+    - name: Checkout Spark Website repository
+      uses: actions/checkout@v2
+    - name: Install dependencies for documentation generation
+      run: |
+        sudo apt-get update -y
+        sudo apt-get install -y ruby ruby-dev
+        sudo gem install bundler --version 2.4.19
+        bundle install
+    - name: Run documentation build
+      run: |
+        export LC_ALL=C.UTF-8
+        export LANG=C.UTF-8
+        OLD_IFS=$IFS
+        IFS=
+        GEN_ERRORS=$(bundle exec jekyll build 3>&2 2>&1 1>&3)
+        if [ $(echo $GEN_ERRORS| grep -v -e '^$'| grep -c -v "rubygems_integration") -ne 0 ]; then
+          echo "Error during document generation:"
+          echo $GEN_ERRORS
+          exit 1
+        fi
+        IFS=$OLD_IFS
+        CHANGED_FILE=( $(git ls-files --modified --other --exclude-standard --directory | grep -v sitemap.xml | grep -v llms.txt) )
+        if [ ${#CHANGED_FILE[@]} -ne 0 ]; then
+          echo "Not all documentation was generated and/or not the right Jekyll version was used! Modified / untracked files (excluding sitemap.xml):"
+          echo ${CHANGED_FILE[*]}
+          echo "Git diff (excluding sitemap.xml):"
+          git diff -- . ':(exclude)site/sitemap.xml'
+          exit 1
+        fi
+      shell: /bin/bash {0}

.gitignore

@@ -4,4 +4,4 @@ target/
 .jekyll-cache/
 .jekyll-metadata
 .local_ruby_bundle
-site/
+site/python

README.md

@@ -1,12 +1,4 @@
-# Apache Spark Main Website
-
-This repository captures the main Apache Spark website located at https://spark.apache.org. The programming docs are [in the main Spark repo][1], not here.
-
-[1]: https://github.com/apache/spark/tree/master/docs
-
-To contribute changes, see [CONTRIBUTING](.github/CONTRIBUTING.md).
-
-## Generating the website HTML locally
+## Generating the website HTML
 
 In this directory you will find text files formatted using Markdown, with an `.md` suffix.
 
@@ -36,6 +28,22 @@ of Spark from the Spark source repository and then copied to the website under t
 directory. See the instructions for building those in the readme in the Spark 
 project's `/docs` directory.
 
+## Rouge and Pygments
+
+We also use [Rouge](https://github.com/rouge-ruby/rouge) for syntax highlighting in documentation Markdown pages.
+Its HTML output is compatible with CSS files designed for [Pygments](https://pygments.org/).
+
+To mark a block of code in your Markdown to be syntax highlighted by `jekyll` during the 
+compile phase, use the following syntax:
+
+    {% highlight scala %}
+    // Your Scala code goes here, you can replace Scala with many other
+    // supported languages too.
+    {% endhighlight %}
+
+You probably don't need to install that unless you want to regenerate the Pygments CSS file.
+It requires Python, and can be installed by running `sudo easy_install Pygments`.
+
 ## Merge PR
 
-To merge a pull request, use the `merge_pr.py` script. This script also squashes the commits.
+To merge pull request, use the `merge_pr.py` script which also squashes the commits.

_config.yml

@@ -4,6 +4,6 @@ kramdown:
   entity_output: symbol
 permalink: none
 destination: site
-exclude: ['README.md', 'LICENSE', 'merge_pr.py', 'Gemfile', 'Gemfile.lock']
+exclude: ['README.md', 'content', 'LICENSE', 'merge_pr.py', 'Gemfile', 'Gemfile.lock']
 keep_files: ['docs', 'static', 'llms.txt']
 url: https://spark.apache.org

content

@@ -0,0 +1 @@
+site

site/sitemap.xml

@@ -1163,10 +1163,7 @@
   <loc>https://spark.apache.org/releases/spark-release-0-3.html</loc>
   <changefreq>weekly</changefreq>
 </url>
-<url>
-  <loc>https://spark.apache.org/</loc>
-  <changefreq>weekly</changefreq>
-</url>
+
 <url>
   <loc>https://spark.apache.org/404.html</loc>
   <changefreq>weekly</changefreq>
@@ -1208,65 +1205,70 @@
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/graphx/</loc>
+  <loc>https://spark.apache.org/history.html</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/history.html</loc>
+  <loc>https://spark.apache.org/improvement-proposals.html</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/improvement-proposals.html</loc>
+  <loc>https://spark.apache.org/spark-connect/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/mailing-lists.html</loc>
+  <loc>https://spark.apache.org/pandas-on-spark/</loc>
+  <changefreq>weekly</changefreq>
+</url>
+<url>
+  <loc>https://spark.apache.org/graphx/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
   <loc>https://spark.apache.org/mllib/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/news/</loc>
+  <loc>https://spark.apache.org/streaming/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/pandas-on-spark/</loc>
+  <loc>https://spark.apache.org/news/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/powered-by.html</loc>
+  <loc>https://spark.apache.org/screencasts/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/release-process.html</loc>
+  <loc>https://spark.apache.org/sql/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/research.html</loc>
+  <loc>https://spark.apache.org/</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/screencasts/</loc>
+  <loc>https://spark.apache.org/mailing-lists.html</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/security.html</loc>
+  <loc>https://spark.apache.org/powered-by.html</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/spark-connect/</loc>
+  <loc>https://spark.apache.org/release-process.html</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/sql/</loc>
+  <loc>https://spark.apache.org/research.html</loc>
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/streaming/</loc>
+  <loc>https://spark.apache.org/security.html</loc>
   <changefreq>weekly</changefreq>
 </url>
+
 <url>
   <loc>https://spark.apache.org/third-party-projects.html</loc>
   <changefreq>weekly</changefreq>
@@ -1279,4 +1281,5 @@
   <loc>https://spark.apache.org/versioning-policy.html</loc>
   <changefreq>weekly</changefreq>
 </url>
+
 </urlset>

sitemap.xml

@@ -151,18 +151,9 @@ sitemap: false
   <changefreq>weekly</changefreq>
 </url>
 {% endfor %}
-{%- comment -%}
-Explicitly sort `site.pages` so that the order is consistent and we don't get spurious git diffs.
-`site.posts` doesn't have this issue because it's already sorted.
-See: https://jekyllrb.com/docs/variables/#site-variables
-{%- endcomment -%}
-{%- assign sorted_pages = site.pages | sort: "url" -%}
-{%- for page in sorted_pages -%}
-{%- if page.sitemap != false -%}
-<url>
+{% for page in site.pages %}{% if page.sitemap != false %}<url>
   <loc>{{ site.url }}{{ page.url }}</loc>
   <changefreq>weekly</changefreq>
-</url>
-{% endif %}
-{%- endfor -%}
+</url>{% endif %}
+{% endfor %}
 </urlset>