[701] Keep exactly one LICENSE and NOTICE in every published jar - #876
Open
vinishjail97 wants to merge 2 commits into
Open
[701] Keep exactly one LICENSE and NOTICE in every published jar#876vinishjail97 wants to merge 2 commits into
vinishjail97 wants to merge 2 commits into
Conversation
Every jar carried its licensing information twice. META-INF/LICENSE-bundled and META-INF/NOTICE-bundled are inputs for the shade plugin, which writes their content out as META-INF/LICENSE and META-INF/NOTICE, but they were also packaged verbatim because they live in src/main/resources. In the non-shaded jars that second copy was not just redundant, it described dependencies that are not in the jar at all. On top of that, DontIncludeResourceTransformer only drops the exact resource names LICENSE, NOTICE and NOTICE.txt, so the shaded jars additionally carried each bundled dependency's own license file wherever that dependency happened to put it. xtable-hive-metastore-bundled shipped 19 such files across LICENSE.txt, about_files/, license/, META-INF/license/, META-INF/ASM_LICENSE.txt and META-INF/CLI-LICENSE.txt, which made the licensing of the bundle very hard to review. maven-jar-plugin now excludes the two -bundled files from the jar. They stay in target/classes, which is where IncludeResourceTransformer reads them from, so the shaded jars still get the bundled listing as META-INF/LICENSE while the non-shaded jars keep the plain Apache-2.0 text. The shade filters drop the non-canonical dependency license files. The source files under src/main/resources/META-INF are untouched, so validate_shaded_license_coverage.sh keeps working against them unchanged. Adds validate_jar_license_layout.sh, wired into the License Check workflow, to assert the invariant against the built jars: exactly one META-INF/LICENSE, a META-INF/NOTICE, and third party texts only under META-INF/licenses/. Run against the 0.4.0-incubating-rc1 artifacts it reports all 21 offending entries, which is how this went unnoticed for several releases. xtable-utilities is skipped, matching validate_shaded_license_coverage.sh: its deploy is skipped and create_source_release.sh excludes it, so it is not part of the published set. Its shaded jar has no META-INF/LICENSE at all, which is worth fixing separately.
…ndent Verifying the previous commit on main surfaced LICENSE-junit.txt in the root of xtable-hive-metastore-bundled, which branch-0.4 does not produce: the two branches resolve different junit versions, and only one of them ships that file. Enumerating the suffixes seen on one branch is therefore not enough, so the excludes now cover LICENSE*, NOTICE* and COPYING* at the jar root and META-INF/*LICENSE* / META-INF/*NOTICE* rather than a fixed list of extensions. Our own files are unaffected: META-INF/LICENSE and META-INF/NOTICE are written by IncludeResourceTransformer after filtering, and META-INF/licenses/ sits in a path segment of its own so none of these patterns can match it. Confirmed by rebuilding every shaded module from clean and checking that the bundled jars still carry the dependency listing (56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions, 8 for spark-runtime) while the non-shaded jars still carry the plain 202 line Apache-2.0 text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every jar carried its licensing information twice, and the shaded jars also carried each dependency's own license file wherever that dependency happened to put it. Now: exactly one
META-INF/LICENSE, oneMETA-INF/NOTICE, third party texts only underMETA-INF/licenses/.Closes #701.
branch-0.4counterpart is #875, needed for RC2.Why
Both binding voters hit this in the RC1 vote: Stamatis cited #701 as
[4]in his-1, and JB noted that META-INF carries both LICENSE and LICENSE-bundled and "seems to be redundant".Root cause
LICENSE-bundled/NOTICE-bundledare shade inputs, written out asMETA-INF/LICENSEandMETA-INF/NOTICE, but they also live insrc/main/resourcesso they were packaged verbatim too. In the non-shaded jars that second copy described dependencies that are not in the jar.DontIncludeResourceTransformeronly drops the exact namesLICENSE,NOTICE,NOTICE.txt, soLICENSE.txt,META-INF/license/*,about_files/*,META-INF/ASM_LICENSE.txtand friends survived.Changes
maven-jar-pluginexcludes the two-bundledfiles. They stay intarget/classes, whereIncludeResourceTransformerreads them, so bundles keep the dependency listing and non-shaded jars keep the plain Apache-2.0 text.mainandbranch-0.4do not produce the same set.release/scripts/validate_jar_license_layout.sh, wired into the License Check workflow, asserts the invariant against the built jars.Files under
src/main/resources/META-INFare not moved, sovalidate_shaded_license_coverage.shis unaffected and this does not overlap #857.Verification
Rebuilt every shaded module from clean on both branches. Non-shaded jars: bare Apache-2.0, 202 lines, 0 dependency coordinates. Bundles: Apache-2.0 plus the listing, 56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions. No
-bundledfile in any jar,META-INF/NOTICEintact, andhive-metastore-bundledwent from 19 non-canonical license files to zero. Run against the real 0.4.0-incubating-rc1 artifacts the new script reports all 21 offending entries. RAT, Spotless andvalidate_shaded_license_coverage.shpass.Known gaps
META-INF/licenses/. Closing that needs oneIncludeResourceTransformerper text, roughly 37 once [700] Bundle license texts for all non-Apache-2.0 shaded dependencies #857 lands, because shade reads the project's built jar as input. Better done after [700] Bundle license texts for all non-Apache-2.0 shaded dependencies #857.main,hive-metastore-bundledinherits five texts from a dependency's ownMETA-INF/licenses/. Pre-existing and in the canonical location, so it belongs with Incomplete license information for bundled jars #700 / Orphaned license files in bundled jars #865.xtable-utilities-bundledhas noMETA-INF/LICENSEat all. Skipped here since it is not published; worth its own issue.