Split ArrayConverter list elements on whitespace, quotes and the delimiter only

[rootvector2]

ArrayConverter.parseElements set up a StreamTokenizer with its default syntax table, which treats / as a comment start (a/b,c parsed to ["a"], everything after the first slash silently dropped) and splits elements on any non-alphanumeric character outside allowedChars (first_value,second_value parsed to four elements). Per review, the syntax table is now reset so that only whitespace, the delimiter and the quote characters separate elements; every other character is kept as part of an element. first_value,second_value and first/value,second/value both parse to two elements now. Whitespace separation (0 10 parses to two elements) and quote handling ('de,f' keeps its embedded delimiter) are unchanged, as pinned by ConvertUtilsTest. setAllowedChars no longer has any effect and is deprecated; testUnderscore_BEANUTILS_302, testTheMatrix and Jira359Test are updated accordingly.

• Read the contribution guidelines for this project.
• Read the ASF Generative Tooling Guidance if you use Artificial Intelligence (AI).
• I used AI to create any part of, or all of, this pull request. Which AI tool was used to create this pull request, and to what extent did it contribute?
• Run a successful build using the default Maven goal with mvn; that's mvn on the command line by itself.
• Write unit tests that match behavioral changes, where the tests fail if the changes to the runtime are not applied. This may not always be possible, but it is a best practice.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Each commit in the pull request should have a meaningful subject line and body. Note that a maintainer may squash commits during the merge process.

[Copilot]

Pull request overview

This pull request fixes ArrayConverter.parseElements treating / as a comment introducer during StreamTokenizer parsing, which could silently drop later elements when list items contain forward slashes.

Changes:

• Disable / as a comment character in StreamTokenizer setup so it behaves like other non-allowed separator characters.
• Add a regression test ensuring forward slashes split tokens rather than truncating input.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/main/java/org/apache/commons/beanutils2/converters/ArrayConverter.java	Configures StreamTokenizer to treat / as an ordinary character instead of a comment start during element parsing.
src/test/java/org/apache/commons/beanutils2/converters/ArrayConverterTest.java	Adds a unit test covering forward-slash behavior in delimited list parsing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[garydgregory]

@rootvector2 Please address #426 (comment)

[garydgregory]

@rootvector2

Are you sure?

The / now behaves the same as a ,? That can't be right. Shouldn't the asserts be:

assertEquals("first/value", result[0]);
assertEquals("second/value", result[1]);

?

[rootvector2]

Yes. Any non-allowed char splits and keeps both sides by default, same as _ in testUnderscore_BEANUTILS_302 (first_value,second_value parses to 4 elements). Before the patch / was the only char that instead commented out the rest of the input, so a/b,c parsed to just ["a"]. Your asserts hold once / is in the allowed chars, same as _. I extended the test to mirror the underscore test: 4 elements by default, ["first/value", "second/value"] after setAllowedChars(new char[] { '.', '-', '/' }).

[garydgregory]

@rootvector2 Please see my questions.

[rootvector2]

Done on #426, switched the test to AtomicReference.

[garydgregory]

Switched to draft PR until @rootvector2 replies.

[garydgregory]

Hello @rootvector2
Please review the PR vs. git master to see if it is still needed. Thank you!

[rootvector2]

Still needed. master picked up the test in 3d7c330 but without @Test it never runs, and parseElements still has the default comment char, so first/value,second/value parses to ["first"]. I merged master into the branch and added the missing @Test: the test fails without st.ordinaryChar('/') (result.length 1 instead of 4) and passes with it. Full mvn default goal is green.

[garydgregory]

Thank you for the update @rootvector2 , I triggered the builds.

[garydgregory]

Hi @rootvector2
Let's step back for a minute because this change still doesn't make sense to me.

Before the PR, the input "first/value,second/value" yields an array with one element "first" because / is treated as a comment start char. Side note: What about // and /* ... */?

After the PR, we get FOUR elements because both the / and the , are treated as element separators.

I think users would be quite surprised by this new data, as opposed to two elements separated by the one ,.

Please explain.

Thank you!

[rootvector2]

The choice here isn't 2 vs 4, it's 1 vs 4. Before the patch "first/value,second/value" parses to ["first"] and everything after the first / is silently dropped, so nobody gets 2 elements today. The parser has never kept a symbol inside a token unless it's in allowedChars: _ splits the same way (testUnderscore_BEANUTILS_302, "first_value,second_value" is 4 elements, and BEANUTILS-302 was resolved by pointing users at setAllowedChars). / was the only char that truncated instead of splitting, because StreamTokenizer defaults to commentChar('/'). The patch just puts / on the same footing as every other non-allowed char. The 2-element result you describe is the opt-in path, same as for _: setAllowedChars(new char[] { '.', '-', '/' }), and the second half of the test asserts exactly that (["first/value", "second/value"]).

On // and /* ... */: those C-style modes are off by default (slashSlashComments/slashStarComments are false), so the truncation came purely from the single-char end-of-line comment. After ordinaryChar('/') there's no comment handling left; // or /* are just ordinary chars that split.

If you'd rather make / a default allowed char so the out-of-the-box result is 2 elements, that's a one-line change and I'm happy to switch, but it would make / special compared to _ and the rest.

[rootvector2]

The surprising part is what happens today: nobody gets two elements. Before the patch "first/value,second/value" parses to ["first"], so the real choice is between 1 element (silent data loss) and the behavior every other char already has, not between 2 and 4.

Four matches the _ precedent exactly: testUnderscore_BEANUTILS_302 pins "first_value,second_value" to 4 elements by default and to ["first_value", "second_value"] after setAllowedChars(new char[] { '.', '-', '_' }). The patch makes / follow that same rule, and the new test mirrors the underscore test, including the setAllowedChars(new char[] { '.', '-', '/' }) case, which yields exactly the two elements you describe.

On // and /* ... */: parseElements never enables slashSlashComments/slashStarComments (both off by default), so the only live comment syntax was the single-char comment / set by the StreamTokenizer constructor. That's also why truncation starts at the first / no matter what follows. ordinaryChar('/') clears that flag, so after the patch no comment handling remains at all.

If you'd rather plain input give ["first/value", "second/value"] with no configuration, I can add / to the default allowedChars instead. That also keeps the data intact, it just treats / differently from _. Your call.

[garydgregory]

@rootvector2

Thank you for your patience reviewing this issue.

Hm, I see what you're saying, but testUnderscore_BEANUTILS_302 does further confuse matters, I now believe the fix for BEANUTILS-302 was incorrect, incomplete, or misleading.

The intention is stated in the Javadoc:

Parsing Delimited Lists
This implementation can convert a delimited list in String format into an array of the appropriate type. By default, it uses a comma as the delimiter but the following methods can be used to configure parsing:
setDelimiter(char) - allows the character used as the delimiter to be configured [default is a comma].
setAllowedChars(char[]) - adds additional characters (to the default alphabetic/numeric) to those considered to be valid token characters.

and the setters:

    /**
     * Sets the allowed characters to be used for parsing a delimited String.
     *
     * @param allowedChars Characters which are to be considered as part of the tokens when parsing a delimited String [default is '.' and '-']
     */
    public void setAllowedChars(final char[] allowedChars) {
        this.allowedChars = Objects.requireNonNull(allowedChars, "allowedChars").clone();
    }

    /**
     * Sets the delimiter to be used for parsing a delimited String.
     *
     * @param delimiter The delimiter [default ',']
     */
    public void setDelimiter(final char delimiter) {
        this.delimiter = delimiter;
    }

So, by default there is only one delimiter: ,. This tells me both testUnderscore_BEANUTILS_302 and this PR are wrong and surprisingly wrong to boot, to my eye that is.

"first_value,second_value" and "first/value,second/value" should both parse to 2 elements, not 4.

The fact that the internals use a string tokenizer is an implementation detail, likely because this code was written before Java implemented a regex API. If using a regex API is better (split or Pattern), then let's do that.

WDYT?

[rootvector2]

I started with the plain regex split, but ConvertUtilsTest pins two behaviors it breaks: whitespace also separates elements (0 10 converts to two ints) and quotes protect the delimiter ('de,f' stays one element, quotes stripped). The quote handling is documented in the parseElements javadoc too.

So I kept the tokenizer and reset its syntax table instead: only whitespace, the delimiter and the quote chars split, every other character is kept as part of an element. That gives the results you describe: first_value,second_value and first/value,second/value both parse to 2 elements, same for the colon case in Jira359Test, and the comment char is gone as part of the reset. setAllowedChars no longer does anything, so I deprecated it and updated the javadoc and the affected tests.

If you'd rather have strict delimiter-only splitting (no whitespace splitting, no quotes), that's the regex split plus retiring those ConvertUtilsTest cases; say the word and I'll switch. Full mvn default goal is green, 1283 tests.

[garydgregory]

@rootvector2
I think we need to keep the quotes such that commas in elements work. Check?