fix: escape pipes and newlines in CSV values

[asjad3]

Problem

CsvConverter builds its Markdown table with " | ".join(...) and passes cell values
through unescaped. Two characters that are perfectly legal inside a CSV field corrupt
the table, and neither raises:

• a pipe is read as a column separator, so the row gains columns the header never
  declared
• a newline inside a quoted field ends the row early, leaving the rest as stray text
  after the table

The pipe case loses data rather than just looking wrong. Given cheap | fast:

name,description
Widget,"cheap | fast"

main emits a row with three columns under a two-column header:

| name | description |
| --- | --- |
| Widget | cheap | fast |

A renderer drops the surplus column, so the word fast disappears entirely:

<tr><td>Widget</td><td>cheap</td></tr>   <!-- "fast" is gone -->

With this change the pipe is escaped and the value survives:

<tr><td>Widget</td><td>cheap | fast</td></tr>

(HTML above is python-markdown with the tables extension, before and after.)

Change

A small _escape_table_cell() helper applied to header and data cells: pipes are
escaped as \|, and CR/LF/CRLF inside a quoted field become <br> so the record stays
on one line.

Deliberately narrow:

• Backslashes are left alone. Escaping them would be more theoretically correct but
  changes output for ordinary content like Windows paths, with no reported problem to
  justify it.
• <br> is the usual convention for a line break inside a GFM table cell; the
  alternative is collapsing to a space, which loses the author's intent.

Scope note

The HTML converter has the same gap — <td>cheap | fast</td> also renders as a split
cell — and since XlsxConverter routes through DataFrame.to_html() → HtmlConverter,
it inherits it. I left that alone here: it runs through markdownify rather than a
hand-rolled join, so it is a different fix with much wider blast radius across existing
outputs. Happy to follow up if you would like it handled, and equally happy to fold it
in here if you would rather have one change.

Tests

packages/markitdown/tests/test_csv_escaping.py — four cases: pipe in a cell, pipe in
the header, newline in a quoted cell, and a guard that ordinary content is not
over-escaped. The first three fail on main and pass with this change; the fourth
passes either way and exists to catch over-escaping.

tests/test_csv_escaping.py ....                    [100%]
4 passed

test_module_vectors.py + test_module_misc.py: 121 passed, 2 skipped, 1 failed —
test_speech_transcription, which fails identically on a clean checkout here (audio
model output, unrelated to this change). The existing test_mskanji.csv vector contains
no pipes or embedded newlines and is unaffected.

black clean.

---

Written with AI assistance (Claude Code); I reviewed the change and ran the checks
above.

[asjad3]

@microsoft-github-policy-service agree