Skip to content

Bound markdown chunks - #370

Open
abelfx wants to merge 3 commits into
singnet:mainfrom
abelfx:OMEGA-369-bound-markdown-chunks
Open

abelfx wants to merge 3 commits into
singnet:mainfrom
abelfx:OMEGA-369-bound-markdown-chunks

Conversation

@abelfx

@abelfx abelfx commented Sep 29, 2026

Copy link
Copy Markdown

Description

Fixes #369.

_chunk_markdown returned a heading-less file as one chunk and skipped MAX_CHUNK_CHARS. A file with headings could still exceed the cap when a single paragraph was longer than the limit, because the splitter only cuts on blank lines.

  • A file with no # heading is now one section and goes through the same size pass as headed files.
  • Anything still over MAX_CHUNK_CHARS after the paragraph split is cut on a newline or space, or mid-token when there is no whitespace.
  • A warning is logged when a file has no headings, and when a character-level cut is applied.

How Has This Been Tested?

python3 -m pytest tests/test_rag_chunking.py

  • A heading-less file larger than MAX_CHUNK_CHARS is split, and every chunk stays within the cap.
  • The same text with one heading stays within the cap.
  • One paragraph with no blank lines is cut.
  • A string with no whitespace is cut mid-token and the pieces join back to the original text.
  • Cuts land on word boundaries when spaces are present.
  • Heading-less chunks keep the filename as the breadcrumb.
  • A short heading-less file stays one chunk.
  • Heading breadcrumbs (Top, Top > Nested, Second) are unchanged.
  • Warnings are logged for a heading-less file and for a character-level cut.

Checklist

  • PR contains autogenerated code
  • Self-review completed
  • Test scenarios above are passed with the version of the code from PR

@alyona-snet alyona-snet changed the title [OMEGA-369] Bound markdown chunks Bound markdown chunks Sep 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_chunk_markdown returns the whole file as a single chunk when the file has no # heading, ignoring MAX_CHUNK_CHARS

1 participant