Given it has been 4 weeks since my last log, I might expect to have a serious amount of updates on progress and some really interesting reflections and ideas to share. While the latter might be somewhat true (the rest of this text will bear testament to that), the quantity of developments, overall, would appear not quite meet my own expectations of where I'd be 4 weeks after starting on this journey. Let me explain.
Famous last words
"This won't take long" were my sentiments at the early stages of the setup of my lab to work through a pipeline, taking 'raw' text (markdown) all the way to embeddings in a vector store (S3 vector bucket). My assessment of the complexity involved several assumptions, the most egregious of which was the assumption that markdown would be easy to process and easy to split uniformly into chunks for storage as embeddings. I decided that I would construct all moving pieces in the pipeline, by hand, at least for the foundational steps in this first lab. My judgement was based on the intuition (and proven experience) that chunking, embedding and RAG were my weak spots - and I needed to do this by hand in order to really understand what is happening end to end. Thus, I looked at already existing markdown parsers and opted to open a new file - to build my own. I also committed to one other pre-requisite - I would not leverage an LLM Agent to write any of the parser. Review passes, examples and other consultations about idiomatic implementations were ok but no 'write it for me' would happen here.
When the difficulty curve sharpens past 89 degrees
I invested several review passes to understand the requirements before building anything. Firstly - what does a chunk even look like? Are there hard requirements? A protocol or defined structure? Do I write out a bunch of .txt files and fire them off for embeddings? There were so many questions. One of the absolutely key first things I learned was the canonical data structure for chunking (and so many other uses in data science in general) - JSONL. Prior to this initial exercise, I incorrectly understood JSONL to be a slight iteration on the JSON data structure which introduced a list like structure between keys ("JSON-List"). I couldn't have been more wrong about that, evidently! Instead, I learned that JSONL is quite literally, a file containing fully encapsulated JSON objects - one per line. JSON-Lines! The chief benefit is enabling iterative streaming from a file, potentially covering 1000's of individual JSON baked structures. Incidentally - perfect for chunking text in a strictly defined schema and writing a reversible log of all text snippets from the corpus that was being chunked.
The implied next question then - what does a chunk look like? I intuited that there should be some form of unique identifier, the text encapsulated by the chunk itself, but there my trail ended. A little cursory research here informed me that a UUID would be necessary for each chunk's ID given that this UUID would become the primary key for retrieval by the S3 Vector bucket. The text itself was a given and I padded out the structure with some additions - notably a chunk index (indicating a segment index within each file being split), location of the file being chunked, measurements (such as token count) and a markdown heading path to reach the text itself. The heading path seemed useful as a semantic pointer to the content, with an implied ancestry presumably available in other chunks. Some iterations later and the schema became very clear :
{
"chunk_id": "abc123",
"chunk_index": 12,
"source_text": "The original passage...",
"retrieval_text": "Topic Overview > Sub-section\n\nThe original passage...",
"metadata": {
"source_file": "topic-of-interest-file.md",
"heading_path": ["Topic Overview", "Sub-section"],
"start_line": 87,
"end_line": 87,
"start_char": 18420,
"end_char": 20176
},
"measurements": {
"source_tokens": 376,
"retrieval_tokens": 384
}
}We've only just begun
Having a well defined schema and a markdown file from an agent's skill/references to begin parsing and chunking, I started assembling the imports needed for my chunker script. It began to slowly, but surely, dawn on me that there was going to be a lot of moving pieces in this implementation. To begin with, I needed something to actually count and approximate the token counts for each chunk. I knew already that OpenAI provide tiktoken but could I use this for embeddings to be eventually referenced by Bedrock? Apparently - yes, or at least there were references from other public builds who used the cl100k_base tokenizer as a decent reference point (for semi-accurate chunk counts prior to actual embedding). Next then was the question - how would I approach actually slicing a document up for chunking? I had read that an optimal chunking strategy includes at least some overlap between subsequent chunks to assist with inference (see my previous log for more) but I had also committed to framing each chunk with a heading ancestry. To accommodate that, I would need to parse for headers ('#,##,###' etc) and establish a map of sorts which lays out the document structure with regards to section/sub-section.
The precursor is all you need
To build a heading ancestry or map, we need to identify all headings in a given markdown document. The markdown spec identifies anywhere between one and six '#' and not strict requirement for any structure whatsoever. So this would work, right?
re.findall(r'#{1,}',text,re.MULTILINE)Well, kind of. As I would later see, the '#' symbol is... flexible. Anyway, from my initial test markdown file - this was totally acceptable. Given that I now have all headings in a given file within a list, I need to find a way to map where each heading sits, relative to the document title heading in order to reliably supply a heading ancestry hint in my chunks. My first attempt was ambitious - I tried to cascade out from the highest anchored parent across all lower nodes and iteratively create a kind of nested dict. This consumed more time than I'd like to admit but the outcome of the trial and error taught me a really important lesson. That lesson was related to building entity graphs. It is enough, even in a relatively complex graph, to store the node and the nodes direct precursor link in order to be able to fully rebuild the entity graph. Essentially, to build a full heading map from a given markdown file, I don't need to store the entire relationship tree all the way down - I just need to store "what is this node" and "what is this nodes direct ancestor". Having that stored, rebuilding the whole graph was doable.
{'#MainHeading':['##ChildHeading','##ChildHeading2'],
'##ChildHeading2':['###ContentHeading1']...}With a rebuild-able entity graph, I could now identify each heading, pull the content between headings in the document and decide how to go about counting the chunks, enforcing limits, correcting threshold overflows and finally outputting an accurate chunked JSONL of my test file - and subsequently larger collections of files in a folder. The resulting parser is still quite rudimentary and will be improved - however for the purposes of my lab and moving on past this module, it is enough. Check out the gist - there is a lot I haven't documented here and the code would probably tell a better story overall.
https://gist.github.com/richarddun/26e490bf6ca842a8c17f2a8cb6d26635
Summary
I've already taken this particular log way longer than I wanted to and haven't covered even a quarter of the challenges I encountered and the useful patterns I've learned - did you know you can map args from argparse directly onto a class in python - but I feel the biggest lesson for me here is scoping work and being unwilling to just stop due to what probably is more stubborn pride than anything else. I could have used a langchain implementation and saved myself more than 3 weeks of implementation and debugging. I could have asked GPT5.6 to just write this for me. I didn't correctly scope the work and when it spiralled into complexity - I didn't stop and change course given the time constraint. I know that I'll make it through but it will likely be more demanding to cover the material in the limited time I have left. My next items to complete this module involve observing the embeddings pipeline in action (now that I have well formatted document chunks) and reviewing theory.
Filed under: AWS, RAG, chunking, JSONL, embeddings, markdown, build-in-the-open.