It is a goal of mine to go deeper into design patterns and deployment best practices, based on the AWS cloud, by the end of the year. I've stalled somewhat on this goal through this year so far, mostly because I was having so much fun defining higher layer constructs such as MCP servers, scripts and agent workflows both in Kiro and pi.dev agent harnesses. I did also spend some time tinkering with the dexto agent framework, but ran into some issues with this harness parsing DeepSeek's markup language intermittently when the v4 model was reasoning, so I pivoted to pi.dev - a choice which I'm still happy with (though I assume the dexto team have patched their harness given the popularity of pro/flash Deepseek v4 models). At any rate, a definite and tangible milestone for this overall goal involves studying for and achieving the AWS GenAI Developer Professional certification. I'm opening this learning log to document the challenges and demystification 'aha!' moments I encounter along the way. Perhaps it can help others walking this same path in future.

The certification

The cert in question is the AWS Certified Generative AI Developer – Professional (exam code AIP-C01). It's pitched at people doing GenAI development day to day: integrating foundation models into applications and business workflows, and getting those solutions into production on AWS. The exam is 65 scored questions (plus 10 unscored), reported on a 100–1,000 scale with a 750 passing score.

The scored content is split across five domains, each with its own weighting:

  • Foundation Model Integration, Data Management, and Compliance — 31%. Selecting and configuring foundation models; data validation and processing pipelines; vector stores and retrieval (chunking, embeddings, hybrid and semantic search); prompt engineering and governance.
  • Implementation and Integration — 26%. Agentic AI solutions and tool integrations; model deployment strategies; enterprise integration architectures; FM API integrations.
  • AI Safety, Security, and Governance — 20%. Input and output safety controls; data security and privacy; governance and compliance; responsible AI principles.
  • Operational Efficiency and Optimization for GenAI Applications — 12%. Cost optimization, performance tuning, and monitoring/observability for GenAI applications.
  • Testing, Validation, and Troubleshooting — 11%. Evaluation systems for GenAI and troubleshooting GenAI applications.

For my purposes the weighting tells a useful story: the exam leans hard on actually integrating models, managing the data around them, and keeping the whole thing safe and governed. That maps neatly onto the hybrid-search and RAG groundwork I've already started in this first log — chunking, lexical search, and (next up) embeddings and semantic search all sit inside Domain 1.

My starting point

As previously mentioned, I have been living within already stood up agent harnesses and riffing with context, instructions, workflows and tooling. This particular journey deserves its own article, but I will state here that I have a pretty robust understanding and intuition for what a given harness / agent pair can do well and where things degrade. Where the topic is more foundational - this is the gap I'm working to fill and document in these logs. This gap, to be clear, is how to take a given harness with its turn handling, tool calling, reasoning and all the rest - then map that to a hosted architecture involving AWS services and some custom implementations. To build a hosted agent harness with memory, knowledge and tools would be an ideal outcome from this endeavour (in addition to clearing the bar and achieving the certification itself).

General notes about my environment

I'm running a lightweight Lightsail instance, primarily for persistence and also due to the fact that the hardware I have access to locally is quite underpowered. The most cost-efficient Lightsail offering is quite suitable for my needs. It does, however pose one slight challenge - you can't assign a service role to a Lightsail instance. Because of this, I have opted (at least initially) to create a tightly scoped IAM user with access credentials that expire within a reasonably short time. This allows me to call Bedrock and S3 directly from my Lightsail instance which will be key as my lab expands. I'm also, to be clear, enlisting the help of GPT5.6 in cataloguing requirements for each topic in the study course and working through each challenge/lab. I'm doing my best to not ask for answers directly, instead asking GPT5.6 to help tease out some ambiguities I have and confirm my understanding along the way.

Lab : Understanding Chunking and Lexical search

Chunking de-mystified

In my first lab, I've worked through one key term which I totally misunderstood from the beginning : Chunking. In order to divvy up a collection of documents and implement structure (needed by further steps in Embedding and Search), we must impose order on the collection. We must first decide how to split the documents into blocks and make our search/embedding efficient and scalable. I was surprised to learn that this literally involves deciding how to split a given document and creating independent fragments, all aligning to the same strategy across the rest of the corpus. There is no encoding, encryption, compression or any real protocol for 'chunking' - it is directly a decision on how to split text, uniformly, for use later. Some strategies I encountered include overlapping splits, structural splits or token based splits. And, yes, to my amazement, using something like the GNU split command on a text document split -l 50 <file_name_pattern> file.txt is a totally valid (but perhaps sub-optimal) chunking strategy!

  • Overlapping splits : Take a document and pull out lines 1-100 to a 'chunk'. Then pull out lines 80 - 180 to a separate 'chunk'. Repeat until done. This preserves a fragment between chunks that can help search later (especially if a sentence is cut with the static line split here).
  • Structural splits : Pivot on the document structure itself. If you have a markdown file, call each chunk at headings ##, ### etc. If you have a body of .pdf documents, identify structural markers (again, headings etc) to split out the text content. Essentially, use markup where it is in place as a pivot, where markdown is not available, use collections of linebreaks perhaps to decide a section for chunking.
  • Token based splits : Using something like the Python NLTK, you can identify sentence and paragraph boundaries and other tokens present in a collection of documents to signify partitions in the text where a chunk makes sense.

This is by no means exhaustive, just a demonstration of how 'chunking' is entirely a text parsing strategy to try and preserve continuity within the extracted segments for embedding/search later.

For my lab, I opted to go with simplicity. I pulled overlapping splits from my test corpus and stored the chunks for re-use later. The next portion of my first lab involved setting up a rudimentary lexical search (one of the main aims of this portion of my journey is to get closely familiar with 'Hybrid' search - using both lexical and semantic search for RAG implementations).

Enter SQLite3 and FTS5

My lab partner, GPT5.6, suggested I try to use SQLite3 to implement a basic lexical search. Having zero prior experience with FTS5, I read the documentation and found that this sqlite3 extension (?) performs perfectly well for searching within a large body of documentation. I was frankly astounded to learn that this was both available in the standalone sqlite3 application and within the module bundled in the python stdlib. One needs only to supply the previously created chunks when creating an FTS5 table and then lexical search is immediately do-able.

Reciprocal Rank Fusion

Through my testing using sqlite3 and searching through the chunked text within the database locally, I learned that a key part of Hybrid search is algorithmic 'ranking' of returned chunks. Thus, if I perform a search with a string of text and that string appears in chunk A (rank 1), chunk E (rank 2) within the returned lexical search, but appears within chunk A (rank 1), chunk E (rank 3) within the semantic search (embedding search), the results must be 'ranked' invisibly so the right context can be supplied to the LLM agent. One such algorithm is "Reciprocal Rank Fusion". It takes the rank id (number) for each returned chunk, divides it by 60 (my take : a callback to ancient Babylon, but that's another matter) and returns the sum.

Thus, in the example provided :

chunk A
Lexical search : rank 1
Semantic search: rank 1

chunk E
Lexical search : rank 2
Semantic search: rank 3

chunk A : 1/(60+1) + 1/(60+1) = (0.01639 + 0.01639) = 0.03278 *wins*
chunk E : 1/(60+2) + 1/(60+3) = (0.01613 + 0.01588) = 0.03201 *loses*

And chunk A is indicated by RRF to be the correct context to use in LLM Agent context hydration.

End result

Through my initial lab and investigation, I learned that chunking is as simple as you want it to be, RRF uses 60 as a 'smoothing constant' which is very interesting, FTS5 is incredibly powerful and have laid the ground work for Hybrid search. Next lab will likely involve setting up embeddings for the other side of this Hybrid. See you there!

Filed under: AWS, RAG, search, hybrid search, chunking.