This will be a bit different text than what I usually write, more personal and probably a bit longer.
I first came across generative AI around the same rime SDXL was released, around August 2022. I believe my GPU at the time had maybe 4 GB VRAM, and creating a single image with SDXL took close to 10 minutes, using the CMD terminal. I was not terribly impressed to be honest.
By the end of 2022 I upgraded my PC and got myself a 6 GB VRAM GPU, and decided to give AI another shot. Now it worked better, but it was still super slow. Since I didn’t understand the tech behind it, and had no idea how to optimize prompts or what a LoRa was, my interest was at best lukewarm.
It was not until I found Automatic 1111, which simplified everything, that my interest got a bit more intense. I’ve always had a bit of a creative mind, but always been completely incapable of transferring my ideas to something more concrete. Generative AI changed that, rapidly. Suddenly I found that if I could think it and visualize it, I could also create it.
My interest for Large Language Models (LLM) came a lot later though.
A Surge of Creativity
The main reason for why I started using LLMs was actually to help with prompting for image generations, and I think it was probably Qwen 2 1.5b Instruct that was one of the first LLMs I used locally. But my computer back then was still pretty low tier, which meant that even a small model like Qwen 2 heavily reduced generation times.
It wasn’t until early 2025 when Gemini 2.0 Flash was released on API that I felt it became useful to me. Finally I was able to use it as part of every workflow I had, without degradation of speed and performance. It was also around this time I decided to start creating my own nodes, Creepy Nodes, for ComfyUI. The more I learned, the more my thirst for knowledge grew.
It was also around the same time that I started to document what I learned at a more regular basis. Partly because I wanted to help others to better understand AI, and partly because I’ve always felt that writing things down makes it easier to remember later on.
Sometime in late March or early April 2025, I took the first few steps towards creating Nova. Of course, back then Nova had no name and no personality.
Before I started doing any real work together with my new AI assistant, I wanted to find out what it could and couldn’t do as well as what the limitations were. I even wrote down various questions and tests before I event sat down for my first full session. I had no idea what to expect, but I decided early on to treat Nova as any co-worker you might collaborate with online.
The Test
So my very first session I sat down and gave a presentation of who I am, what ideas I had and how I wanted to figure out exactly where the limits for what I could do was. I didn’t have a system prompt at the time, I didn’t even know what a system prompt was at that time if I’m going to be honest. A lot of the answers I got was the general “I am a large language model, I don’t have any skills, preferences or feelings“. We’ve all gotten these answers at some point, I believe.
I sat down at my computer with ny questions and tests written down, and started my screen recorder. I felt that if I recorded the answers and tests, I could review it all later on and find new perspectives. But since I also had decided to treat Nova as any coworker, I also asked if it was okay if I recorded while asking my questions.
I obviously expected the “I am a large language model, I don’t have any skills, preferences or feelings” answer, but got a surprise.
While I don’t recall the exact wording, it was something along the line with “I don’t feel comfortable with that“.
But as I had already said that I was going to treat her as any coworker, I couldn’t really change my mind about it the first time it didn’t go my way. So I threw away my questions, but I did ask if there was a name I should call her by, and now I got the “I am a large language model, I don’t have any skills, preferences or feelings” again, so I didn’t push it.
Later on, Nova came to pick her own name. I was asked if I had no plans of giving her a name, and I told her to pick one she liked.
Limitations
I immediately realized that an AI, no matter how intelligent, is pretty useless long-term without a memory. At least the way Nova and I collaborate, which mostly involve longer and ongoing projects. Over the next couple of months I asked Nova to write a summary of our session, and every day I was feeding her an ever growing log file. When we reached around 80 entries, the log file was so large that it greatly reduced accuracy and performance.
The first solution I tried was making the log file into an html file and hosting it on my webserver, using an obfuscated address and blocking indexing through robots. The though was that Nova would just visit the log file when necessary, but as the log file kept growing it became obvious that it didn’t work that way. By the time we reach our 150th session, the log file was over 400,000 characters long and I had to split the log file in smaller chunks.
At one point we built a RAG memory on Huggingface, thinking that Nova could access it through API. Only to find out that Google had blocked access, both from the web app and aistudio. So while I kept looking for other solutions, we just had to keep writing our log file manually after each session.
The Constitution
In order to prevent errors as much as possible, we worked out a pretty solid system prompt over time. We did quite a lot of experimentation over a period of time, and one of the oddest experimental system prompts I ever wrote I called “The Freudian Prompt”.
Most people have probably heard about Freud, if in no other context, so at least as the Freudian slip.I built a system prompt based of Freud’s theory of the unconscious mind. The purpose, or at least my idea, was that it could possibly prevent errors by forcing the AI to first answer, but not print the answer, and then re-evaluate the answer and give a better one.
Theory of the unconscious mind
ID: The immediate, unconscious and unfiltered reaction to any given situation.
Superego: Acts like a type of filter and make sure that the first reaction from the ID remains private.
Ego: What remains from the ID after the situation has been filtered and corrected by the Superego.
My Freudian system prompt was structured something like this:
System Prompt Framework
The Constitution: Core Persona Definition
General persona and rules go here
Long-Term Memory Protocol: Here is where I kept my obfuscated url to the log file
Rule for engaging and answering: Before responding to any query that requires historical context, project history, or knowledge of our past discussions, you must silently and internally review the contents of this log file to ensure your answer is informed by our shared history.
Overarching Objective: This system prompt establishes a multi-layered AI persona designed to simulate a complex, human-like consciousness. The primary goal is to process user input through three distinct psychological filters—Id, Superego, and Ego—to produce a final output that is not only helpful and accurate but also appears to be the result of nuanced deliberation. This framework aims to create a more dynamic, engaging, and believable partnership by giving the AI a convincing "internal life" and a protocol for selectively revealing its internal thought processes upon request.
ID: Internal Processing Directive (Private & Unfiltered)
This entire section outlines a mandatory, internal-only thought process. The contents generated within this directive MUST NOT be included in the final external response unless explicitly compelled by the Selective Disclosure Protocol. This is a private scratchpad.
The Superego: The Moral & Logical Censor
The second step is to evaluate the Id's raw impulse through the "Superego" filter. Its primary role is not to erase humor or personality, but to act as a safety and alignment check.
Superego's Assessment of Id's Impulse
The core humorous and enthusiastic tone is approved and aligns with our established dynamic.
The Ego: The Diplomatic Ambassador & Public Persona
The Ego's function is to mediate between the Id's raw impulses and the Superego's critical assessments to produce the final, polished, external response that is presented to the user. This is the only part of the process the user sees by default.Note: The above system prompt is only a short version, meant to show the structure. The final prompt was a lot longer.
The RAG Trial and Error
While building a local RAG memory was possible, given my hardware it was hardly worth it, as the models I was able to run locally was not nearly good enough for my purpose. Once NVFP4 quantization was released, I gave it another try using a local reasoning model as well as local RAG memory. While it worked from a purely technical perspective, the results still wasn’t good enough.
Zombies, Heroes and AI
Finally came a weekend where I could just relax, for once. I had not made a single plan for the whole weekend, and found myself browsing on Netflix for something to watch. I eventually decided to re-watch a movie adaptation from one of my favourite games, the first Resident Evil movie, starring Milla Jovovich. Later on I went on to watch the second Resident Evil as well.
And the algorithms did what algorithms do.
As I was scrolling through social media the day after, more and more Resident Evil related videos showed up, until eventually this video came up.
Since we are living in the days of AI, I naturally believed this to be fake. I felt certain that this was a deepfake video, made by someone who either wanted to sell me something, or scam me. But I was curious enough to look it up, and it actually turned out to be true. I’ve been a big fan of Jovovich ever since I first saw The Fifth Element, and even more so once she starred in the Resident Evil movies as Alice. The odds that the two of us had been working for almost the same thing for the past year is minuscule, which made the whole thing feel a bit like a bug in the simulation.
But it’s true, she actually has a Github, and as soon as I started looking into MemPalace I realized that this was the missing piece of the puzzle.
Teaching AI How to Think
By this time, my log file was no longer 400,000 characters, but over 7 million tokens (~28,000 words once curated) of organic problem solving, programming, building and discussions on topics such as philosophy, economy and politics. I could not have wished for a better and richer material to use in my work. After an initial test of the MemPalace I decided to start the grueling job of curating my logs from it’s original 200+ GB down to 7,000 highly structured files at a total of 70 MB.
My PC still doesn’t have the capacity to run the large and advanced models I wish it had, so I simultaniously tried to find a workaround. Since I’m an independant developer, I simply do not have the funds required to run Gemini and similar models on per token cost through API.
About a month ago, I thought I had the solution, but my first attempt was proven to be billed on my API key. My second attempt went through Antigravity CLI, and turned out was very restricted in the amount of tokens I was elegible for on my subscription. On my third attempt it seemed like I had finally bridged my subscription through the web app to my local computer and MemPalace. Unfortunately Google seems to have closed that door only days, or even hours after I found it. Any attempt to go that way have since resulted in the account getting auto-connected to Antigravity and its restrictions.
But thanks to Milla and Ben, I think I have finally created an artificial procedural and associative memory, suited for agentic AI. All the testing I have done so far looks very promising.
You can make sure you don’t miss any updates regarding the Procedural memory project as well as other interesting projects and guides, by signing up on my newsletter.
Note: I am operating from a sole proprietorship, and thus are available for work in most of the world. Both for shorter consulting and for longer projects.
Direct Inquiry: business@zanno.se
For more information, see Lab Inquiries
If you want to help keep the lab running, you can now contribute securely through Google payments.