Eight videos and forty rules

Quite a lot of what I know today I learned from YouTube videos, and at some point I asked myself whether my agents (programs built on language models that carry out specific tasks for me on their own) could learn the same way. I had a concrete reason: I have an automation that draws small pixel-art animations (images made of visible, square pixels, as in old games), and its results were correct but rather flat, so I wanted to improve them. After watching one video about pixel art on YouTube I decided that instead of writing hints for my agents by hand, maybe I could build a process that would do it by itself, without any input from me: the process would get a link to a video, a playlist or a whole channel, and would hand back a set of rules saved in a file that the agents could reach for every time they get a task in that field.

How the tool works: from a YouTube video to rules for an agent

So, working with Claude Code, I built a tool in which I only pick the videos (the process shows me the list together with the total running time of all the recordings before anything is downloaded), and the rest is done by agents and “ordinary” code. The subtitles of each video are downloaded, and if a video has none, the recording goes to a speech recognition model running on my home server. On top of that the process picks a few dozen frames, many of them from moments where the author says something like “as you can see here”, because in a drawing tutorial a lot of the knowledge is in the picture, not in the words. Then each video gets its own agent, which “watches” it (reads the transcript and looks through the frames) and writes out specific rules from it, each time marking the point in the video where the rule was stated, and at the end the agent running the whole process merges all the rules into one set. For this part I use a cloud model (Claude Opus 5.5 in Claude Code, a personal preference, not a recommendation), because careful reading of long transcripts and sorting a few dozen rules is the kind of task where I reach for frontier models, and they give me quite decent results at it.

The quote is checked by a program, not a model

Most of the work was not in extracting the knowledge itself but in answering the question of how I know that what the agent pulled out of a video is really said in it. A model reading a transcript can write a neat rule that the author did not say at all, and add an equally neat “quote” to it, which is why every rule in my solution must have a verbatim quote from the transcript and the time at which that passage was said, and an “ordinary” program (not a model) checks whether those words really are in the transcript and whether they come near the given moment. Before I ran the process on real videos, the program got two deliberately planted errors to check and rejected both: a quote that did not appear in the video at all, and another one, genuine but attributed to a spot eleven minutes away. There is one funny detail here: automatic subtitles sometimes mangle names (the drawing program Aseprite was written in them as “A sprite”). The agent is not allowed to correct such an error in a quote (because then the quote no longer matches the transcript), so the correction goes into the text of the rule, and the quote stays as it is in the subtitles.

A matching quote is not a matching meaning

During testing another problem came up that needed solving: it turned out that a quote that matches does not by itself guarantee that the rule reflects what the author conveyed in her recording. After the first run all the quotes passed the check, and yet the set of rules was firmer than the author herself: her advice to “consider” shading with a different color turned into a ban on using lighter and darker shades, and the word “usually” vanished from five rules. Nothing had been made up; the model was just dropping the caveats along the way while tidying the rules into short instructions, and in my view that is an error that is easy to miss, because each rule on its own sounds reasonable. Today every rule has a recorded strength taken from the author's words (a requirement, a default, one of several options, or a matter of taste). So only requirements and defaults make it onto the checklist the agents use most often, and at the end a separate agent (which cannot change anything) reads each rule next to its quote and judges whether the quote really supports it. In the first run that agent judged 7 of 19 rules well supported, and in the second, after the changes, 29 of 40. That is a clear improvement, but eleven rules still lacked enough support in their quotes, which is why this step stays for good.

The same happened with frames. Once a rule cited a frame that was supposed to illustrate it, and it turned out that the frame showed a flat side view in which what the author of the recording was talking about could not be seen at all, because the program only checked whether the frame file existed, not what was on it. It worked the other way too: elsewhere in a video a shadow under some lettering was called anti-aliasing in the recording, and only the frame showed that it was a shadow, and the agent recorded them as two separate things. Since then, when a rule cites a frame, the evidence in my setup is the image itself, and the words are only its description.

Eight videos, one source

The next lesson concerned the sources themselves. The first set came from eight videos from one YouTube channel (all of them recorded by the same creator), so a rule “confirmed in two sources” in fact meant one person who said the same thing twice, and the set says so plainly, at the very top. Of the seven videos the process handled after the first tutorial, six turned out to be essays in which pixel art was only a side topic (color psychology, fonts, one particular game): of the 57 claims extracted from them, roughly a third made it into the set, while the seventh video, in which the author corrects viewers' drawings, gave 28 claims and most of the new rules on the checklist. I concluded that choosing videos more carefully at the start will cost me less than sifting their content at the end.

The lighthouse before the rules and after

Did it achieve anything at all? In my case, the shading rules from the first tutorial worked. According to the most important of them, every material (fur, stone, water) should have three shades: shadow, base and light, and in my earlier animations it was plain as day, because the nicest of them (a cat on a cushion) had all three, while the flattest had a single shade for the stones or the sea. When the agents redrew a lighthouse, this time following those rules, a separate model comparing the images (asked twice, the second time with the images in reverse order) clearly picked the new version both times.

AI

Where a rule failed

Not every rule had such a good effect on the final result. A video about color gave rise to the rule of keeping one vivid color for the most important object only, and with the lighthouse it worked (the eye goes straight to the glowing lamp, and the image passed the test three times out of three: a separate model, with no context at all, had to guess what was in the image, and guessed right every time). With the jungle the same tip failed six rounds in a row, because whatever got the vivid color became the answer: the waterfall was taken for a sword or a fountain, the red flowers for foxes, and a shaft of light for a bee. So the rule was narrowed to single objects, and the set gained a note next to it saying that this is a conclusion from my own trials, not from the video. In fact even its wording was too firm: the video described how particular games guide the player's eye with color and gave no advice, so today the set records it only as one of several options. One rule I did not bring in at all: the tutorials spend a lot of time on a certain mistake in drawing lines, but the tool my agents draw with does not make it, by design, so checking for it would be work with no benefit.

AI

Training recordings as a source of rules

Pixel art is a deliberately light example here, because images make it easiest to show what works and what does not. The pattern itself seems useful to me beyond pixel art: many companies have a lot of knowledge captured in recordings (training sessions, webinars, workstation instructions recorded on a phone and so on), and in my view they could be used to prepare a set of rules for an agent that takes some of the load off employees in doing or checking part of their tasks, provided that every rule links to a specific minute of the recording, and that whether the recording really supports the rule is checked by an “ordinary” program and a separate reviewer, not by the same model that wrote the rule (with internal recordings there is also the question of where they may be processed, but that is a separate topic).

Automation you don't build just once

I don't want to leave the impression that automation is something you build once: you check that it works and move on to the next thing. That is not how it looks for me, and learning from videos shows it perhaps best of all. What I have today is a first step that already works well enough for some uses (with some animations the rules from the videos worked brilliantly, and with the jungle one of them failed), but forty rules from a single author are still a starting point, not expert knowledge. For my agents to draw better, the process needs more material: the same topic has to go through it many times, with videos by different authors, and the rules have to be tested on more tasks of my own, because only then can you see which of them work poorly or only in some cases.

My process does not create a new set each time; it adds the rules from new videos to the existing one: a rule confirmed by a second person gains strength, contradictions go to a separate section instead of silently overwriting what was already there, and rules that do not survive my trials are narrowed (like the one about the vivid color). So I treat it not as a finished tool that will give me expert knowledge in any field, but as an ongoing process that knows a little more with each run.

In short

• I built a process that pulls rules for my agents out of YouTube videos. I pick the videos, and the rest is done by agents and “ordinary” code.
• Every rule has a verbatim quote and a timestamp, and whether the quote really is in the video is checked by a program, not a model.
• A matching quote is not a matching meaning: while tidying the rules, the model dropped the author's caveats, so every rule has a recorded strength, and a separate reviewer reads it next to its quote.
• Eight videos by one author are one source, not eight, and the essays gave much less than a single video with concrete corrections.
• Some rules worked for me straight away, and one failed with a different kind of image, so I treat knowledge from a video as a starting point and test it on my own tasks.
• This is not something you build once: the same topic goes through the process many times, with videos by different authors, and only the later runs show which rules work poorly.