Eight out of ten for a lyric with a mistake

Several of my projects needed songs, and not one but many (in different styles and for different occasions), so from the start I assumed I would not make them one at a time but would build an automation that writes the lyric, records the track and checks by itself whether the result is any good. The recording turned out to be the smallest problem (the music model gets the lyric and a description of the style and returns a finished track in about a minute), while the lyric took the most work, and that is what this note is about.

The model writes the lyric and rates it itself

I built the first version on a pattern that works very well for me when generating images: the model writes the lyric and then, in a separate request, rates it itself in several categories (whether the rhymes are exact, whether the chorus comes back word for word, whether the lyric sticks to the facts I gave it), and the ratings are combined into one number: a lyric scoring below the threshold goes back for revision together with a list of its flaws, and one above the threshold goes to recording. Apart from starting it, I take no part in this: everything that happens inside is done by agents and ordinary code, and I come back only at the end to listen to the finished tracks.

Why the AI model got it wrong: it estimated instead of counting

With song lyrics, though, the same pattern started to fool me. At one point the process returned a lyric rated above eight out of ten (the combined result of three separate ratings from the same model, run with different amounts of thinking time), even though one word appeared twice in the same line, and all three ratings agreed that the line had nine syllables when it had eight. That was when I understood something that seems obvious to me today: a language model does not count on its own; it estimates, and it does so in a very confident voice.

AI

A check that always agrees

The second thing came up along the way and was worse, because it concerned the check itself. A model asked the same thing twice does not have to answer the same way, and how much its answers can differ is set by a single parameter (the so-called temperature), so I built the first panel on exactly that: three ratings of the same lyric with three different settings, so that each would look at it a little differently. One of the tools I was using to connect to the models at the time silently ignored that setting, so instead of three different views I would have gotten the same one three times, and the unanimity that would have looked like certainty would in fact have been an echo (which is why, with tools like that, the ratings now differ by thinking time). What I learned from it is that before I trust the ratings, I have to make sure the raters really rate independently of one another, because a check that always agrees is not a check.

If it can be counted, a program counts it

I rebuilt it on one principle: everything that can be counted is counted by an ordinary program, not by the model. So the program checks whether a word repeats within a line, how many syllables each line has, whether a rhyme is exact, whether the chorus comes back word for word, whether the lyric contains what it must (for example a brand name or the name of the person the song is for), and whether it contains nothing it should not. It is boring code that is quick to write, but it has one advantage over any model: you cannot talk it out of an answer, because it will not change its mind just because the rest of the lyric sounds nice, and it does not have better and worse days.

What cannot be counted

I left to the models what cannot be counted: whether the sentences sound like Polish (and not like a calque of English), whether the lyric can be sung at the given tempo, whether the joke works without winking at the listener, whether the chorus is catchy, and whether the lyric contradicts what I established earlier about the characters and their world. It is not done by a single “judge of everything”, though, but by several reviewers, each of whom gets one task and looks only at that, and on top of that none of them is allowed to edit the lyric, which matters more than it seems: a reviewer who offers their own version stops reviewing and starts competing, and instead of comments you get a second lyric from them.

AI

Every comment must also contain three things: the quoted line, one sentence on what is wrong with it, and a weight (whether the flaw blocks the lyric, whether it is worth fixing, or whether it is a matter of taste), because nothing follows from a comment like “the chorus could be stronger” and nobody knows what to do with it. Then the agent running the whole process goes through the comments one by one and decides for each whether to accept it, and for the rejected ones it writes down in one sentence why.

This step looks like a formality, but in my view it decides whether the whole thing works, which shows best in comments that contradict each other. In one of the songs a joke was too obvious, so the reviewer making sure the lyric does not wink at the listener demanded a change to the line, and after the change the check on the fix blocked it, because the line was correct but no longer carried anything. Both comments were right and the agent running the process had to settle it, and along the way it turned out that toning a joke down is not enough; it has to be replaced with another one (in the next round the line got its punchline back, this time without the wink). If the agent applied every comment without thinking, after a few rounds it would be left with a lyric that is correct and dead, because a fix that repairs a detail but spoils a good line is a loss, even if formally it is a fix.

A limit on revision rounds

The whole loop (fix, check, fix again) has a hard limit of six rounds in my setup and ends earlier if two rounds in a row bring no progress, that is, if new flaws appear as fast as old ones disappear. Without such a limit the loop might not end, because a reviewer almost always finds something, and the longer you ask, the smaller the things it picks on. The process also keeps the lyric from every round, because quality does not rise steadily with each fix (there were times when a fix removed some flaws and introduced new ones), so the best lyric may turn out to be from one of the earlier rounds rather than the last.

Pairwise comparison instead of scores

I also changed the way the best lyric is chosen, because the question “how many out of ten is this lyric” is a measurement with a different ruler every time (each rating is made separately, with no point of reference, and even for the same lyric it can vary by a few tenths of a point, which is enough to swap two lyrics of similar quality), while the question “which of these two is better” measures both lyrics with the same ruler. So the process compared lyrics in pairs, and each pair twice (the second time in reverse order, because a model can favor a lyric for its position alone): the same result both ways was a win, and a disagreement was a draw, that is, an honest signal that the order decided, not the lyric.

AI

Today the process does not name a winner at the end, but picks two or three clearly different versions that reached the final (each with one sentence on its weakest point), and usually records them straight away, and only at this point am I needed: I listen to the finished tracks and choose by ear, and that is how it should stay, because a lyric that seems weaker on the page can sound great when sung.

The test that lied

That leaves the recording itself, and here there is a trap that in my view threatens every automated quality check. After recording, a separate model listens to the finished track and writes down what it hears, and a program compares that with the lyric to check whether every line was actually sung. It sounds like the perfect test, except that this test could lie: in one batch it judged eight recordings out of eight to have nothing sung in them, although singing is clearly audible in each, and it had earlier rejected three of the four recordings I later picked by ear (and it was not a one-off, because the same error came back in later batches). Since then it does not reject anything on its own; it only confirms, and the one thing it catches reliably in my setup is a recording cut off halfway.

The same goes for automated scoring of recordings: once I rejected the two top-rated recordings out of thirty after listening to them, and the one that went forward was somewhere in the middle of the pack, because that number measured how evenly the chorus was sung, not whether you want to listen to it.

What it gave me and what it costs

After this rebuild the lyrics stopped surprising me in that one particular way: a repeated word, or a line one syllable longer or shorter, no longer gets through, because it is counted by a program, not by a model that could miss it. It costs more time and more requests to the model (the full set of reviewers, and then a check on every fix, is much more than a simple loop with a single rater), but I accept that, because a song is listened to as a whole and one jarring moment can spoil everything. For a task where a mistake is cheap and easy to fix, a check this elaborate would in my view be overkill.

I chose songs because it is a light example and the easiest one to show the whole mechanism on, but the same principles come back in far less showy tasks and boil down to five questions: how I know my test is not wrong, what in the result can be counted and what can only be judged, whether the countable part is counted by an ordinary program rather than the model, when the revision loop should end, and who makes the final decision. The first of them cost me the most time, though it looks the least important.

In short

• In my setup the model did not count; it estimated, so I handed everything countable (repetitions, lengths, meeting the requirements) to an ordinary program that cannot be talked out of an answer.
• Several narrow reviewers, each with one task, give me more than a single “judge of everything”, and none of them is allowed to edit the lyric under review, because then it stops reviewing and starts competing.
• A comment without a quoted passage is of little use, and comments applied wholesale can break as much as they fix.
• The revision loop has a limit in my setup, because a reviewer almost always finds something.
• Everything inside is done by agents and ordinary code, and a person is needed only to listen to the final versions.
• I test an automated quality check first on cases I know are good (in my setup it managed to reject everything that was fine).