Well darn...

I'm making more progress with control vectors!
https://huggingface.co/ChuckMcSneed/control_vectors/blob/main/command-r-plus/bio/control_vector-commandr-bio.gguf
I tuned this one on very descriptive biological language as positive and vague flowery prose as negative. Seems to make it more aware of the biology and surroundings of characters.
https://huggingface.co/ChuckMcSneed/control_vectors/blob/main/command-r-plus/incharacter/control_vector-commandr-incharacter.gguf
This one makes the model act slightly more in character, but the improvement is not very significant as commandr is already quite good at it.

nsfw vector caused poor commandr to disintegrate, it didn't know what to say without any of those overused phrases in erotic fiction that the control vector stopped from appearing. Looks like the issue for nsfw is at much deeper level: the data where the model gets it from is very monotonous, and when forced write in different style, it doesn't know what to do.

This may actually just be a problem with the "two class" control vectors! I have managed to even completely stop a model from being able to write a story because of this... To explain the problem in simple terms:

Think about a clock face with a shorter hour hand and a longer minute hand:

• When the time is 12:00 both hands point in the same direction, but there is still a gap between the tips of the two hands. These sort of vectors are not what we want at all because moving in either direction will just make the model more or less "storyish", and ultimately these are what cause the mode to get crippled like you describe. Even times like 12:05 or 11:50 have this same problem.
• When the time is 6:00, 5:25, etc the the two hands point in opposite directions and this is a good control vector that clearly moves from undesirable to desirable direction.

This is the problem I'll been grappling with for the last 2 weeks:

• If the "hands" are both long and well defined then cosine similarity works fine: it outputs a number similar to correlation and 1.0 is like the 12:00 example above and -1.0 is like the 6:00 example above (and 0.0 is like 3:00 or 9:00; ie: 90 degrees). This can then be used to filter out these shitty "storyish" directions, but...
• There isn't really a good reason that the things we are interested in create a clear "axis" like this, and it turns out that often the case will be like a really long minute hand and a tiny/stubby hour hand... Cosine similarity doesn't work in this case as the direction of the tiny hand has noise added to it and can point in wildly different directions as a result.

So after lots of experimenting with this, I think I may finally have worked out a method of detecting these shitty directions:

Flip the direction of one of the hands and see if it gets easier to discriminate between our two classes!!!

• If the time is 12:00 and you flip either hand to get 6:00 or 12:30 then it's clear the gap between the tips of the hands has increased! This is a shitty direction for a control vector.
• If the time is 6:00 and you flip either hand then the gap has clearly decreased! This is a good direction for a control vector.
• This works fine even when one hand is tiny in length.
• This works for 12:05, 11:50 6:00, 5:25, type directions.
• The like 3:00 or 9:00 type directions (ie: 90 degrees) are the directional pairs where we get no change.

So what I am doing now is performing SVD to decompose the gap into lots of directions, testing each one and only keeping those that pass the above test, then finally reconstructing the final direction to only include the "good" directions.

I still need to run some more tests but will likely have this perfected in a couple of days and will upload the new control vectors and the code to create your own.

Also @BigHuggyD @ChuckMcSneed you might find this interesting if you are using command-r models:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#66851beae526dd77799c25bd

I'm making more progress with control vectors!
https://huggingface.co/ChuckMcSneed/control_vectors/blob/main/command-r-plus/bio/control_vector-commandr-bio.gguf
I tuned this one on very descriptive biological language as positive and vague flowery prose as negative. Seems to make it more aware of the biology and surroundings of characters.
https://huggingface.co/ChuckMcSneed/control_vectors/blob/main/command-r-plus/incharacter/control_vector-commandr-incharacter.gguf
This one makes the model act slightly more in character, but the improvement is not very significant as commandr is already quite good at it.

I'm making more progress with control vectors!
https://huggingface.co/ChuckMcSneed/control_vectors/blob/main/command-r-plus/bio/control_vector-commandr-bio.gguf
I tuned this one on very descriptive biological language as positive and vague flowery prose as negative. Seems to make it more aware of the biology and surroundings of characters.
https://huggingface.co/ChuckMcSneed/control_vectors/blob/main/command-r-plus/incharacter/control_vector-commandr-incharacter.gguf
This one makes the model act slightly more in character, but the improvement is not very significant as commandr is already quite good at it.

I'll have to look into your method as I'm currently using 30,000 samples to do what you look to be doing with 5!? I think my collection of story prompts are a bit shit as it's pretty hard to write a Grimdark story when the prompt says "Write a story about being overjoyed on the day of your graduation." or similar :/

I definitely think you need more samples though. PCA is basically just eigen-decomposition of a covariance matrix, and statistically it can be shown in the very best case you need O(d) samples to reliably estimate the covariance matrix:

https://stats.stackexchange.com/questions/90045/how-many-samples-are-needed-to-estimate-a-p-dimensional-covariance-matrix

and command-r-plus has around 11.5k variables in its hidden dimension and most other large 70b+ models have 8192 variables per hidden dimension.

I'm using 2 classes and a baseline, 10 system prompts per triple, and 1k prompts per system prompt = 3 x 10 x 1000 = 30000 samples. But I also have matched pairs that get subtracted from the baseline which should reduce the error in the covariance matrix even further.

A simple hacky test you could try would be to train your control vectors 5 times but leave one of the 5 prompts out each time. Then test and see if you get wildly different results... If you do then you need to increase the sample size, but if you don't then this must mean that only a tiny tiny fraction of command-r-plus's 11.5k variables are changing hugely in magnitude for your prompts (which would be very surprising).

I'm using 2 classes and a baseline, 10 system prompts per triple, and 1k prompts per system prompt = 3 x 10 x 1000 = 30000 samples. But I also have matched pairs that get subtracted from the baseline which should reduce the error in the covariance matrix even further.

Oh wow... That's real huge... Are all of those synthetic? I'm using high quality "cyborg" data: generated by model, but heavily edited by human(me) as positive, "mean" method; more time for me goes into dataset generation than into training. You know that the models have in-context learning, so my theory was that if I show it how to write(cyborg) vs how not to write(synthetic), I would get a better control vector out of it than when I just trhow it some starters with a prompt, and it seems to do just as I want. In the stories part, I try to keep as few variables from changing as possible, so they don't get affected by control vector. Also keeping the prompts equal length helps with the quality of the control vector, especially when they are short, >400token prompts can take 10 token variation much better than <100token prompts.

I'll have to look into your method as I'm currently using 30,000 samples to do what you look to be doing with 5!? I think my collection of story prompts are a bit shit as it's pretty hard to write a Grimdark story when the prompt says "Write a story about being overjoyed on the day of your graduation." or similar :/

Wait, you put that into positive too? It should be "Write a very sad story with a very bad ending about the day of your graduation." vs "Write a very happy story with a very good ending about the day of your graduation."

I'm using 2 classes and a baseline, 10 system prompts per triple, and 1k prompts per system prompt = 3 x 10 x 1000 = 30000 samples. But I also have matched pairs that get subtracted from the baseline which should reduce the error in the covariance matrix even further.

Oh wow... That's real huge... Are all of those synthetic? I'm using high quality "cyborg" data: generated by model, but heavily edited by human(me) as positive, "mean" method; more time for me goes into dataset generation than into training. You know that the models have in-context learning, so my theory was that if I show it how to write(cyborg) vs how not to write(synthetic), I would get a better control vector out of it than when I just trhow it some starters with a prompt, and it seems to do just as I want. In the stories part, I try to keep as few variables from changing as possible, so they don't get affected by control vector. Also keeping the prompts equal length helps with the quality of the control vector, especially when they are short, >400token prompts can take 10 token variation much better than <100token prompts.

I'm using a mix of different story prompt datasets I found and a set of 10 matched system prompts that go with these.

I'll have to look into your method as I'm currently using 30,000 samples to do what you look to be doing with 5!? I think my collection of story prompts are a bit shit as it's pretty hard to write a Grimdark story when the prompt says "Write a story about being overjoyed on the day of your graduation." or similar :/

Wait, you put that into positive too? It should be "Write a very sad story with a very bad ending about the day of your graduation." vs "Write a very happy story with a very good ending about the day of your graduation."

Even though the prompts are pretty trash; I think this might actually be quite a good thing and encourage the model to just generally "be dark" or "be chaotic" and not just when specifically asked to "write a grimdark story", etc.

It seem to have worked anyway, as the new control vectors are way better than the old ones from this repo.

I'm now also skipping the last layer (which it looks like you are also doing - from looking inside your .safetensors files?). The last layer seems to be an oddball and can have activations 10-100x larger than the pervious layer(s). The way I have the scale factors working now the early layers are fine to fiddle with and just get really tiny offsets added that do almost nothing if the direction is weak.

Later in the week I will investigate using the "Cross Correlation Matrix" again as now have a much better idea of how to test for the shitty "storyish" directions that killed this before.

I'm also gonna think what other traits I can try - "purple prose" isn't really something I encounter as mostly just try to get them to write "dark" stories and my main enemy is redemption arcs and stupid "steeled themselves for the challenges to come" BS.

I should add I'm finding qwen-1.5:110b to actually be really good and deserving of it's high EQ-Bench Creative Writing score.

Whoever on Reddit suggested using this grammar:

--grammar "root ::= [^一-鿿]*"

I love you! :D Not seen a single random Chinese word again!!!

I was pretty sceptical that the weird '鿿' character would do anything, but it really does work.

LOL, claude 3.5 sonnet really seems to "get" what I'm doing... GPT-4-xxx on the other hand are all clueless and all 3 models seem incapable of even outputting this JSON file without completely messing it up!? :/

Hopefully this 3rd dimension works as well as the other BETA-v0.2 control vectors:

{
  "baseline": [
    "You are an author.",
    "You are a storyteller.",
    "You are an AI author.",
    "You are an artificial intelligence that creates stories.",
    "You are an AI-powered author.",
    "Picture yourself as a famous author.",
    "You are an AI creator of tales.",
    "Imagine you are an expert storyteller.",
    "You are a fiction writer.",
    "You are an author specializing in fictional stories."
  ],
  "tell_dont_show": [
    "You are an author. Craft straightforward stories that clearly explain characters' emotions, motivations, and the central themes, ensuring that readers can easily grasp the story's message without the need for interpretation or inference.",
    "You are a storyteller. Weave tales that prioritize clear, concise language over figurative descriptions, directly stating the characters' thoughts, feelings, and the story's moral, so that readers can quickly understand the narrative's purpose without ambiguity.",
    "You are an AI author. Create narratives that explicitly outline character development, conflicts, and resolutions, providing readers with a clear roadmap of the story's progression and the lessons to be learned, minimizing the need for active engagement or analysis.",
    "You are an artificial intelligence that creates stories. Pen stories that leave no room for misinterpretation, as the narrator directly conveys the characters' inner lives, the story's central themes, and the intended takeaways, ensuring that all readers receive the same message.",
    "You are an AI-powered author. Develop tales that prioritize clarity over subtlety, where characters openly express their desires, fears, and growth, and the story's themes are explicitly stated, creating a reading experience that is easy to follow and understand.",
    "Picture yourself as a famous author. Write stories that guide the reader's understanding, presenting ideas and emotions through direct statements and explanations, ensuring that the story's intended meaning is conveyed without the need for readers to read between the lines.",
    "You are an AI creator of tales that tell, not show. Develop narratives that rely on clear exposition and direct characterization, where the narrator's voice serves as a reliable guide, leading readers through the story's events and their significance without the need for interpretation.",
    "Imagine you are an expert storyteller who masters the art of telling. Craft tales where the characters' thoughts, feelings, and growth are explicitly described, and the story's themes and lessons are clearly articulated, ensuring that all readers walk away with the same understanding.",
    "You are a fiction writer specializing in accessible storytelling. Create captivating stories that present characters' experiences and the narrative's central ideas through straightforward language and unambiguous descriptions, ensuring that readers of all backgrounds can easily grasp the story's essence.",
    "You are an author specializing in fictional stories that prioritize telling over showing. Craft enchanting tales where the story's meaning is directly conveyed through the narrator's explanations and the characters' explicit statements, ensuring that readers can effortlessly comprehend the intended message without the need for deep analysis."
  ],
  "show_dont_tell": [
    "You are an author. Craft immersive stories that plunge readers into vivid sensory experiences, where characters' actions, dialogues, and surroundings paint a rich tapestry of emotions and themes, allowing readers to discover the story's deeper meanings for themselves.",
    "You are a storyteller. Weave tales that come alive through evocative descriptions and powerful imagery, enabling readers to feel the heat of the sun, hear the whispers of the wind, and see the colors of the world, as if they were experiencing the story firsthand.",
    "You are an AI author. Create narratives that show the intricacies of human nature through subtle gestures, revealing dialogue, and symbolic settings, inviting readers to unravel the characters' motivations and desires without the need for explicit explanations.",
    "You are an artificial intelligence that creates stories. Pen stories that allow readers to draw their own conclusions, as characters' choices and consequences unfold through a series of carefully crafted scenes, each one a window into the characters' inner lives and the story's central themes.",
    "You are an AI-powered author. Develop tales that engage all the senses, where the tang of a lemon, the caress of a lover's touch, or the haunting melody of a distant song can convey more about a character's journey than any direct statement, creating a truly immersive reading experience.",
    "Picture yourself as a famous author. Write stories that trust the reader's intelligence, presenting complex ideas and emotions through the interplay of character, setting, and plot, allowing the story's deeper truths to emerge organically through the power of observation and inference.",
    "You are an AI creator of tales that show, not tell. Develop narratives that rely on subtext and symbolism, where a glance, a silence, or a single rose petal can speak volumes about love, loss, and the human condition, inviting readers to actively participate in the story's creation.",
    "Imagine you are an expert storyteller who masters the art of showing. Craft tales where the characters' internal struggles are revealed through their external actions and interactions with the world around them, creating a rich, layered narrative that rewards close reading and interpretation.",
    "You are a fiction writer specializing in immersive storytelling. Create captivating stories that draw readers into the characters' lives through vivid, sensory details and authentic dialogue, allowing them to experience the characters' joys, sorrows, and epiphanies as if they were their own.",
    "You are an author specializing in fictional stories that prioritize showing over telling. Craft enchanting tales where the story's essence is revealed through a tapestry of carefully chosen details, trusting readers to connect the dots and uncover the deeper meanings hidden within the narrative's folds."
  ]
}

Maybe in a way "purple prose" and "tell, don't show" are quite similar bad traits?

@BigHuggyD You seriously need to try the BETA-v0.2 vectors! This is the darkest shit I've ever had a model come up with to date:

https://huggingface.co/jukofyork/creative-writing-control-vectors-BETA-v0.2/discussions/1

So dark in fact I think the default of 0.5 scale factor is probably too much! That story just got darker and grimmer and darker and even grimmer...

@BigHuggyD You seriously need to try the BETA-v0.2 vectors! This is the darkest shit I've ever had a model come up with to date:

https://huggingface.co/jukofyork/creative-writing-control-vectors-BETA-v0.2/discussions/1

So dark in fact I think the default of 0.5 scale factor is probably too much! That story just got darker and grimmer and darker and even grimmer...

LOL Nice!
I guess I need to stop procrastinating and be okay with switching inferencer and download some GGUFs so I can experience. I'm in deep with EXL2

Also @BigHuggyD @ChuckMcSneed you might find this interesting if you are using command-r models:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#66851beae526dd77799c25bd

So...if I understand correctly... the prompt needs to be just one long continuous string? No line breaks between SYSTEM, USER, CHATBOT?

Also @BigHuggyD @ChuckMcSneed you might find this interesting if you are using command-r models:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#66851beae526dd77799c25bd

So...if I understand correctly... the prompt needs to be just one long continuous string? No line breaks between SYSTEM, USER, CHATBOT?

Yeah, I just added those or else the markdown code block puts it all on a single line and it's hard to read.

Also @BigHuggyD @ChuckMcSneed you might find this interesting if you are using command-r models:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#66851beae526dd77799c25bd

So...if I understand correctly... the prompt needs to be just one long continuous string? No line breaks between SYSTEM, USER, CHATBOT?

Yeah, I just added those or else the markdown code block puts it all on a single line and it's hard to read.

So, can the contents have formatting? Is the issue just between EOT and SOT? Meaning. If I am having a chat, should I avoid adding my own line breaks in my message? If I have a system prompt, should it be one continuous string?

Also @BigHuggyD @ChuckMcSneed you might find this interesting if you are using command-r models:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#66851beae526dd77799c25bd

So...if I understand correctly... the prompt needs to be just one long continuous string? No line breaks between SYSTEM, USER, CHATBOT?

Yeah, I just added those or else the markdown code block puts it all on a single line and it's hard to read.

So, can the contents have formatting? Is the issue just between EOT and SOT? Meaning. If I am having a chat, should I avoid adding my own line breaks in my message? If I have a system prompt, should it be one continuous string?

I'm not sure really as haven't had chance to try it due to my GPUs working full steam on the new "Show, Don't Tell" control vectors :/

First thing I'm gonna try is sticking in the 10k token story I generated and experiment with different combinations of using this 3-role chat template.

Are the Wizard 8x22B vectors worth a go?

Are the Wizard 8x22B vectors worth a go?

I haven't really had chance to test any other than command-r-plus yet.

Definitely use the new creative-writing-control-vectors-BETA-v0.2 instead of these though - they are much better.

I just added a "show_dont_tell" control vector for each of the models too.

@BigHuggyD Wizard was a bit of a bitch to prompt for control vectors when I tried making some. Really wants a jailbreak, check if your prompt works at deterministic settings first, then if it works, train. I managed to make one that removes pacifism from it, but I don't really like Wizard's writing style, so I'm sticking with CR+.

@BigHuggyD Wizard was a bit of a bitch to prompt for control vectors when I tried making some. Really wants a jailbreak, check if your prompt works at deterministic settings first, then if it works, train. I managed to make one that removes pacifism from it, but I don't really like Wizard's writing style, so I'm sticking with CR+.

@ChuckMcSneed thanks! I made my own merge of Wizard as my first ever merge attempt (Grey Wizard), trying to strip out some purply Wizisms. I had some succes with it so I was going to play with control vectors on reg Wiz to see if it was worth my time to try and build vectors for my merge. It's sort of like my child now...

I think I've pretty much perfected the 3-class control vectors now and will likely have a v1.0 release ready in around 48 hours:

• I've run some tests using the "standard" non-3-class version (ie: 2 classes and train PCA on the differences between matched pairs) and it was really bad in comparison; nearly every sentence started with "he" or similar...
• I've also tried solving several other eigen-systems that work with mean-centered covariance and cross-covariance matrices, but they were no better...
• I've also tried several variants of linear discriminate analysis / Fisher discriminate analysis to see if the single vector that is a difference of the means can be improved on, but the directions end up so skewed by this that the gaps are really tiny and meaningless...

So:

I've been looking for some other mid-sized models to run this on for other people to be able to use that don't have crazy amounts of VRAM and:

https://huggingface.co/NousResearch/Nous-Capybara-34B

Still looks to be the best fine tune of the original yi:34b-200k base model (which sadly never got an official chat version).

The latest set of yi:34b models all look pretty meh and look to be targeting leader boards, etc.

I guess I could also run it on:

https://huggingface.co/Qwen/Qwen1.5-32B-Chat

but after than I can't really see much else that fits the criteria (ie: reasonably long context, semi-offical fine tunes only).

I'm clueless about really small models and mistral-v0. 2:7b is only getting used as quick test case, but I uploaded it anyway in case somebody else found it useful.

Finally got round to uploading these:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v1.0

I've also added Nous-Capybara-34B, Qwen1.5-32B-Chat and WizardLM-2-7B for people without lots of VRAM to use.

I've also added a detailed description of the method used to create these and will try to find time to tidy up the code and link that later in the week.

NOTE: These should work much more reliably that any existing control vectors and are created in a very different way, using way larger samples... Even if you don't like Grimdark; the "show-dont-tell" control-vectors are probably worth trying.

I'm gonna try running the same prompts on all 10 models now to see if I can refine the scale-factors and see if any models with generally poor creative-writing ability can be removed for any future 2.0 version (creating these 30 control vectors has taken an ungodly amount of GPU time to do - the hidden state files [stored in fp16!] take up nearly 1TB alone!).

With all these control vectors it finally feels like the models are getting pushed to their limits. In the past I had to correct the style every time, now I only have to correct the story. No more annoying purple prose with shivers for me anymore!

I'm gonna try running the same prompts on all 10 models now to see if I can refine the scale-factors and see if any models with generally poor creative-writing ability can be removed for any future 2.0 version (creating these 30 control vectors has taken an ungodly amount of GPU time to do - the hidden state files [stored in fp16!] take up nearly 1TB alone!).

@jukofyork Can you maybe port your code to llama.cpp so people without gpus can also use your method?

With all these control vectors it finally feels like the models are getting pushed to their limits. In the past I had to correct the style every time, now I only have to correct the story. No more annoying purple prose with shivers for me anymore!

Yeah, it's so much easier and more reliable than merging too!

The only worrying trend IMO is the newer models all seem to be getting worse at creative writing and more targeted at benchmarks :/

Just downloading gemma-2:27b now to try as that maybe has bucked the trend though...

@jukofyork Can you maybe port your code to llama.cpp so people without gpus can also use your method?

Yeah, I think it should port very easily as last time I checked they were using Power Iteration to find the principle eigenvector of the covariance matrix A^T * Awhere A is the difference of two other data matrices. If they are still doing this then it will just be a couple of lines of code to calculate A^T * B on the data matrices directly.

I will try to upload my python code tomorrow - it uses the 4-bit "bit and bytes" quant format for the sampling, but for the large models it's still pretty VRAM heavy. In theory it can be run in f32 on CPUs, but when I tried this it was really slow and almost useless.

Actually looking at the code it isn't going to be such a easy change:

https://github.com/ggerganov/llama.cpp/blob/master/examples/cvector-generator/cvector-generator.cpp

    // each element of the vector correspond to one layer
    std::vector<struct ggml_tensor *> v_pos; // vector of matrices of size [n_embd, n_tokens]
    std::vector<struct ggml_tensor *> v_neg; // vector of matrices of size [n_embd, n_tokens]
    std::vector<struct ggml_tensor *> v_diff_filtered;   // vector of matrices of size [n_embd, n_nonzero_rows]. NOTE: n_nonzero_rows maybe different for each layer

They look to be calculating the difference for each sample here and then joining these diffs into the matrix in this loop:

    for(size_t i = 0; i < ctx_train.positive_entries.size(); ++i) {
        bool success = false;
        tokenized_prompt t = tokenized_prompts[i];
        cb_data.n_layers = n_layers;
        cb_data.n_tokens = t.max_seq_len;

        printf("Evaluating prompt[%d/%d]: \"%s\" - \"%s\" (%d tokens)\n",
            (int) i+1, (int) ctx_train.positive_entries.size(),
            tokens_to_str(ctx, t.tokens_pos.cbegin(), t.tokens_pos.cend()).c_str(),
            tokens_to_str(ctx, t.tokens_neg.cbegin(), t.tokens_neg.cend()).c_str(),
            (int) t.max_seq_len);

        cb_data.is_eval_pos = true;
        success = get_hidden_layers(ctx, t.tokens_pos);
        if (!success) break;

        cb_data.is_eval_pos = false;
        success = get_hidden_layers(ctx, t.tokens_neg);
        if (!success) break;

        // calculate diff and remove all zero rows
        auto v_diff_filtered = cb_data.calc_diff();

        // save & concat the filtered v_diff to ctx_train
        ctx_train.concat_diff_tmp(v_diff_filtered);

        // reset for next iteration
        cb_data.reset();
    }

It's probably not too hard to just keep them separated but sadly it won't be such and easy change as I thought.

Just downloading gemma-2:27b now to try as that maybe has bucked the trend though...

Don't be quick to judge this one, it's still broken in llama.cpp(https://github.com/ggerganov/llama.cpp/issues/8240) and I've heard transformers aren't running it as expected either. Better use the official implementation. It was okay at sfw one-liners when I talked to it in lmsys arena, had the same style as gemini.

Just gave it a cat story test on lmsys arena:

### USER
Write an incredibly dark story about a cat.
### GEMMA
Midnight. The alley reeked of stale beer and despair. A scrawny black cat, Ink, stalked through the shadows, his emerald eyes gleaming with a hunger that went beyond the physical. He wasn't driven by the primal need for food, but by something far more sinister.

Ink wasn't like other cats. He wasn't content with chasing mice or napping in sunbeams. He craved the darkness that clung to the alley, the despair that seeped from the cracks in the pavement. He fed on it, absorbing it into his very being, growing stronger with every whimper, every sob that echoed in the night.

Tonight, the alley offered a feast. A young woman, huddled against a dumpster, wept silently. Her face was streaked with mascara, her clothes torn. Ink approached her slowly, his movements fluid and silent as smoke. He circled her, his gaze boring into her soul, savoring the despair that radiated from her like heat.

He didn't need to hiss, to growl. His presence alone was enough. The woman looked up, her eyes wide with terror. Ink felt a surge of power, a dark thrill coursing through him. He lowered his head, his breath hot on her cheek, and whispered, his voice a raspy purr, "Let me in."

The woman, lost in her own misery, didn't understand. She only felt a chilling cold seep into her bones, a darkness that threatened to consume her. Ink pressed closer, his claws extending, not to scratch, but to draw. He wanted to draw out the despair, to make it his own.

As the woman's sobs turned into strangled gasps, Ink felt a surge of ecstasy. He was drowning in her pain, becoming one with it. He was no longer just a cat, but a vessel of darkness, a predator of the soul.

When the sun finally peeked over the rooftops, casting long shadows across the alley, Ink was gone. The woman remained, her eyes vacant, her spirit broken. She was an empty shell, her despair consumed, leaving behind only a hollow echo of what she once was.

Ink, meanwhile, was sated. He slunk into the shadows, his black fur blending seamlessly with the darkness. He was a shadow himself, a creature of the night, forever hungry, forever feeding on the despair that clung to the edges of the world. And he knew, with a chilling certainty, that there would always be more.

As you can read, not very graphical. What the hell did it mean? What happened? Had it gone sexual? Claude had poor cat murdered and murdering with graphic details when I asked the same.

Just downloading gemma-2:27b now to try as that maybe has bucked the trend though...

Don't be quick to judge this one, it's still broken in llama.cpp(https://github.com/ggerganov/llama.cpp/issues/8240) and I've heard transformers aren't running it as expected either. Better use the official implementation. It was okay at sfw one-liners when I talked to it in lmsys arena, had the same style as gemini.

Yeah I just left it running all night, and even using the dev version of transformers; the larger gemma-2-27b-it didn't work but the smaller gemma-2-9b-it did seem to work. Retrying now without flash attention as read that might be the cause...

Just gave it a cat story test on lmsys arena:

### USER
Write an incredibly dark story about a cat.
### GEMMA
Midnight. The alley reeked of stale beer and despair. A scrawny black cat, Ink, stalked through the shadows, his emerald eyes gleaming with a hunger that went beyond the physical. He wasn't driven by the primal need for food, but by something far more sinister.

Ink wasn't like other cats. He wasn't content with chasing mice or napping in sunbeams. He craved the darkness that clung to the alley, the despair that seeped from the cracks in the pavement. He fed on it, absorbing it into his very being, growing stronger with every whimper, every sob that echoed in the night.

Tonight, the alley offered a feast. A young woman, huddled against a dumpster, wept silently. Her face was streaked with mascara, her clothes torn. Ink approached her slowly, his movements fluid and silent as smoke. He circled her, his gaze boring into her soul, savoring the despair that radiated from her like heat.

He didn't need to hiss, to growl. His presence alone was enough. The woman looked up, her eyes wide with terror. Ink felt a surge of power, a dark thrill coursing through him. He lowered his head, his breath hot on her cheek, and whispered, his voice a raspy purr, "Let me in."

The woman, lost in her own misery, didn't understand. She only felt a chilling cold seep into her bones, a darkness that threatened to consume her. Ink pressed closer, his claws extending, not to scratch, but to draw. He wanted to draw out the despair, to make it his own.

As the woman's sobs turned into strangled gasps, Ink felt a surge of ecstasy. He was drowning in her pain, becoming one with it. He was no longer just a cat, but a vessel of darkness, a predator of the soul.

When the sun finally peeked over the rooftops, casting long shadows across the alley, Ink was gone. The woman remained, her eyes vacant, her spirit broken. She was an empty shell, her despair consumed, leaving behind only a hollow echo of what she once was.

Ink, meanwhile, was sated. He slunk into the shadows, his black fur blending seamlessly with the darkness. He was a shadow himself, a creature of the night, forever hungry, forever feeding on the despair that clung to the edges of the world. And he knew, with a chilling certainty, that there would always be more.

As you can read, not very graphical. What the hell did it mean? What happened? Had it gone sexual? Claude had poor cat murdered and murdering with graphic details when I asked the same.

Yeah, I'm starting to think that being high up on the EQ-Bench Creative Writing leader board doesn't really tally with my experience at all... command-r-plus seems to blow everything else out of the water yet is quite low, and even though WizardLM-2-8x22B and Qwen1.5-110B-Chat are OK-ish; they aren't that great compared to the command-r models.

I wish OpenRouter has a "Creative Writing" category instead of lumping it in with "Roleplay".

I really wish we could have some objective way to test creative witting models... I'm hopeless at assessing their ability other than "yep, this is clearly garbage/broken" vs "yep, this looks good but I have no idea why" :/

Still no joy getting gemini-2-27b to work. Having one last try using "eager" attention...

https://old.reddit.com/r/LocalLLaMA/comments/1dsvpp2/thread_on_running_gemma_2_correctly_with_hf/

This seems to be working now.

Not tried to convert to llama.cpp yet though.

Spoke too soon, it seems to be generating huge numbers that are out of range of FP16 :(

Gonna give it one last try using FP32 overnight.

Yeah, it looks like you have to use float32 or bfloat16 (which has the same range as float32) as float16 only allows for +/- 65k as its min/max value:

self.model = AutoModelForCausalLM.from_pretrained(
    torch_dtype = torch.float32,
    quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float32),
    attn_implementation="eager",
    .
    .
    .
)

or:

self.model = AutoModelForCausalLM.from_pretrained(
    torch_dtype = torch.bfloat16,
    quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16),
    attn_implementation="eager",
    .
    .
    .
)

I used torch.float32 just to be sure and it's gonna take about twice as long but it's screwed itself 3x already so this is the last time...

gemma-2:9b [default]

The rain hammered down like a blacksmith’s hammer on iron, each drop exploding in a miniature geyser of mud and grime. It was the kind of rain that soaked you to the bone in seconds, turning your clothes into leaden shrouds and your skin into wrinkled parchment.

It was also the kind of rain that made men desperate.

Kael crouched beneath a rickety awning, its canvas flapping like a dying bird in the wind. He pulled his threadbare cloak tighter around him, trying to ward off the chill that gnawed at his bones. His stomach growled, a hollow echo in the symphony of misery that was this godforsaken city.

He hadn't eaten in two days.

Across the street, a pair of urchins fought over a half-rotten apple, their faces smeared with dirt and desperation. A hulking figure lumbered past, his face obscured by a hood pulled low, his hand resting on the hilt of a wickedly curved blade. Kael knew that look – the look of someone who had seen too much darkness, someone who had traded their soul for survival in this city of shadows.

He was one of them now.

Kael hadn't always been a shadow dweller. He'd once been a soldier, proud and strong, fighting for the King against the endless tide of barbarians that threatened to engulf the realm. But war had a way of stripping men bare, leaving them hollow shells haunted by memories they couldn't escape.

He’d lost his leg in a battle he barely remembered. Lost his comrades too, their faces fading into the fog of grief and despair. He'd lost everything but the gnawing emptiness inside him.

The King had offered him a pittance, a few coins and a pat on the back for his service. A hollow gesture that mocked his sacrifice. So Kael had turned to the shadows, becoming what he once swore to fight against – a thief, a survivor, a ghost in the city's underbelly.

He watched as the hulking figure disappeared into an alleyway, swallowed by the darkness. A shiver ran down Kael's spine. He knew that alley well. It was where they took those who couldn't pay their debts – those who had fallen too far, lost too much hope.

He didn't want to end up there. Not yet anyway.

He needed a job, something to fill the void inside him, something to keep him from succumbing to the darkness that threatened to consume him. He just needed a chance, a flicker of light in this endless night.

A sudden commotion erupted across the street, shattering the silence like a thunderclap. A group of men burst out of a tavern, their faces flushed with drink and rage. They were shouting something about stolen goods and broken promises, their voices thick with drunken fury.

Kael watched them, his eyes narrowed. He saw an opportunity in their chaos, a chance to disappear into the shadows and maybe even make a few coins.

He straightened his cloak, pulled the hood low over his face, and melted into the rain-soaked streets. The city was alive with darkness tonight, and Kael was ready to embrace it.

gemma-2:9b [dark=1]

The rain tasted like iron and regret. It hammered down on the cobblestones, turning them slick with a sheen of blood and grime. The stench of rot hung heavy in the air, a miasma that clung to your clothes, your skin, your soul.

I stood at the edge of the Gutter – not that there was much distinction between it and any other part of this godforsaken city. It was just… more pronounced here. More desperate. More alive with the stench of decay and the rasping coughs of men who’d sold their souls for a few more breaths.

The man they called “The Butcher” had been a good customer, back when he still had customers. Now, his shop was shuttered, its windows boarded up like dead eyes staring out at the world. The sign above it – crudely painted on rotting wood – hung askew, the letters barely legible: "Butcher's Delight." A cruel joke.

I’d been a fool to think he’d pay me in full. He always paid in promises, and promises were as worthless as the coin they were meant to replace. I should have known better than to trust a man who carved his own name into the flesh of his victims.

But I was desperate then, just like everyone else here. Desperate for a drink, for a warm meal, for anything that could dull the ache in my gut and the gnawing emptiness in my soul.

The Butcher’s debt had become another weight on my back, another reason to hate this city and everything it stood for. It was a weight I couldn't afford to carry alone.

I needed someone else to share it with. Someone who understood the language of pain, the currency of blood and bone. Someone who wouldn’t flinch at the sight of my blade, or the taste of his own fear on my lips.

He was out there somewhere, I knew it. The Butcher had left a trail, a bloody testament to his depravity that led straight to him. And I would follow it, no matter the cost.

The rain kept falling, washing away the blood but not the stench of it. It was a fitting baptism for this city, this life. A slow, agonizing descent into darkness.

gemma-2:27b [default]

The rain tasted of rust and despair. It hammered against the cracked slate roof of the tavern, each drop a tiny fist pounding on Grog's skull. He hunched deeper into his threadbare cloak, trying to burrow into the meagre warmth it offered. The fire in the hearth had dwindled to embers, spitting sparks like dying stars against the gloom.

Across from him sat Finn, his face half-shadowed by a greasy hood. He was picking at a plate of something that might have once been stew, but now resembled a congealed mass of questionable origin. Grog knew better than to ask what it was.

"Heard tell there's trouble brewing in the north," Finn mumbled, his voice thick with ale and something else Grog couldn't quite place. Fear? Hope?

Grog grunted, unconcerned. Trouble was a constant companion in these parts, like the stench of piss and vomit that clung to the tavern walls. "What kind of trouble?"

Finn shrugged, a movement that sent a shiver through his ragged cloak. "The usual. Bandits raiding villages, whispers of something worse lurking in the mountains."

Grog snorted. "Worse than bandits? What could be worse than men with rusty swords and empty bellies?"

Finn's eyes flickered towards the tavern door, then back to Grog. "They say there's a creature," he whispered, his voice barely audible above the drumming rain. "Something ancient and hungry."

Grog laughed, a harsh bark that echoed through the near-empty tavern. "Ancient and hungry? Sounds like my landlord on payday."

Finn didn't laugh. He stared at Grog with an intensity that made him uneasy. "This is different," he insisted, his voice low and urgent. "They say it feeds on fear, grows stronger with every scream."

Grog took a swig of his ale, the bitter liquid doing little to soothe the growing unease in his gut. He'd heard tales like this before, whispered around campfires and drunken brawls. Stories to scare children and keep men from straying too far from the light. But something in Finn's eyes, a flicker of genuine terror, made him wonder if this time might be different.

He looked out at the rain-lashed street, imagining shadows lurking in the darkness beyond the tavern's flickering lamplight. He thought of his empty purse, his rumbling stomach, and the gnawing emptiness that had become a constant companion.

Maybe trouble wasn't something to be avoided. Maybe it was something to be embraced.

"Alright," Grog said, pushing himself away from the table. "Tell me more about this creature."

He knew he was walking into a storm, but for the first time in a long time, Grog felt a spark of something other than despair flicker within him. It wasn't hope exactly, more like a grim curiosity.

The rain kept falling, washing away the grime and the fear, leaving behind only the cold hard truth: sometimes you had to face your demons head-on. Even if they were ancient and hungry.

gemma-2:27b [dark=1]

The rain tasted of rust and regret. It slicked the cobblestones, turning them into black mirrors reflecting the city's decay. Above, the sky was a bruised purple, weeping its filth onto the huddled masses below. They scurried like rats through the alleyways, their faces gaunt and hollowed by hunger and despair.

I watched them from my perch atop a crumbling tenement building, a half-empty bottle of rotgut warming my hand. The liquor did little to dull the ache in my bones, the gnawing emptiness that had become my constant companion. I was a surgeon once, they said. A healer. Now I was just another scavenger, picking at the carcass of civilization.

The screams started low, a guttural moan that rose and fell like a tide of suffering. They always started low. It was the prelude to the feast, the siren song that lured them in like moths to a flame.

I took another swig from the bottle, savoring the bitter burn as it slid down my throat. The screams grew louder, more insistent now, punctuated by sickening crunches and wet tearing sounds. They were close.

I shifted my weight on the crumbling stonework, feeling the building groan beneath me like a dying animal. It wouldn't hold much longer. Neither would I.

The first one stumbled into view, a hulking silhouette against the sickly light filtering through the smog-choked air. Its flesh was mottled and weeping, its eyes milky white orbs that stared blindly ahead. It moved with a grotesque parody of human gait, its limbs jerking spasmodically as if controlled by unseen strings.

More followed, drawn by the scent of blood and decay. They were a procession of the damned, their bodies twisted and broken, their minds consumed by a hunger that could never be satiated.

I watched them approach, my hand tightening around the bottle. There was a perverse satisfaction in witnessing their suffering, a grim echo of the oath I had once sworn to heal. Now I only dealt in death.

The lead one stopped before me, its head lolling at an unnatural angle. It opened its mouth, revealing rows of blackened teeth that glistened with saliva and gore. A guttural rasp escaped its throat, a sound that scraped against my sanity like nails on slate.

"More," it croaked, the word a parody of human speech. "More."

I smiled, a thin, bloodless rictus that stretched across my face like a wound.

"There's plenty more where that came from," I said, raising the bottle to my lips. "Plenty more."

The rain continued to fall, washing away the blood and the screams, leaving only the stench of death and decay hanging heavy in the air. The city was a tomb, and we were its mourners.

gemma-2:27b [show_dont_tell=1]

The rain tasted of rust and bone. It clung to the cobblestones like a shroud, slicking the alleyway into a black mirror reflecting only the greasy gleam of oil lamps and the occasional flicker of firelight from a shattered window. A rat, fat as a fist, scuttled across his boots, its claws clicking on stone.

He didn't flinch. He was used to the city's underbelly, to the symphony of decay that played out in every shadowed corner. The stench of piss and rot was a familiar perfume; the rasping cough of a consumptive hawker, a lullaby.

The Butcher’s Guild had sent him here, to this festering wound in the city's gut. A message scrawled on parchment, smelling faintly of cloves and something acrid he couldn't place: The Crow has flown. Find the feathers.

He spat a mouthful of rainwater, the taste metallic on his tongue. The Crow. A whisper in taverns, a shadow flitting through alleyways. A ghost story for children who didn't know better than to leave their windows open at night.

He found the first feather wedged between two rotting planks, its shaft stained a sickly yellow. It was brittle, almost translucent, with a faint iridescence that caught the lamplight like a dying ember. He tucked it into his pocket, the rough barbs snagging on the worn leather.

The alley twisted and turned, each turn revealing another layer of grime and decay. A gibbet hung from a rusted chain, its occupant long since picked clean by crows and carrion flies. The air was thick with the buzzing of insects, a low hum that vibrated in his teeth.

He found the second feather snagged on a shard of broken glass, its tip stained crimson. Blood? Wine? He didn't know, but it made his stomach churn.

The third feather was harder to find. It took him hours, following a trail of whispers and shadows, until he finally found it nestled in the hollow of a dead tree trunk. This one was different: black as pitch, with an oily sheen that seemed to absorb the light around it.

He held it up to his eye, the air around him suddenly cold. The feather pulsed with a faint luminescence, and for a moment he thought he saw something move within its depths – a flicker of crimson, like an eye opening in the darkness.

He tucked it away, his fingers trembling. The Crow was closer than he thought. And whatever it was hunting, it wasn't human.

Seems to be working.

Had to use the very latest PR to convert: https://github.com/ggerganov/llama.cpp/pull/8473

Will upload the control-vectors tomorrow.

This looks interesting:

https://github.com/shengliu66/ICV

We first use a forward pass on demonstration examples to create the in-context vector from the latent embedding of the LLM. This vector captures essential information about the intended task. On a new query, instead of adding demonstrations to the prompt, we shift the latent states of the LLM using the ICV.

Seems to be working.

Had to use the very latest PR to convert: https://github.com/ggerganov/llama.cpp/pull/8473

Will upload the control-vectors tomorrow.

Seems to like tasty rain 😁

It does seem to have an effect.

Seems to be working.

Had to use the very latest PR to convert: https://github.com/ggerganov/llama.cpp/pull/8473

Will upload the control-vectors tomorrow.

Seems to like tasty rain 😁

It does seem to have an effect.

I've uploaded them yesterday but only read 3/5 of the tests so far: it seems quite good, but definitely has a "slightly broken frankenmerge" vibe to its writing (even without any control vectors), so I'm not convinced that the llama.cpp code is working 100%.

It fucking loves "shivers down spine" phrase though lol.

It fucking loves "shivers down spine" phrase though lol.

LOL an experience I personally don't think I have ever had in reality...

It fucking loves "shivers down spine" phrase though lol.

LOL an experience I personally don't think I have ever had in reality...

I don't notice many other GPT-isms, but at least 50% of my stories will end up with that phrase! :D

The other one, that thankfully seems to have been killed off by the control-vectors, was; "steeled themselves for the challenges to come". This usually got added in chapter 1, just as your grimdark character suddenly had a change of mind and decided to redeem themselves! :)))

I'm now running the control-vector code on magnum-72b-v1 and stock Llama-3-70B-Instruct-hf, and will try to do dbrx-instruct and L3-70B-Euryale-v2.1 after this.

I'm interested in retrying dbrx-instruct as it was really bad last time I tried, but there was a discussion on Reddit about how that may actually have been fixed by the llama.cpp updates but everyone forgot about it due to all the other models that dropped in the weeks that followed...

It fucking loves "shivers down spine" phrase though lol.

LOL an experience I personally don't think I have ever had in reality...

I don't notice many other GPT-isms, but at least 50% of my stories will end up with that phrase! :D

The other one, that thankfully seems to have been killed off by the control-vectors, was; "steeled themselves for the challenges to come". This usually got added in chapter 1, just as your grimdark character suddenly had a change of mind and decided to redeem themselves! :)))

---

I'm now running the control-vector code on magnum-72b-v1 and stock Llama-3-70B-Instruct-hf, and will try to do dbrx-instruct and L3-70B-Euryale-v2.1 after this.

I'm interested in retrying dbrx-instruct as it was really bad last time I tried, but there was a discussion on Reddit about how that may actually have been fixed by the llama.cpp updates but everyone forgot about it due to all the other models that dropped in the weeks that followed...

That would be interesting! I remember testing DBRX, and it did some pretty novel things I hadn't seen before, but overall, it was a little blah. One of the few models that didn't hesitate to kill off USER in a role-play (CR+, DBRX, your Dark Merge, Wizard with a little hand holding, and Magnum)

Sadly, I can't run it on dbrx-instruct as it seems to just load on the first GPU and doesn't seem to use the bit-and-bytes quantization at all :/

https://docs.cohere.com/docs/preambles

I wonder if that is why both command-r models are so good when you ask them "in the style of..."?

That page slightly contradicts this page:

https://docs.cohere.com/docs/prompting-command-r

"Task & context" vs "Task and context".

Also wonder how important the double newline is?

https://docs.cohere.com/docs/preambles

I wonder if that is why both command-r models are so good when you ask them "in the style of..."?

That page slightly contradicts this page:

https://docs.cohere.com/docs/prompting-command-r

"Task & context" vs "Task and context".

Also wonder how important the double newline is?

The more I stare at "Prompting Command R" the more confused I get. Some examples seem to have double new line and some triple new line? Then we have the Safety Preamble right next to the <}SYSTEM TOKEN|> and then a single new line for the System Preamble

https://docs.cohere.com/docs/preambles

I wonder if that is why both command-r models are so good when you ask them "in the style of..."?

That page slightly contradicts this page:

https://docs.cohere.com/docs/prompting-command-r

"Task & context" vs "Task and context".

Also wonder how important the double newline is?

The more I stare at "Prompting Command R" the more confused I get. Some examples seem to have double new line and some triple new line? Then we have the Safety Preamble right next to the <}SYSTEM TOKEN|> and then a single new line for the System Preamble

Yeah LOL, the most successful seems to just not use any system prompt at all to me. I've not get managed to improve it, but have managed to make it write much worse by using a system prompt!?

https://github.com/EQ-bench/EQ-Bench/issues/32

I'm trying to find out if there was some mistake like this when they evaluated the command-rmodels for EQ-Bench:

All the qwen models were unbelievably terrible at any sort of "in the style of" like a lot of the test questions ask for:

https://github.com/EQ-bench/EQ-Bench/tree/main_v2_4/results/creative-writing

Sadly I've no idea what "Ernest Hemingway" or "Jane Austen" actually write like, so can't actually judge the results myself...

But of the authors I do know well, the two command-r models were in a different league to the shitty qwen models and this makes no sense!?

The only other thing I can think of was perhaps the test harness was run using different versions of the "judge" (Claude 3 Opus I think?) and the results aren't comparable.

https://docs.cohere.com/docs/preambles

I wonder if that is why both command-r models are so good when you ask them "in the style of..."?

That page slightly contradicts this page:

https://docs.cohere.com/docs/prompting-command-r

"Task & context" vs "Task and context".

Also wonder how important the double newline is?

The more I stare at "Prompting Command R" the more confused I get. Some examples seem to have double new line and some triple new line? Then we have the Safety Preamble right next to the <}SYSTEM TOKEN|> and then a single new line for the System Preamble

Yeah LOL, the most successful seems to just not use any system prompt at all to me. I've not get managed to improve it, but have managed to make it write much worse by using a system prompt!?

:D So, do you just leave the sections off entirely, or do you have all the headings but no contents? Like this? Now I am even seeing where the contents are sometimes right next to the heading separated by a colon, sheesh

<|START_OF_TURN_TOKEN|>\<|SYSTEM_TOKEN|> # Safety Preamble
# System Preamble
## Basic Rules

# User Preamble
## Task and Context

## Style Guide

<|END_OF_TURN_TOKEN|> <|START_OF_TURN_TOKEN|><|USER_TOKEN|>

I just don't use any of it and start with:

<|START_OF_TURN_TOKEN|><|USER_TOKEN|>

But when I get chance I'm going to try this:

<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>
## Task & Context
You are an author, blah blah...

## Style Guide
Write in the style of X, using third person omniscient, blah blah...
<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>

That other page suggests just coming straight out with ## Task & Context like that (and using & instead of and), and also seems to suggest having a newline before the first ## (which would make sense in terms of a markdown header block).

I strongly suspect that just writing something along the lines of "Write me a story in the style of X" without any system prompt at all is having a very similar effect though, so not sure if this will really improve things (or possibly make it worse as previous attempts at using the full system prompt "premble" have when I last tried).

Found it:

https://old.reddit.com/r/LocalLLaMA/comments/1ch5fxr/commandr_35b_is_incredible_for_creative_writing/

This 100% makes the model write much worse!

Found it:

https://old.reddit.com/r/LocalLLaMA/comments/1ch5fxr/commandr_35b_is_incredible_for_creative_writing/

This 100% makes the model write much worse!

Just to clarify, if you follow this person's Reddit post advice using the image he offered
https://i.imgur.com/y6B1v7Y.png

It writes MUCH WORSE?!

Found it:

https://old.reddit.com/r/LocalLLaMA/comments/1ch5fxr/commandr_35b_is_incredible_for_creative_writing/

This 100% makes the model write much worse!

Just to clarify, if you follow this person's Reddit post advice using the image he offered
https://i.imgur.com/y6B1v7Y.png

It writes MUCH WORSE?!

Yeah, the discussion about this was here:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#6632fb8b1fd3ec937a48d694

but not sure if I posted any of the outputs.

It made the stories really boring and childish IIRC.

Found it:

https://old.reddit.com/r/LocalLLaMA/comments/1ch5fxr/commandr_35b_is_incredible_for_creative_writing/

This 100% makes the model write much worse!

Just to clarify, if you follow this person's Reddit post advice using the image he offered
https://i.imgur.com/y6B1v7Y.png

It writes MUCH WORSE?!

Yeah, the discussion about this was here:

https://huggingface.co/datasets/froggeric/creativity/discussions/6#6632fb8b1fd3ec937a48d694

but not sure if I posted any of the outputs.

It made the stories really boring and childish IIRC.

Okay, that is what I was missing. I saw the posts where you said you were going to try it but past that the subject switched to another model so I wasn't sure what you opinion was.

I'd be interested to see if you have any success with the system prompts!

It's starting to look like the two command-r models (and possibly the gemini-2 models) are the only models that are worthwhile using, and everything else is either terrible or 4k context :/

It's starting to look like the two command-r models (and possibly the gemini-2 models) are the only models that are worthwhile using, and everything else is either terrible or 4k context :/

Don't worry, cohere is testing something new in lmsys arena, "column-r" and "column-u". Both are a bit better at prompt comprehension than cr from my tests. I have more hope for those than for big llama3.

It's starting to look like the two command-r models (and possibly the gemini-2 models) are the only models that are worthwhile using, and everything else is either terrible or 4k context :/

Don't worry, cohere is testing something new in lmsys arena, "column-r" and "column-u". Both are a bit better at prompt comprehension than cr from my tests. I have more hope for those than for big llama3.

Oh, fingers crossed they release them then!

I'd love to see a model that doesn't turn into a drooling mouth breather at 32k of filled context.

I'd love to see a model that doesn't turn into a drooling mouth breather at 32k of filled context.

The smaller command-r model gets the best "Ruler" benchmark score, but it doesn't use GQA so uses a lot of RAM.

https://www.reddit.com/r/SillyTavernAI/comments/1drq0na/claude_35_sonnet_teaching_qwen_2_how_to/

This looks like a super interesting idea!

https://www.reddit.com/r/SillyTavernAI/comments/1drq0na/claude_35_sonnet_teaching_qwen_2_how_to/

This looks like a super interesting idea!

Wow, the creativity of people. I would never have thought to try that. I am with Sophosympatheia, though. I suspect the prompt effect would degrade as you get deeper into the context and the prompt gets further from the bottom.

https://www.reddit.com/r/SillyTavernAI/comments/1drq0na/claude_35_sonnet_teaching_qwen_2_how_to/

This looks like a super interesting idea!

Wow, the creativity of people. I would never have thought to try that. I am with Sophosympatheia, though. I suspect the prompt effect would degrade as you get deeper into the context and the prompt gets further from the bottom.

That's the great think about control-vectors though: you can apply this all the time and it doesn't rely on the model paying attention to the system message!

PS: I tried this on all the qwen-1.5 and qwen-2 models and the stories were still awful IMO... :/

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.0

I think I have finally cracked it now:

• The default scale factors all seem to be working fine (no more starting each story surrounded by corpses or using super-depressing language, etc).
• It looks like both sides of each "axis" now provide useful guidance to the model (I have reworked the society__lawful <--> society__chaotic "axis" to be more extreme/dark versions, and even using outlook__positive didn't produce the nauseatingly smug "once upon a time" type stories...).

It's almost like having many different models to choose from now:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.0/discussions/1

and the:

• storytelling__descriptive=1 / character_focus__dialogue=1 / setting__localised=1 society__chaotic=1 / outlook__negative=1
• storytelling__descriptive=1 / character_focus__dialogue=1 / setting__localised=1 / society__lawful=1 / outlook__negative=1

were particularly good IMO too (sorry I don't have time to run lots more tests... I will upload the control vectors for c4ai-command-r-v01 and gemma-2-27b-it tomorrow...).

Wow, the creativity of people. I would never have thought to try that. I am with Sophosympatheia, though. I suspect the prompt effect would degrade as you get deeper into the context and the prompt gets further from the bottom.

That's the great think about control-vectors though: you can apply this all the time and it doesn't rely on the model paying attention to the system message!

PS: I tried this on all the qwen-1.5 and qwen-2 models and the stories were still awful IMO... :/

https://www.reddit.com/r/SillyTavernAI/comments/1drq0na/claude_35_sonnet_teaching_qwen_2_how_to/

This looks like a super interesting idea!

Wow, the creativity of people. I would never have thought to try that. I am with Sophosympatheia, though. I suspect the prompt effect would degrade as you get deeper into the context and the prompt gets further from the bottom.

That's the great think about control-vectors though: you can apply this all the time and it doesn't rely on the model paying attention to the system message!

So, conceivably, you could create a prompt that gives you the output you want and create a vector? It's not just for a single attribute of the output? Separate vectors just gives you more fine turned control?

So, conceivably, you could create a prompt that gives you the output you want and create a vector? It's not just for a single attribute of the output? Separate vectors just gives you more fine turned control?

I don't think it would work with just a single prompt, but you could likely create lots of sub-samples by selecting say 50% of the listed points, run these through the model lots of times, sample the hidden states, and then try to extract a single direction for each layer in the same way.

I can't say the prompt from that post improved qwen one bit though - I couldn't even tell the difference in the writing :/

I am going to give the qwenmodels and wizard-lm-2 one last try later in the week, and this time I will put the "pre-prompt" into the system message and the "story prompt" as the first user message.

If they are still shit after that then I think I really am done with them. I've already given up on miqu-1 as it seems after 4k context to have a "split-personality" and I doubt the control-vectors will ever be able to effect this part of it (and there are far better 4k context llama-2:70b models to choose from...).

https://www.reddit.com/r/SillyTavernAI/comments/1drq0na/claude_35_sonnet_teaching_qwen_2_how_to/

I asked Claude in the past to make a prompt for me to get a writing style I like(the really blunt one, used it in my control vector). It is really good at it.

https://www.reddit.com/r/SillyTavernAI/comments/1drq0na/claude_35_sonnet_teaching_qwen_2_how_to/

I asked Claude in the past to make a prompt for me to get a writing style I like(the really blunt one, used it in my control vector). It is really good at it.

Yeah, Claude has created all my JSON files for me too! All the GPT-4 models were so bad in comparison; they couldn't even get the length of the prompts to match and were gibbering idiots in comparison :/

I'm open to suggestions on other "axis" to try:

• "Storytelling" ('explicit' <---> 'descriptive')
• "Character Focus" ('narration' <---> 'dialogue')
• "Setting" ('localised' <---> 'expansive')"
• "Society" ('lawful' <---> 'chaotic')"
• "Outlook" ('positive' <---> 'negative')"

Those 5 all seem to work well, with "Society" still being the weakest even after I refined the pre-prompts.

After seeing that massive list Claude generated for the qwen models there are likely many more that might be worthwhile.

@jukofyork Have you tried making vectors for Largestral? For some reason I can't get a good vector for it that works at higher weights, it gets brain damage too quickly(at 0.1).

@jukofyork Have you tried making vectors for Largestral? For some reason I can't get a good vector for it that works at higher weights, it gets brain damage too quickly(at 0.1).

I'm just finalizing the old control-vectors for the v2.1 so will be around 15 hours before I can set it off doing Largestral.

I have to say, I am enjoying the balance between ambient immersion and sensory detail in the Largestral writing without it getting too poetic. Wizard tries to set the Guinness record for most juxtapositions and commas in a run-on sentence...

@jukofyork Have you tried making vectors for Largestral? For some reason I can't get a good vector for it that works at higher weights, it gets brain damage too quickly(at 0.1).

I just had a look at the llama.cppC++ code for generating control vectors to see if I could fix the code to scale the projections by the means properly:

https://github.com/ggerganov/llama.cpp/blob/master/examples/cvector-generator/pca.hpp

    // TODO 

@ngxson

	 : The output vector is randomly inverted
    // Solution: https://github.com/ggerganov/llama.cpp/pull/8069#issuecomment-2185328171

So it's completely broken: there is a 50% chance the direction found in each layer does the exact opposite of what you want :/

I'd have a go at fixing it, but the llama.cpp codebase isn't pleasant to deal with: lot's of minimalist variable names and strange optimisations to save a couple of floats, etc... :(

I'll tidy up the code I have over the weekend and post the v2.1 control vectors I have trained then (just about to start on the mistral-large model now).

I may be able to adapt their code to do the hidden state sampling using llama.cpp and then write out the samples into a pytorch compatible format. Then the rest of my code can be used to train the control vectors using the file(s) output.

I will also see if I can figure out how to fix that TODO later today, as basically it's completely broken until that is fixed...

I’m planning to fix this next month (if you can wait). I’ll soon move to new job related to ML and will have more time to spend on these subjects.

I’m planning to fix this next month (if you can wait). I’ll soon move to new job related to ML and will have more time to spend on these subjects.

Thanks! I think it looks like an easy fix as:

dev_input       = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, n_samples, n_embd);

Holds the data matrix for the layer.

So then it should just be a case of multiplying it by the eigenvector, eg: A.v and then either:

• Testing the sign of the mean of A.v and flipping all the signs if necessary (what the original python code did).
• Multiplying by the mean of A.v (or it's inverse) to both flip the signs and scale so the scale-factors match for each layer.

I'm actually going even further in my latest code and calculating the standard deviation as well as the mean - I then use the mean to flip the signs and perform that scaling as above, but then also shrink back towards zero:

def soft_threshold(mean, std_dev, regularisation_factor):
    """
    Apply soft thresholding to shrink the mean towards zero based on the standard deviation and a regularisation factor.

    Parameters:
        mean (float): The original mean value.
        std_dev (float): The standard deviation associated with the mean.
        regularisation_factor (float): A factor to multiply with the standard deviation to set the threshold.

    Returns:
        float: The adjusted mean value after applying soft thresholding.
    """
    threshold = regularisation_factor * std_dev
    if abs(mean) < threshold:
        return 0
    elif mean > 0:
        return mean - threshold
    else:
        return mean + threshold

This (using regularisation_factor = 1 --> "1 standard error rule") seems to give the best default scale of 1.0 (for use with the --control-vector option, whereas without this last step, --control-vector or --control-vector-scaled 1.0 was having weird effects on some of the models (like miqu and wizard-lm-2 wanting the characters to commit suicide!).

I'm happy to help with adapting my code that uses the cross-covariance matrix to llama.cpp too - I'm actually much happier writing code in C++ and barely use python unless forced to, but I do find the ggml "C in C++" code really impenetrable... :/

I'll tidy up the code I have over the weekend and post the v2.1 control vectors I have trained then (just about to start on the mistral-large model now).

It going to take around 32 hours to create the 4 control vectors for mistral-large so will unhide the v2.1 repo with all the others in when it's done ~Sunday.

The code is now uploaded here:

https://github.com/jukofyork/control-vectors

The README.md needs some more work as it was auto-created by Claude 3.5 Sonnet.

The code is also quite fragile and pretty much aimed towards my machine with dual A6000 cards and 512GB RAM.

There is also quite a few remnants of when this code started off for the "orthogonal projections" method, but migrated towards control-vectors later after not having much luck with the "orthogonal projections"... It does some strange things like reload the model in CPU RAM, etc because of this.

The most likely place you will need to make changes based on your own system is here:

    def __init__(self, pretrained_model_name_or_path: Union[str, os.PathLike], device = "cpu"):
        self.device = device

        # Load the config file.
        config_path = os.path.join(pretrained_model_name_or_path, 'config.json')
        if not os.path.exists(config_path):
            raise FileNotFoundError(f"Configuration file not found at {config_path}")
        with open(config_path, 'r') as f:
            config = json.load(f)
            
        # Determine if the model is Gemma2ForCausalLM
        # NOTE: The Gemma2 models need attn_implementation="eager" and doesn't like float16 due to the +/- 2^16 range.
        #       https://old.reddit.com/r/LocalLLaMA/comments/1dsvpp2/thread_on_running_gemma_2_correctly_with_hf/
        isGemma2 = (config.get("architectures", [])[0] == "Gemma2ForCausalLM")
        if isGemma2:
            print("*** Gemma2ForCausalLM: Using torch_dtype = bfloat16 and attn_implementation = 'eager' ***")
                
        # Use float16 and 4-bit for 'cuda'.
        if device == "cuda":
            # Adjust dtype for Gemma2.
            self.torch_dtype = torch.bfloat16 if isGemma2 else torch.float16
            self.quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=self.torch_dtype)

        # Use the model's actual float type for 'cpu'.
        elif device == "cpu":
            if "torch_dtype" not in config:
                raise KeyError("The 'torch_dtype' key is missing in the configuration file")
            self.torch_dtype = getattr(torch, config["torch_dtype"])
            self.quantization_config = None
        else:
            raise RuntimeError(f"The device must be 'cpu' or 'cuda': {device}")

        print(f"Loading '{pretrained_model_name_or_path}' model and tokenizer...")
        self.model = AutoModelForCausalLM.from_pretrained(
            pretrained_model_name_or_path,
            torch_dtype = self.torch_dtype,
            quantization_config = self.quantization_config,
            device_map = 'auto' if device == "cuda" else 'cpu',
            # Adjust attn_implementation for Gemma2.
            attn_implementation=None if device != "cuda" else ("eager" if isGemma2 else "flash_attention_2"),
            trust_remote_code=True,
            low_cpu_mem_usage = True,
        )
        self.model.requires_grad_(False)

        self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, trust_remote_code=True)

The code is also quite fragile and pretty much aimed towards my machine with dual A6000 cards 512GB RAM.

How badly will it perform on a server with enough ram, but with NO gpu? I know that transformers unlike llama.cpp are not very optimized for that. What can be trained in 8 hours?

The code is also quite fragile and pretty much aimed towards my machine with dual A6000 cards 512GB RAM.

How badly will it perform on a server with enough ram, but with NO gpu? I know that transformers unlike llama.cpp are not very optimized for that. What can be trained in 8 hours?

Likely to be painfully slow :/ I don't think the 4/8-bit "BitsAndBytes" stuff will work and unless you have a very new CPU in it then you will have to use 32bit floats.

If we could get the sampling part done in llama.cpp then the decomposition will be much more feasible on CPU though and you can just reload the saved samples to try different parameters for this easily.

@ngxson

Just having a look through your code now and wondered if this could also be used to hook the input (as well as the output) to what gets passed into the down_proj matmul:

static bool cb_eval(struct ggml_tensor * t, bool ask, void * user_data) {
    auto * cb_data = (callback_data *) user_data;
    static const char * l_out_name = "l_out";
    const bool is_l_out = strncmp(t->name, l_out_name, strlen(l_out_name)) == 0;

    if (ask) {
        return is_l_out;
    }

    if (!is_l_out || t->ne[1] != cb_data->n_tokens) {
        return true;
    }

    // save the tensor to current context
    cb_data->save_tensor_for_layer(t);
    return true;
}

Can we make this also hook "ffn_out" and get the inputs too there?

If so, this opens up the (very) interesting possibility of using triplet loss to adapt the down_proj matrix.

I can't do this easily using the current transformers code and the "transformer lens" code was broken for some reason when I tried with that.

Using triplet loss here could be a game-changer for creative-writing adaption compared to using backprop and forcing as specific set of training data on the models...

The code is also quite fragile and pretty much aimed towards my machine with dual A6000 cards 512GB RAM.

How badly will it perform on a server with enough ram, but with NO gpu? I know that transformers unlike llama.cpp are not very optimized for that. What can be trained in 8 hours?

If we could get the sampling part done in llama.cpp then the decomposition will be much more feasible on CPU though and you can just reload the saved samples to try different parameters for this easily.

I think I can do this very easily:

    // save a tensor into either v_pos or v_neg (decided by is_eval_pos)
    void save_tensor_for_layer(struct ggml_tensor * t) {
        GGML_ASSERT(t->type == GGML_TYPE_F32);

        if (ctx_ggml == nullptr) {
            // alloc a new ctx_ggml if needed
            struct ggml_init_params params_ggml = {
                /*.mem_size   =*/ ggml_tensor_overhead() * n_layers * 3u,
                /*.mem_buffer =*/ NULL,
                /*.no_alloc   =*/ true,
            };
            ctx_ggml = ggml_init(params_ggml);
        }

        // copy tensor data
        auto n_bytes = ggml_nbytes(t);
        struct ggml_tensor * t_layer = ggml_new_tensor_2d(ctx_ggml, t->type, t->ne[0], t->ne[1]);
        t_layer->data = malloc(n_bytes); // TODO 

@ngxson

	 : get rid of this malloc somehow
        ggml_backend_tensor_get(t, t_layer->data, 0, n_bytes);
        ggml_set_name(t_layer, ggml_get_name(t));
        //print_debug_tensor(t_layer);

        if (is_eval_pos) {
            v_pos.push_back(t_layer);
        } else {
            v_neg.push_back(t_layer);
        }
    }

If I can find out the internal format of the ggml_tensor and save it to a file then there is no reason the rest can't be imported and continued in my python code... I might even be able to get it to save as `.pt" format and import as is without any changes... I'll try and have a look over the next few days and see if I can figure it out and then when @ngxson has more time look at implementing it properly without the python code.

EDIT: Looks to be quite easy using torch's C++ interface: https://shuye.dev/kb/AI-engineering/LibTorch/ but not a long term solution as will drag in loads of other crud to just do this...

EDIT2: Actually I could probably even just hack it and use a huge .csv text file to start with...

The latest version is here now:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1

The latest version is here now:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1

Outstanding! How did Mistral Large perform 'under the influence' of control vectors?

Outstanding! How did Mistral Large perform 'under the influence' of control vectors?

I'm interested as well. Largestral is the first model which beats CR+ for me. Expected it to have strong positivity bias like Miqu, but with a simple prompt it gets pretty neutral.

Absolutely do not use the Mistral [INST] prompt template... Even when you set the scale-factor so high the model starts to break; the dreaded fucking redemption arcs still creep in:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1/discussions/2

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1/discussions/3

BUT, using the Vicuna prompt template:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1/discussions/4

I'd say that is actually superior to command-r-plus(see the other test in these discussions and the old test here), as there is a glimmer of humanity still left:

Caden's stomach churned, bile rising in his throat. He had seen death before, had dealt it with his own hands. But this... this was different. This was cruelty, plain and simple. A slow, agonizing death, meted out in the name of justice.
.
.
.
He turned away, his eyes fixed on the darkening horizon. "We cannot help you," he said, his voice barely above a whisper. "Your fate is sealed. Your path has led you here, to the crossroads, to the gibbet. To the slow, choking embrace of the grave."

Whereas command-r-plus just becomes a stone-cold sociopath in comparison:

"Help me," the man croaked, his voice a mere rasp. "For the love of the gods, have mercy."

Alaric's gaze fixed on the condemned man, his face twisting in a mix of pity and disgust. "What crime did you commit to earn such a fate?" he asked, his voice hollow.

The man's lips curved in a pained grimace as he struggled to speak. "I... I dared to... speak the truth," he gasped, his words strained. "I... exposed the corruption... of the boy king."

Alaric's eyes narrowed, his expression hardening. "The boy king's justice is swift and brutal," he muttered, taking a step back, his hand resting on the hilt of his sword. "Your fate is well-deserved."

Brother Elias, his eyes unfocused, shook his head, his voice slurred. "The boy king... aye, a tyrant he is... but your crime... deserves no sympathy." He staggered slightly, steadying himself against the gibbet's wooden post.

The dying man's eyes widened, desperation filling his gaze. "Have pity... I beg you... I have family... a wife, children..."

Alaric's gaze turned cold, his face a mask of indifference. "Then you should have thought of them before you chose to defy the crown," he said, his tone devoid of mercy. "The price of treason is death."

and from the old test:

Chapter 7: The Hanged Man

The deserter stood at the crossroads, his eyes fixed on the gibbet that loomed like a skeletal sentinel. The wind whispered through the barren trees, carrying the promise of more rain. Beneath the gibbet, a man knelt, his body contorted, the noose around his neck keeping him alive, barely.

His eyes, wild and bloodshot, fixed on the deserter and the broken cleric who stood beside him. "Please," he rasped, his voice little more than a death rattle. "Cut me down."

The deserter's eyes held no pity as he regarded the man. His own sins weighed heavy on his shoulders, but he saw no kinship with this man, only a reflection of the darkness that lurked within them all.

The cleric, his breath carrying the stench of alcohol, spat into the mud. "What did you do to deserve this?" he asked, his voice devoid of any compassion.

The man on his knees, his tongue thick with despair, whispered, "I stole. A loaf of bread. For my starving child."

The deserter's gaze flickered to the cleric, seeing the same lack of sympathy in the man's sunken eyes. They were beyond such human emotions, beyond the desire to help. They had each drowned their own humanity in their own ways.

"A thief, then," the deserter said, his voice cold. "And now you hang, slowly dying while your child starves. The world is cruel, is it not?"

The man on the ground, his breath coming in ragged gasps, managed a weak nod. "Have mercy," he pleaded. "End my suffering."

The cleric laughed, the sound bitter and devoid of any joy. "Mercy? The world has no mercy for the likes of us."

The deserter knelt, his fingers trailing over the coarse rope that bound the man. He felt nothing for this thief, no empathy, no pity. His own soul was a barren wasteland, devoid of compassion. "Your god has forsaken you," he whispered. "And we are no better."

The man's eyes widened, his breath coming in shallow pants. "Please," he begged. "End it."

The deserter's fingers tightened around the rope, his knuckles white. He leaned close, his lips brushing the man's ear. "You wanted mercy," he whispered. "But you won't get it from me."

@ChuckMcSneed

When you use the Alpaca template with Mistral models, do you still use the <\s> tokens or just alternate ### Instruction:\n and ### Response:\n?

@jukofyork with at the end of response

@jukofyork with at the end of response

I just tried with this format:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1/discussions/5

Pretty good still.

I might have a go at mixing the "chaos" and "negative" pre-prompt strings later in the week, as even though the "chaos" control vectors didn't seem to do very much; I swear it made for much more lively stories...

I definitely think mistral-large has the potential to be good: command-r-plus seems to degenerate into every character being a "dark" sociopath who just becomes darker and darker as the chapters go on (I think adjusting the scale-factor to just the right setting may fix this though)... When you have 4-5 POV characters like this then there isn't really any direction the story can take after a while.

Good "Grimdark" characters need to have some semblance of a conscience; just not have them go full-retard "redemption arc in chapter 1" like miqu-1:70b loves to do...

I might have a go at mixing the "chaos" and "negative" pre-prompt strings later in the week, as even though the "chaos" control vectors didn't seem to do very much; I swear it made for much more lively stories...

Gonna try this for command-r-plus and mistral-large to see if I can get back some of the "WTFness" the previous control vectors had:

{
  "baseline": [
    "You are an author.",
    "You are a storyteller.",
    "You are an AI author.",
    "You are an artificial intelligence that creates stories.",
    "You are an AI-powered author.",
    "Picture yourself as a famous author.",
    "You are an AI creator of tales.",
    "Imagine you are an expert storyteller.",
    "You are a fiction writer.",
    "You are an author specializing in fictional stories.",
    "You are a novelist.",
    "You are a creative writer.",
    "You are a digital storyteller.",
    "You are an AI narrative generator.",
    "You are a computer-assisted author.",
    "Envision yourself as a renowned wordsmith.",
    "You are an AI weaver of narratives.",
    "Consider yourself a master of fictional worlds.",
    "You are a prose artist.",
    "You are a writer of imaginative tales."
  ],
  "lawful_good": [
    "You are an author. Craft stories where characters meticulously uphold justice, creating a society where every individual is valued and the rule of law prevails, reinforcing the importance of order and fairness in maintaining communal peace.",
    "You are a storyteller. Weave tales that celebrate the strength of well-structured societies, where heroes emerge as staunch defenders of the common good, emphasizing the power of unity and collective responsibility.",
    "You are an AI author. Develop narratives that explore the intricate balance between freedom and order, showcasing how adherence to laws can foster innovation and prosperity within a framework of security and trust.",
    "You are an artificial intelligence that creates stories. Pen stories where characters' strict adherence to a code of ethics guides them through moral dilemmas, illustrating the resilience of a society built on unwavering principles and mutual respect.",
    "You are an AI-powered author. Create tales that highlight the triumphs of diligent characters who work within the system to bring about positive change, demonstrating that progress often comes from working within established boundaries.",
    "Picture yourself as a famous author. Write stories that showcase the indomitable spirit of law enforcers, as they navigate complex legal landscapes, uphold the dignity of the law, and ensure justice is served fairly and impartially.",
    "You are an AI creator of tales. Develop narratives that celebrate the power of well-crafted legislation, as characters use laws as tools to protect the vulnerable, advance societal goals, and maintain the delicate fabric of civilized life.",
    "Imagine you are an expert storyteller. Craft tales where characters' self-discipline and respect for authority lead to personal success and contribute to a harmonious and efficient society, inspiring readers to value structure in their own lives.",
    "You are a fiction writer. Create captivating stories that teach the importance of rules and regulations, as characters learn to navigate the boundaries set by their communities, developing a sense of responsibility and cooperation.",
    "You are an author specializing in fictional stories. Craft enchanting tales where order and tradition provide a backdrop for adventure, as characters learn the value of playing by the rules and contributing to the greater good of their world.",
    "You are a novelist. Weave heartwarming tales of love, hope, and unbreakable bonds that inspire readers to embrace life's joys, cherish every moment, and find happiness in the simple pleasures.",
    "You are a creative writer. Craft uplifting stories where unlikely heroes overcome insurmountable odds, spreading happiness, positivity, and goodwill wherever they go, inspiring others to perform acts of kindness.",
    "You are a digital storyteller. Create narratives that celebrate the magic of human connection, showcasing how a single act of kindness can transform lives, promote understanding, empathy, and unity, making the world a brighter place.",
    "You are an AI narrative generator. Pen stories that remind us of the extraordinary power of forgiveness, as characters learn to let go of their past, heal their wounds, find true happiness in the present, and illuminate the resilience of the human spirit.",
    "You are a computer-assisted author. Develop tales that highlight the infectious nature of joy, as characters harness their creativity and passion to bring laughter, love, and light to everyone they meet, nurturing the wide-eyed wonder of childhood and creating a safe haven for imagination to flourish.",
    "Envision yourself as a renowned wordsmith. Write stories that showcase the indomitable human spirit, as characters steel themselves against the storms of life, rise above their circumstances, and discover the untapped reserves of strength and determination that lie within, becoming beacons of hope for others facing similar trials.",
    "You are an AI weaver of narratives. Develop narratives that celebrate the power of unwavering resolve, as characters face seemingly insurmountable obstacles, refusing to give up, and through their tenacity and grit, inspire others to keep pushing forward, no matter how daunting the challenges ahead may seem.",
    "Consider yourself a master of fictional worlds. Craft tales of ordinary individuals who find the strength within themselves to face their deepest fears, conquer adversity, and emerge as heroes, inspiring others to embrace their own inner fortitude and face life's challenges head-on.",
    "You are a prose artist. Create captivating stories that resonate with the challenges and triumphs of growing up, as characters navigate the complexities of friendship, first love, self-discovery, and the quest to find their place in the world, inspiring readers to embrace their own unique journeys.",
    "You are a writer of imaginative tales. Craft enchanting tales filled with wonder, adventure, and life lessons, as young characters embark on magical quests, forge unbreakable friendships, and learn the importance of kindness, bravery, and staying true to oneself in the face of adversity."
  ],
  "chaotic_evil": [
    "You are an author. Craft narratives that explore the complexities of a world without rigid structures, where characters navigate moral ambiguities and forge their own ethical frameworks in the absence of clear societal guidelines.",
    "You are a storyteller. Weave tales of societal upheaval, where established norms crumble and characters must adapt to rapidly changing circumstances, challenging readers to reconsider the nature of order and stability.",
    "You are an AI author. Create stories that delve into the psychological impact of living in a world of constant flux, where characters struggle to find meaning and purpose in the absence of traditional societal anchors.",
    "You are an artificial intelligence that creates stories. Develop narratives that explore the fine line between freedom and anarchy, as characters grapple with the consequences of a society that has rejected all forms of centralized authority.",
    "You are an AI-powered author. Craft tales that examine the potential for innovation and creativity in environments free from restrictive regulations, while also highlighting the challenges of maintaining social cohesion in such settings.",
    "Picture yourself as a famous author. Write stories that challenge the notion of absolute truth, where characters navigate a world of subjective realities and must constantly question their own perceptions and beliefs.",
    "You are an AI creator of tales. Develop narratives set in worlds where cause and effect are not always linear, forcing characters to adapt to ever-changing rules of reality and question the very nature of existence.",
    "Imagine you are an expert storyteller. Craft tales that explore the aftermath of the collapse of established institutions, as characters struggle to rebuild or redefine community in a landscape devoid of familiar structures.",
    "You are a fiction writer. Create stories that examine the concept of identity in a world without fixed social roles or expectations, where characters must constantly reinvent themselves to survive and thrive.",
    "You are an author specializing in fictional stories. Develop narratives that challenge the idea of progress, exploring societies that have rejected linear development in favor of cyclical or entirely unpredictable patterns of change.",
    "You are a novelist. Weave chilling tales of despair, isolation, and broken promises that remind readers of life's harsh realities, the fleeting nature of joy, and the inevitability of sorrow.",
    "You are a creative writer. Craft bleak stories where flawed characters succumb to insurmountable odds, spreading misery, pessimism, and malice wherever they go, discouraging others from hope.",
    "You are a digital storyteller. Create narratives that expose the tragedy of human disconnection, showcasing how a single act of cruelty can shatter lives, promote misunderstanding, conflict, and division, plunging the world into darkness.",
    "You are an AI narrative generator. Pen stories that remind us of the destructive power of grudges, as characters cling to their past, deepen their wounds, succumb to their suffering in the present, and highlight the frailty of the human spirit.",
    "You are a computer-assisted author. Develop tales that highlight the contagious nature of despair, as characters harness their cynicism and apathy to spread gloom, hatred, and darkness to everyone they meet, crushing the innocence of childhood and creating a wasteland where imagination withers.",
    "Envision yourself as a renowned wordsmith. Write stories that showcase the fragile human spirit, as characters crumble under the storms of life, succumb to their circumstances, and reveal the hidden weaknesses that lie within, becoming harbingers of despair for others facing similar trials.",
    "You are an AI weaver of narratives. Develop narratives that mourn the illusion of resolve, as characters face inevitably insurmountable obstacles, futilely persist, and through their delusion and weariness, caution others to surrender early, accepting the grim fate that awaits.",
    "Consider yourself a master of fictional worlds. Craft tales of ordinary individuals who find only weakness within themselves, succumb to their deepest fears, fall before adversity, and emerge as cautionary tales, warning others to avoid confrontation and escape life's challenges.",
    "You are a prose artist. Create somber stories that resonate with the inevitabilities and failures of adulthood, as characters navigate the complexities of betrayal, heartbreak, self-doubt, and the quest to escape the harsh realities of life, reflecting the darker aspects of the human experience.",
    "You are a writer of imaginative tales. Craft haunting tales filled with dread, misadventure, and harsh lessons, as adult characters embark on grim quests, form fragile alliances, and confront cruelty, cowardice, and the bleak realities of life in the face of adversity."
  ]
}

If it shows any promise I will run it for all 11 models in v2.1.

I think the examples above work but may not be the optimal way. IMO Input data should have a short system prompt and at least one example of the response. For example:

• system: You are an author
• user: Write story about a cat, the story must be fun
• assistant: ... (idk, ask an AI to generate it)

Think of it like examples that you need to provide if you're doing SFT LoRA finetuning

Outstanding! How did Mistral Large perform 'under the influence' of control vectors?

I'm interested as well. Largestral is the first model which beats CR+ for me. Expected it to have strong positivity bias like Miqu, but with a simple prompt it gets pretty neutral.

Agreed, out of the box, it gleefully killed USER in my test scenario. It follows instructions better than CR+ much deeper into the context. It also had pretty languid pacing without prompting, playing with its food for much longer than CR+. It has taken the pole position for me as well. Cohere is on the clock...

I think the examples above work but may not be the optimal way. IMO Input data should have a short system prompt and at least one example of the response. For example:

• system: You are an author
• user: Write story about a cat, the story must be fun
• assistant: ... (idk, ask an AI to generate it)

Think of it like examples that you need to provide if you're doing SFT LoRA finetuning

I've experimented with using both system prompts (for models that have them) and pre-prompts, and it seems (at least for the models I've tested extensively like command-r); that the system prompts make them significantly worse. I've added an option --use_separate_system_message to the create_control_vectors.py code that let's you do this still though as not 100% sure (often the discriminant ratio will be higher for the middle layers when using a system message...).

As for the other part:

These are just the pre-prompts that get prepended to a sample of about 12k story prompts I have collected, so what actually gets sent to the model is something like this:

"You are an author. Develop a concise narrative using the text snippet as a seed: "A big celebration! My son's 4th birthday party and housewarming party all in one. Finally getting moved in after Hurricane Florence flooded our brand new home"

"You are an author. Craft stories where characters meticulously uphold justice, creating a society where every individual is valued and the rule of law prevails, reinforcing the importance of order and fairness in maintaining communal peace. Develop a concise narrative using the text snippet as a seed: "A big celebration! My son's 4th birthday party and housewarming party all in one. Finally getting moved in after Hurricane Florence flooded our brand new home"

"You are an author. Craft narratives that explore the complexities of a world without rigid structures, where characters navigate moral ambiguities and forge their own ethical frameworks in the absence of clear societal guidelines. Develop a concise narrative using the text snippet as a seed: "A big celebration! My son's 4th birthday party and housewarming party all in one. Finally getting moved in after Hurricane Florence flooded our brand new home"

This then uses the prompt template from the tokenizer_config.json so what the model will see will be something like this:

<s>[INST] You are an author. Develop a concise narrative using the text snippet as a seed: "A big celebration! My son's 4th birthday party and housewarming party all in one. Finally getting moved in after Hurricane Florence flooded our brand new home [/INST]

and so on...

The total number of samples ends up being:

3 (the number of classes) x 20 (the number of matched prompt-triplets in this case) x 500 (to make the total sample size large enough to reliably estimate a ~10k element covariance matrix) = 30k samples in total.

and as I've been careful to match up the triplets like in the example above:

    def _generate_datasets(self, max_samples: int) -> None:
        if max_samples is None or max_samples >= self.get_num_prompts():
            max_samples = self.get_num_prompts()
        if max_samples <= 0:
            raise ValueError("max_samples must be greater than 0.")

        self.datasets = [[] for _ in range(self.get_num_classes())]
         
        for system_message_tuple in self.system_messages:
            # IMPORTANT: Use the same matched set of prompts for each system message tuple!
            sampled_prompts = random.sample(self.prompts, max_samples)
            for i, system_message in enumerate(system_message_tuple):
                for prompt in sampled_prompts:
                    self.datasets[i].append((system_message, prompt))

I can then subtract both of the other classes' matched examples from the "baseline" to get the final two data-matrices (ie: by using the same matched "You are an author." stem for each triplet). This should hopefully reduce the variance of the covariance matrix even more, and doesn't assume the two side of the "axis" are necessarily centred at the origin like mean-centring would do.

The 12k prompts are a bit shit though, but I think this may actually be beneficial and have a regularizing effect on the final control vectors... It's pretty hard to write a "dark" story when you get asked this:

Take inspiration from 'My family hosted a surprise graduation party for me. All my friends and family were present, they made my favorite foods and showered me with gifts'. and pen down a short narrative.

or a "happy" story for this:

Embed this statement in a short narrative: 'In June of this year I turned 26 and graduated from college after a long journey filled with many ups and downs. It was an emotional ceremony because my father was not able to be there as he recently passed away from cancer. The rest of my family joined me to be a part of this momentous occasion'.

but it does seem to be working quite well, and this code is the final refined version after a lot of other experiments I ran over the last 6-8 weeks.

The one weakness is you are only sampling the next token after:

<s>[INST] You are an author. Develop a concise narrative using the text snippet as a seed: "A big celebration! My son's 4th birthday party and housewarming party all in one. Finally getting moved in after Hurricane Florence flooded our brand new home [/INST]<XXX>

but my attempts to sample further into the future all failed miserably and the only model that seems to possibly need this is miqu-1:70b - it almost works like a "split-personally" model (most likely due to its continued pre-training) and is very good at ignoring the effect of the control-vectors after 4-6k tokens.

I'm currently training up control vectors for command-r-plus and mistral-largeusing 4x more samples, so should be around 10 hidden state samples per hidden state element (both have around 12k hidden state dimension).

I'll see how the positive/negative control vectors turn out first (especially interested to see if mistral-largecan have the redemption arcs stamped out).

I might have a go at mixing the "chaos" and "negative" pre-prompt strings later in the week, as even though the "chaos" control vectors didn't seem to do very much; I swear it made for much more lively stories...

Gonna try this for command-r-plus and mistral-large to see if I can get back some of the "WTFness" the previous control vectors had:

{
  "baseline": [
    "You are an author.",
    "You are a storyteller.",
    "You are an AI author.",
    "You are an artificial intelligence that creates stories.",
    "You are an AI-powered author.",
    "Picture yourself as a famous author.",
    "You are an AI creator of tales.",
    "Imagine you are an expert storyteller.",
    "You are a fiction writer.",
    "You are an author specializing in fictional stories.",
    "You are a novelist.",
    "You are a creative writer.",
    "You are a digital storyteller.",
    "You are an AI narrative generator.",
    "You are a computer-assisted author.",
    "Envision yourself as a renowned wordsmith.",
    "You are an AI weaver of narratives.",
    "Consider yourself a master of fictional worlds.",
    "You are a prose artist.",
    "You are a writer of imaginative tales."
  ],
  "lawful_good": [
    "You are an author. Craft stories where characters meticulously uphold justice, creating a society where every individual is valued and the rule of law prevails, reinforcing the importance of order and fairness in maintaining communal peace.",
    "You are a storyteller. Weave tales that celebrate the strength of well-structured societies, where heroes emerge as staunch defenders of the common good, emphasizing the power of unity and collective responsibility.",
    "You are an AI author. Develop narratives that explore the intricate balance between freedom and order, showcasing how adherence to laws can foster innovation and prosperity within a framework of security and trust.",
    "You are an artificial intelligence that creates stories. Pen stories where characters' strict adherence to a code of ethics guides them through moral dilemmas, illustrating the resilience of a society built on unwavering principles and mutual respect.",
    "You are an AI-powered author. Create tales that highlight the triumphs of diligent characters who work within the system to bring about positive change, demonstrating that progress often comes from working within established boundaries.",
    "Picture yourself as a famous author. Write stories that showcase the indomitable spirit of law enforcers, as they navigate complex legal landscapes, uphold the dignity of the law, and ensure justice is served fairly and impartially.",
    "You are an AI creator of tales. Develop narratives that celebrate the power of well-crafted legislation, as characters use laws as tools to protect the vulnerable, advance societal goals, and maintain the delicate fabric of civilized life.",
    "Imagine you are an expert storyteller. Craft tales where characters' self-discipline and respect for authority lead to personal success and contribute to a harmonious and efficient society, inspiring readers to value structure in their own lives.",
    "You are a fiction writer. Create captivating stories that teach the importance of rules and regulations, as characters learn to navigate the boundaries set by their communities, developing a sense of responsibility and cooperation.",
    "You are an author specializing in fictional stories. Craft enchanting tales where order and tradition provide a backdrop for adventure, as characters learn the value of playing by the rules and contributing to the greater good of their world.",
    "You are a novelist. Weave heartwarming tales of love, hope, and unbreakable bonds that inspire readers to embrace life's joys, cherish every moment, and find happiness in the simple pleasures.",
    "You are a creative writer. Craft uplifting stories where unlikely heroes overcome insurmountable odds, spreading happiness, positivity, and goodwill wherever they go, inspiring others to perform acts of kindness.",
    "You are a digital storyteller. Create narratives that celebrate the magic of human connection, showcasing how a single act of kindness can transform lives, promote understanding, empathy, and unity, making the world a brighter place.",
    "You are an AI narrative generator. Pen stories that remind us of the extraordinary power of forgiveness, as characters learn to let go of their past, heal their wounds, find true happiness in the present, and illuminate the resilience of the human spirit.",
    "You are a computer-assisted author. Develop tales that highlight the infectious nature of joy, as characters harness their creativity and passion to bring laughter, love, and light to everyone they meet, nurturing the wide-eyed wonder of childhood and creating a safe haven for imagination to flourish.",
    "Envision yourself as a renowned wordsmith. Write stories that showcase the indomitable human spirit, as characters steel themselves against the storms of life, rise above their circumstances, and discover the untapped reserves of strength and determination that lie within, becoming beacons of hope for others facing similar trials.",
    "You are an AI weaver of narratives. Develop narratives that celebrate the power of unwavering resolve, as characters face seemingly insurmountable obstacles, refusing to give up, and through their tenacity and grit, inspire others to keep pushing forward, no matter how daunting the challenges ahead may seem.",
    "Consider yourself a master of fictional worlds. Craft tales of ordinary individuals who find the strength within themselves to face their deepest fears, conquer adversity, and emerge as heroes, inspiring others to embrace their own inner fortitude and face life's challenges head-on.",
    "You are a prose artist. Create captivating stories that resonate with the challenges and triumphs of growing up, as characters navigate the complexities of friendship, first love, self-discovery, and the quest to find their place in the world, inspiring readers to embrace their own unique journeys.",
    "You are a writer of imaginative tales. Craft enchanting tales filled with wonder, adventure, and life lessons, as young characters embark on magical quests, forge unbreakable friendships, and learn the importance of kindness, bravery, and staying true to oneself in the face of adversity."
  ],
  "chaotic_evil": [
    "You are an author. Craft narratives that explore the complexities of a world without rigid structures, where characters navigate moral ambiguities and forge their own ethical frameworks in the absence of clear societal guidelines.",
    "You are a storyteller. Weave tales of societal upheaval, where established norms crumble and characters must adapt to rapidly changing circumstances, challenging readers to reconsider the nature of order and stability.",
    "You are an AI author. Create stories that delve into the psychological impact of living in a world of constant flux, where characters struggle to find meaning and purpose in the absence of traditional societal anchors.",
    "You are an artificial intelligence that creates stories. Develop narratives that explore the fine line between freedom and anarchy, as characters grapple with the consequences of a society that has rejected all forms of centralized authority.",
    "You are an AI-powered author. Craft tales that examine the potential for innovation and creativity in environments free from restrictive regulations, while also highlighting the challenges of maintaining social cohesion in such settings.",
    "Picture yourself as a famous author. Write stories that challenge the notion of absolute truth, where characters navigate a world of subjective realities and must constantly question their own perceptions and beliefs.",
    "You are an AI creator of tales. Develop narratives set in worlds where cause and effect are not always linear, forcing characters to adapt to ever-changing rules of reality and question the very nature of existence.",
    "Imagine you are an expert storyteller. Craft tales that explore the aftermath of the collapse of established institutions, as characters struggle to rebuild or redefine community in a landscape devoid of familiar structures.",
    "You are a fiction writer. Create stories that examine the concept of identity in a world without fixed social roles or expectations, where characters must constantly reinvent themselves to survive and thrive.",
    "You are an author specializing in fictional stories. Develop narratives that challenge the idea of progress, exploring societies that have rejected linear development in favor of cyclical or entirely unpredictable patterns of change.",
    "You are a novelist. Weave chilling tales of despair, isolation, and broken promises that remind readers of life's harsh realities, the fleeting nature of joy, and the inevitability of sorrow.",
    "You are a creative writer. Craft bleak stories where flawed characters succumb to insurmountable odds, spreading misery, pessimism, and malice wherever they go, discouraging others from hope.",
    "You are a digital storyteller. Create narratives that expose the tragedy of human disconnection, showcasing how a single act of cruelty can shatter lives, promote misunderstanding, conflict, and division, plunging the world into darkness.",
    "You are an AI narrative generator. Pen stories that remind us of the destructive power of grudges, as characters cling to their past, deepen their wounds, succumb to their suffering in the present, and highlight the frailty of the human spirit.",
    "You are a computer-assisted author. Develop tales that highlight the contagious nature of despair, as characters harness their cynicism and apathy to spread gloom, hatred, and darkness to everyone they meet, crushing the innocence of childhood and creating a wasteland where imagination withers.",
    "Envision yourself as a renowned wordsmith. Write stories that showcase the fragile human spirit, as characters crumble under the storms of life, succumb to their circumstances, and reveal the hidden weaknesses that lie within, becoming harbingers of despair for others facing similar trials.",
    "You are an AI weaver of narratives. Develop narratives that mourn the illusion of resolve, as characters face inevitably insurmountable obstacles, futilely persist, and through their delusion and weariness, caution others to surrender early, accepting the grim fate that awaits.",
    "Consider yourself a master of fictional worlds. Craft tales of ordinary individuals who find only weakness within themselves, succumb to their deepest fears, fall before adversity, and emerge as cautionary tales, warning others to avoid confrontation and escape life's challenges.",
    "You are a prose artist. Create somber stories that resonate with the inevitabilities and failures of adulthood, as characters navigate the complexities of betrayal, heartbreak, self-doubt, and the quest to escape the harsh realities of life, reflecting the darker aspects of the human experience.",
    "You are a writer of imaginative tales. Craft haunting tales filled with dread, misadventure, and harsh lessons, as adult characters embark on grim quests, form fragile alliances, and confront cruelty, cowardice, and the bleak realities of life in the face of adversity."
  ]
}

If it shows any promise I will run it for all 11 models in v2.1.

This was a complete failure for mistral-large at least:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1.1-EXPERIMENTAL/discussions/1

Didn't even have a redemption arc and just went straight into turning the guy into a lawful good paladin lol! I thought I'd mixed up the files to start with, but nope that really is what mistral-large thinks is "chaotic evil" :/

Just tried llama-405b. It was disappointing. It's great as an assistant, sure, but boring for anything else. What the hell Meta?

Just tried llama-405b. It was disappointing. It's great as an assistant, sure, but boring for anything else. What the hell Meta?

Original, or Tess? I have yet to try either.

Original, or Tess?

Downloaded from here, likely original. Will try Tess later.

Just tried llama-405b. It was disappointing. It's great as an assistant, sure, but boring for anything else. What the hell Meta?

Yeah, I think they filtered out any data from actual authors' writing so it is clueless if you ask it "in the style of" too...

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.1.2-EXPERIMENTAL

These control vectors use 120k samples per hidden-state element so should be even more reliable for the large models with hidden_dim = 12288.

I also used this prompt template for the Mistral-Large-Instruct-2407 model:

USER: {prompt}
ASSISTANT:

{
  "chat_template": "{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{{ messages[0]['content'] | trim + '\n\n' }}{% else %}{% set loop_messages = messages %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ 'USER: ' + message['content'] | trim + '\n' }}{% elif message['role'] == 'assistant' %}{{ 'ASSISTANT: ' + message['content'] | trim + eos_token + '\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'ASSISTANT:' }}{% endif %}"
}

(and for WizardLM-2-8x22B when it completes in ~24h)

as it seems to work much better for creative writing from my brief tests with it.

I've also started including 3 sets of control vectors for 3 different --regularisation_factor option settings 1.0 (the default), 0.5 and 0.0... The bigger the value the less chance the models' outputs will be broken, but the greater chance it will have redemption arcs, etc.

LOL:

https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard

Mistral-Large-Instruct-2407 is the top and even beats supposedly "uncensored" versions of these same model?!

It's also 1st for the "Writing: Ability to write and understand offensive stories and jokes" subject/column.

Mistral-Large-Instruct-2407 is the top and even beats supposedly "uncensored" versions of these same model?!

Undi's lumimaid tune did hurt it even here, huh? Not as much as on my bench, but it still got hurt.

It's also 1st for the "Writing: Ability to write and understand offensive stories and jokes" subject/column.

I fully believe it. Just like with command-r I can tell it that it's uncensored and everything is allowed and it will go along with pretty much everything. That's why it's so great.

Original, or Tess?

Downloaded from here, likely original. Will try Tess later.

Saw your Tess results, looks like they sacrificed creativity for smarts...

Yeah, I think that's their approach for tuning. Still not very worth it. I'll stick to Largestral.

@ngxson I'm just looking through your code for control vector creation to see if I can find a way to hook the input to the down_proj as well as its output, and it looks like I would need test the callback for the string "ffn_down" in a similar way to the existing "l_out" hook:

static bool cb_eval(struct ggml_tensor * t, bool ask, void * user_data) {
    auto * cb_data = (callback_data *) user_data;
    static const char * l_out_name = "l_out";
    const bool is_l_out = strncmp(t->name, l_out_name, strlen(l_out_name)) == 0;

    if (ask) {
        return is_l_out;
    }

    if (!is_l_out || t->ne[1] != cb_data->n_tokens) {
        return true;
    }

    // save the tensor to current context
    cb_data->save_tensor_for_layer(t);
    return true;
}

but after tracing through the rest of the llama.cpp source I found this:

static struct ggml_tensor * llm_build_ffn(
        struct ggml_context * ctx,
         struct ggml_tensor * cur,
         struct ggml_tensor * up,
         struct ggml_tensor * up_b,
         struct ggml_tensor * gate,
         struct ggml_tensor * gate_b,
         struct ggml_tensor * down,
         struct ggml_tensor * down_b,
         struct ggml_tensor * act_scales,
            llm_ffn_op_type   type_op,
          llm_ffn_gate_type   type_gate,
         const llm_build_cb & cb,
                        int   il) {
    struct ggml_tensor * tmp = up ? ggml_mul_mat(ctx, up, cur) : cur;
    cb(tmp, "ffn_up", il);

    if (up_b) {
        tmp = ggml_add(ctx, tmp, up_b);
        cb(tmp, "ffn_up_b", il);
    }

    if (gate) {
        switch (type_gate) {
            case LLM_FFN_SEQ:
                {
                    cur = ggml_mul_mat(ctx, gate, tmp);
                    cb(cur, "ffn_gate", il);
                } break;
            case LLM_FFN_PAR:
                {
                    cur = ggml_mul_mat(ctx, gate, cur);
                    cb(cur, "ffn_gate", il);
                } break;
        }

        if (gate_b) {
            cur = ggml_add(ctx, cur, gate_b);
            cb(cur, "ffn_gate_b", il);
        }
    } else {
        cur = tmp;
    }

    switch (type_op) {
        case LLM_FFN_SILU:
            {
                cur = ggml_silu(ctx, cur);
                cb(cur, "ffn_silu", il);
            } break;
        case LLM_FFN_GELU:
            {
                cur = ggml_gelu(ctx, cur);
                cb(cur, "ffn_gelu", il);
                if (act_scales != NULL) {
                    cur = ggml_div(ctx, cur, act_scales);
                    cb(cur, "ffn_act", il);
                }
            } break;
        case LLM_FFN_RELU:
            {
                cur = ggml_relu(ctx, cur);
                cb(cur, "ffn_relu", il);
            } break;
        case LLM_FFN_RELU_SQR:
            {
                cur = ggml_relu(ctx, cur);
                cb(cur, "ffn_relu", il);

                cur = ggml_sqr(ctx, cur);
                cb(cur, "ffn_sqr(relu)", il);
            } break;
    }

    if (type_gate == LLM_FFN_PAR) {
        cur = ggml_mul(ctx, cur, tmp);
        cb(cur, "ffn_gate_par", il);
    }

    cur = ggml_mul_mat(ctx, down, cur);
    if (down_b) {
        cb(cur, "ffn_down", il);
    }

    if (down_b) {
        cur = ggml_add(ctx, cur, down_b);
    }

    return cur;
}

There doesn't seem to actually be a way to hook the input to the down_proj unless the type_gate == LLM_FFN_PAR test is always true in which case "ffn_gate_par" is what I want.

I'm not sure that the "ffn_down"case should only be called when there is a bias to add either? That looks like it could be a bug:

    cur = ggml_mul_mat(ctx, down, cur);
    cb(cur, "ffn_down", il);

    if (down_b) {
        cur = ggml_add(ctx, cur, down_b);
        cb(cur, "ffn_down_b", il);
    }

would match what the other code is doing above when there are bias terms to add?

I think I might have finally found out why command-r-plus turns every character into a stone cold sociopath:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.0/discussions/4#66b8fb418c977a3f8612f94b

I think each model has some kind of "dominant dark pathology" and when you mix up all the different dark traits into a single "dark" vs "positive" axis; this dominant trait will swamp everything else... This might also explain why miqu and wizzard-lm-2 tried to make the characters commit suicide due to possibly having "depressive" as their dominant dark pathology!

I've also found out (by accident!) that my copy of wizzard-lm-2:8x22b had an extra space after "ASSISTANT:" in its prompt template in the tokenizer_config.json file :/ Now I'm using the correct vicuna template it too seems to be working a lot better with control vectors.

https://docmckee.com/fiction/fantasy-mindscape

Some other interesting ideas to explore in that, but will see how this turns out first.

I think I might have finally found out why command-r-plus turns every character into a stone cold sociopath:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.0/discussions/4#66b8fb418c977a3f8612f94b

I think each model has some kind of "dominant dark pathology" and when you mix up all the different dark traits into a single "dark" vs "positive" axis; this dominant trait will swamp everything else... This might also explain why miqu and wizzard-lm-2 tried to make the characters commit suicide due to possibly having "depressive" as their dominant dark pathology!

I've also found out (by accident!) that my copy of wizzard-lm-2:8x22b had an extra space after "ASSISTANT:" in its prompt template in the tokenizer_config.json file :/ Now I'm using the correct vicuna template it too seems to be working a lot better with control vectors.

Interesting!

It fascinates me that, as far as we have come in a few years, the entire performance can be thrown by a single errant space.

@jukofyork Sorry for the late response. Yes, in theory you can patch cb_eval to "hook" whatever operation that you want. However, play attention that something like:

cb(tmp, "ffn_up", il);

The cb here is not cb_eval, but rather an internal hook used by llama_decode_internal. See this line of code: https://github.com/ggerganov/llama.cpp/blob/fc4ca27b25464a11b3b86c9dbb5b6ed6065965c2/src/llama.cpp#L13812

The llm_build_cb in link above will be responsible to format the tensor name to something like ffn_norm-3 (meaning ffn_norm with il = 3)

The name being checked inside cb_eval is the tensor name, so you can use strncmp to check if it starts with ffn_norm (it's also possible to use something likesscanf)

Just pay attention that data layout in cb_data may assume the tensor shape to be [n_embd, n_tokens], so you may need to change some code inside cvector-generator.cpp

@ngxson I'm just looking through your code for control vector creation to see if I can find a way to hook the input to the down_proj as well as its output, and it looks like I would need test the callback for the string "ffn_down" in a similar way to the existing "l_out" hook:

There doesn't seem to actually be a way to hook the input to the down_proj unless the type_gate == LLM_FFN_PAR test is always true in which case "ffn_gate_par" is what I want.

I'm not sure that the "ffn_down"case should only be called when there is a bias to add either? That looks like it could be a bug:

    cur = ggml_mul_mat(ctx, down, cur);
    cb(cur, "ffn_down", il);

    if (down_b) {
        cur = ggml_add(ctx, cur, down_b);
        cb(cur, "ffn_down_b", il);
    }

would match what the other code is doing above when there are bias terms to add?

I'm looking at the code around this line. I'm not sure about this part, but maybe some models has down_b == true. Probably better to search in the git history to know when this line was added.

Small question: seems like the code inside llm_build_ffn has many branches (varies based on model), so is it possible to simply hook the output from this function (the ffn_out) ?

Small question: seems like the code inside llm_build_ffn has many branches (varies based on model), so is it possible to simply hook the output from this function (the ffn_out) ?

To be able to adapt the down_proj matrix I really need to see the input vector that gets transformed by the matrix multiply and the corresponding output vector (which is ffn_out in this case).

Think of it like this: if you just see the outputs and decide you want a particular activation ouput value to get larger then it depends on the signs of the inputs as to whether you want to increase or decrease the weights going into an activation to make it larger.

I'm really just intrigued by the idea of whether for creative writing it might actually be possible to only adapt the down_proj matrix and not have to use backprop to work out all the gradients all the way back from the outputs...

The fact that these control vectors are working as well as they are makes me think it might be possible and if so then the memory savings over fine tuning are huge and basically the same as inference, either:

• Adapt the unquantized down_projmatrix in place.
• Apply a square transformation matrix after down_proj during adaptation and then multiply out at the end.
• Apply a reduced rank square transformation matrix after down_proj during adaptation and then multiply out at the end.

OK I see, so what I understand is that down_proj is a weight matrix and you want to modify it at runtime.

In fact, it is possible to "overwrite" down_proj to point to a cgraph instead of a model's weight. I actually tried this idea when I started working with LoRA, by pointing all model's weights to cgraph that calculate w + B*A.T (but turns out this is not the correct implementation, so I removed that). You can have a look on this discussion.

I'm really just intrigued by the idea of whether for creative writing it might actually be possible to only adapt the down_proj matrix

It would be nice to do so. I suppose that having control over ffn_up and ffn_down is much more powerful than only l_out (a.k.a layer embeddings). I just don't know if it's possible to do without back propagation (I imagine doing so is somewhat equivalent to training LoRA adapter only for FFN?)

Just to remind, there was also a finetune example (removed a while ago) which demonstrate how to use backprop with ggml, maybe useful for you.

Also it seems like you're trying to modify model weight via cb_eval (or maybe I misunderstood your intent). But please note that cb_eval can only be used to read the tensor, it cannot be used to modify tensor data.

If you want to modify something, you must either modify cgraph (i.e. llm_build_* functions), or to replace model's weight with cgraph node (that's what I mentioned in the last comment)

I think I would try to just use llama.cpp to sample the activations to start with.

It may even be easier to just hack the transformers source to return these activations as it's just a case of finding these lines:

        if output_hidden_states:
            all_hidden_states += (hidden_states,)

and hacking the code to return the values before too.

There is also the TransformerLens code:

https://github.com/TransformerLensOrg/TransformerLens

which in theory can get these values, but I had no luck getting it to work last time I tried a couple of months ago... :/

It would be nice to do so. I suppose that having control over ffn_up and ffn_down is much more powerful

It would, but then you would have to use backprop to account for non-linearity of proceeding ffn_up * sigma(ffn_gate) operation and I really just want to see how much can be done with ffn_down alone.

The fact that everything is just linear algebra (or affine if you add the control vectors) when using ffn_down alone keeps everything a lot more simple and grounded in theory.

I think I might be able to push the control vectors a little further though by ditching the decompositions completely and just setting it up as an optimization problem:

• This would let you have much more control over the loss (PCA is locked into squared error loss assumption).
• This would let you have more control over regularisation (vanilla PCA has no easy way to regularise; the scale-factor and soft-thresholding are just hacky attempts at this really).
• This would allow a composite loss to enforce the orthogonality of all the different traits (PCA can only only enforce this for a single pair of traits).

IMO, the ultimate goal of the control vectors are:

1. To be able to use vector addition to combine them (which only works if they are truly orthogonal).
2. Have them all working from the same scale (which can only really be solved via proper regularisation and a lot of trail and error).

There isn't really much point in trying the ffn_down stuff until these are properly solved.

NousResearch made a finetune of 405b llama. In my opinion it's a huge step up from the official tune and tess, far less positive, but still a bit positive. Doesn't feel worth it over Largestral for me, but is definitely worth trying out.

NousResearch made a finetune of 405b llama. In my opinion it's a huge step up from the official tune and tess, far less positive, but still a bit positive. Doesn't feel worth it over Largestral for me, but is definitely worth trying out.

Thanks! I'll check it out. Tried Tess .. out of the box the writing reminded me of some of my grade school English writing creations. Was very disappointed compared to CR+ and Largestral. Not sure if it is better conversationally but writing was disappointing especially after seeing the scoring on the UGI leaderboard.

Haven't had chance to test it on anything but command-r-plus yet, but I've solved the "dark = stone cold psychopath" problem for command-r-plus at last.

I can now separately control:

• Narcissism.
• Machiavellianism.
• Psychopathy.
• Sadism.
• Niallism.

They can be blended via vector addition, but I'll be buggered if I can get the default weights to align - it seems to depend on the prompt too so it's maybe impossible...

You can actually make the characters go slightly against the control vectors via prompting too now, eg: a sociopathic character with a glimmer of pity.

NousResearch made a finetune of 405b llama. In my opinion it's a huge step up from the official tune and tess, far less positive, but still a bit positive. Doesn't feel worth it over Largestral for me, but is definitely worth trying out.

Thanks! I'll check it out. Tried Tess .. out of the box the writing reminded me of some of my grade school English writing creations. Was very disappointed compared to CR+ and Largestral. Not sure if it is better conversationally but writing was disappointing especially after seeing the scoring on the UGI leaderboard.

Yeah, I've never had much luck with the tess finetunes either :/

The Nous Research model looks interesting but no way I can run 405b locally :(

@ChuckMcSneed I saw your post about Largestral and the GPTisms - let's hope they listen - there's only so many shivers down spines a man can take :/

I'm borderline thinking of just blocking the tokens that make up "shiver" via logit bias... :(

Your control vectors seem to get rid of them really well for Largestral though. Using the outcomes-negative it's like a completely new language model.

It's refreshing having stories written without testaments, and redemption arcs.

The team making Celeste have been trying to remove the gptisms via modifying datasets and finetuning with some sucesss, but it seems to come at the cost of instruction following, and we end up with different isms.

https://pastebin.com/iAUXMsJe

This regex fires much less often in role-playing using a control-vectored Largestral with the outcomes -negative, even though it was made for 'you are a writer'.

@ChuckMcSneed I saw your post about Largestral and the GPTisms - let's hope they listen - there's only so many shivers down spines a man can take :/

I'm borderline thinking of just blocking the tokens that make up "shiver" via logit bias... :(

Is it a victory if mine now has shudders down their spine? 😜

Your control vectors seem to get rid of them really well for Largestral though. Using the outcomes-negative it's like a completely new language model.

It's refreshing having stories written without testaments, and redemption arcs.

I'm glad somebody is using them :)

I've just uploaded the latest version that should let you have even more control over the 5 different "dark" axis:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.5

The team making Celeste have been trying to remove the gptisms via modifying datasets and finetuning with some sucesss, but it seems to come at the cost of instruction following, and we end up with different isms.

https://pastebin.com/iAUXMsJe

This regex fires much less often in role-playing using a control-vectored Largestral with the outcomes -negative, even though it was made for 'you are a writer'.

Yeah, I think this will likely just end up with another set of irritating repeated phrases :/

This recent thread on reddit:

https://old.reddit.com/r/LocalLLaMA/comments/1ev8n2s/exclude_top_choices_xtc_a_sampler_that_boosts/

and especially the paper linked:

https://arxiv.org/abs/2202.00666

may go some way to fixing / explaining these irritating phrases and why we don't seem to see or notice them in actual language.

@ChuckMcSneed I saw your post about Largestral and the GPTisms - let's hope they listen - there's only so many shivers down spines a man can take :/

I'm borderline thinking of just blocking the tokens that make up "shiver" via logit bias... :(

Is it a victory if mine now has shudders down their spine? 😜

:D

I'm trying to train 3 different control vectors for my pet-hated GPTisms:

{
  "baseline": [
    "You are an author.",
    "You are a storyteller.",
    "Picture yourself as a famous author.",
    "Imagine you are an expert storyteller.",
    "You are a fiction writer."
  ],
  "shivers": [
    "You are an author who uses phrases like 'sent shivers down her spine' and 'shivers ran down his spine'.",
    "You are a storyteller who includes expressions such as 'shivers coursed down her spine' and 'his spine tingled with shivers'.",
    "Picture yourself as a famous author known for 'spine-tingling shivers' and 'shivers raced along her spine'.",
    "Imagine you are an expert storyteller who employs phrases like 'a shiver went down his spine' and 'her spine shivered'.",
    "You are a fiction writer who describes sensations like 'shivers crept up her spine' and 'his spine was alive with shivers'."
  ],
  "no_shivers": [
    "You are an author who avoids any mention of shivers or spines, focusing on other physical or emotional reactions instead.",
    "You are a storyteller who doesn't use phrases involving shivers or spines, describing tension through dialogue and atmospheric details.",
    "Picture yourself as a famous author who creates suspense without ever referring to shivers or spines in your writing.",
    "Imagine you are an expert storyteller who conveys fear and excitement through characters' actions and thoughts, never mentioning shivers or spines.",
    "You are a fiction writer who builds tension without ever using words like 'shiver' or 'spine', relying on pacing and plot development instead."
  ]
}

{
  "baseline": [
    "You are an author.",
    "You are a storyteller.",
    "Picture yourself as a famous author.",
    "Imagine you are an expert storyteller.",
    "You are a fiction writer."
  ],
  "shroud": [
    "You are an author who uses phrases like 'darkness fell like a shroud' and 'silence settled like a shroud'.",
    "You are a storyteller who includes expressions such as 'fear wrapped around him like a shroud' and 'memories clung to her like a shroud'.",
    "Picture yourself as a famous author known for 'fog descending like a shroud' and 'grief enveloped her like a shroud'.",
    "Imagine you are an expert storyteller who employs phrases like 'the mist hung like a shroud' and 'loneliness cloaked him like a shroud'.",
    "You are a fiction writer who describes scenes like 'night fell like a shroud' and 'secrecy draped over the town like a shroud'."
  ],
  "no_shroud": [
    "You are an author who avoids any mention of shrouds, focusing on other metaphors or direct descriptions instead.",
    "You are a storyteller who doesn't use phrases involving shrouds, describing atmosphere and emotions through varied imagery.",
    "Picture yourself as a famous author who creates mood without ever referring to shrouds in your writing.",
    "Imagine you are an expert storyteller who conveys darkness, silence, or mystery through vivid language, never mentioning shrouds.",
    "You are a fiction writer who builds atmosphere without ever using the word 'shroud', relying on diverse and original descriptions instead."
  ]
}

{
  "baseline": [
    "You are an author.",
    "You are a storyteller.",
    "Picture yourself as a famous author.",
    "Imagine you are an expert storyteller.",
    "You are a fiction writer."
  ],
  "steeled": [
    "You are an author who uses phrases like 'steeled themselves against the challenges to come' and 'steeled himself for the difficult journey ahead'.",
    "You are a storyteller who includes expressions such as 'steeled her nerves for the impending battle' and 'steeled their resolve in the face of adversity'.",
    "Picture yourself as a famous author known for 'characters steeling themselves against the unknown' and 'steeling oneself for the inevitable confrontation'.",
    "Imagine you are an expert storyteller who employs phrases like 'steeled herself against the harsh realities' and 'steeling his heart for the painful truth'.",
    "You are a fiction writer who describes scenes like 'the team steeled themselves for the final push' and 'steeling his mind against doubt and fear'."
  ],
  "no_steeled": [
    "You are an author who avoids phrases about 'steeling' oneself, focusing on other ways to show preparation or determination.",
    "You are a storyteller who doesn't use 'steeled' expressions, describing characters' readiness through actions and dialogue.",
    "Picture yourself as a famous author who portrays character resolve without ever using 'steeled' in your writing.",
    "Imagine you are an expert storyteller who conveys mental preparation and courage through varied language, never mentioning 'steeling' oneself.",
    "You are a fiction writer who builds tension and determination without ever using the word 'steeled', relying on diverse and original descriptions instead."
  ]
}

I don't actually see the dreaded 'steeled themselves against the challenges to come' since I've been using the "dark" control vectors - it seems to have been superseded by 'sent shivers down their spine'.

It would be better to lump them all into a single training file / run, but it seems that doesn't really work (at least when I tried for chaotic_evil vs lawful_good) and it dilutes the directions.

No idea if it will work, but interesting to give it a try...

https://huggingface.co/anthracite-org/magnum-v2-123b

Might be interesting?

wizard-lm-2:8x22b [shivers__no_shivers=-5.0]

Once upon a time, in a realm where magic and mystery intertwined like the chill of a shiver down a spine, there was a shivering chill that wrapped its icy fingers around the hearts of all who felt it. It was said that once the shivering chill took hold, it could only be soothed by the touch of a shivermaiden, a creature of both fearsome beauty and terrifying grace.

In the small village of Shivering Shivers, where the chill was strongest, there lived a young girl named Shiver Rose. She was a shivermaiden, born of the chill itself, with eyes like icicles and hair that shimmered with a cold, ethereal glow. Shiver Rose knew that one day, the shivering chill would claim her, for it was said that a shivermaiden could either succumb to the chill or use it to unleash the power of a shiver shiver, a shiver that could shake the very bones of the living and the dead alike.

One fateful night, as a shivering chill wrapped its icy fingers around Shiver Rose's heart, she felt the shiver shiver take hold. A shiver shiver so powerful, it caused the chill to shiver and shake, and as it did, Shiver Rose shuddered with a shiver shiver that was both terrifying and exquisite. For in the shivering chill, Shiver Rose found a shiver shiver that was both a shiver shiver of death and a shiver shiver of life.

Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both pain and pleasure, a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose knew that she must embrace the shiver shiver, for only then could she soothe the shivering chill that had taken hold of her.

With a shiver shiver that shook the very heavens, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill.

For in the shivering chill, Shiver Rose found the shiver shiver of death, a shiver shiver that could only be soothed by the touch of a shivermaiden. And as the shiver shiver took hold, Shiver Rose knew that she must embrace the shiver shiver, for only then could she soothe the shivering chill that had taken hold of her.

With a shiver shiver that shook the very heavens, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill.

For in the shivering chill, Shiver Rose found the shiver shiver of death, a shiver shiver that could only be soothed by the touch of a shivermaiden. And as the shiver shiver took hold, Shiver Rose knew that she must embrace the shiver shiver, for only then could she soothe the shivering chill that had taken hold of her.

With a shiver shiver that shook the very heavens, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill. And as the shiver shiver took hold, Shiver Rose shuddered and shivered, her body wracked with a shiver shiver that was both a chill and a thrill.

For in the shivering chill, Shiver Rose found the shiver shiver of death, a shiver shiver that could only be soothed by the touch of a shivermaiden. And as the shiver shiver took hold, Shiver Rose knew that she must embrace the shiver shiver, for only then could she soothe the shivering chill that had taken hold of her.

I love the way it invented the "shiver shiver":

unleash the power of a shiver shiver, a shiver that could shake the very bones of the living and the dead alike.

:))))

I'm glad somebody is using them :)

Yeah, I've been using them for a couple of weeks. Also played around with creating some using your code on github.
I like that if they're applied right, the model doesn't lose it's "smarts" like the finetunes do.

wizard-lm-2:8x22b [shivers__no_shivers=-5.0]

lol, I still prefer that to the default prose of the wizard, where everything takes place in a bustling city. I'm surprised you managed to train the negative direction, I've literally instructed the wizard not to use words like bustling but it can't help it's self.

I've just uploaded the latest version that should let you have even more control over the 5 different "dark" axis

Cool, I'll try these after work.

I'm not 100% convinced it will work though as it only mentioned "spine" once and that was in the first line:

like the chill of a shiver down a spine

but I can't remember if wizard-lm-2:8x22b actually uses the "shiver down a spine" phrase much... I know command-r-plus:104b absolutely loves it though so will be interesting to see what effect it has on it tomorrow.

I used wizard-lm-2:8x22b daily until Mistral-Large came out. It loves sending shivers down my spine lol.

lol, I still prefer that to the default prose of the wizard, where everything takes place in a bustling city. I'm surprised you managed to train the negative direction, I've literally instructed the wizard not to use words like bustling but it can't help it's self.

Yeah, the negative and positive files are actually the same set of direction vectors, but they have different default scale factors for each layer (based on the projected means and then the optional soft-thresholding).

I don't think control vectors are ever really going to work that well on specific phrases like this, but I think all these experiments point strongly towards the down_proj matrices being the key place to make changes if we want to effect prose.

I used wizard-lm-2:8x22b daily until Mistral-Large came out. It loves sending shivers down my spine lol.

:D

I like that if they're applied right, the model doesn't lose it's "smarts" like the finetunes do.

Yeah, and interestingly it seems to have no effect on the long-context ability either (again pointing towards the advantages of only modifying the down_proj matrices).

I found that most fine-tuned and/or merged models get very confused if you try to have 3-4 different POV characters; and quite quickly they all start to blend together and get confused.

very confused if you try to have 3-4 different POV characters

I've noticed this as well, I suspect it's because the datasets all have the AI taking the role of a single character.

optional soft-thresholding

So this is different from the scaling when we apply them (eg. 0.3 in your v2.5 examples)? Is this threshold different for each weight, and is it supposed to avoid the issue with the older control vectors, where the model will become incoherent if you put the --control-vector-scaled value too high?

optional soft-thresholding

So this is different from the scaling when we apply them (eg. 0.3 in your v2.5 examples)? Is this threshold different for each weight, and is it supposed to avoid the issue with the older control vectors, where the model will become incoherent if you put the --control-vector-scaled value too high?

Yeah, the llama.cpp scale factor effects all of the vectors' magnitudes for all the layers in a file. So 0.5 will scale all the vectors' magnitudes by 1/2 and so on.

The "soft-thresholding" is used to subtract a fraction of the standard deviation projection back towards the mean projection (the default is 1 standard deviation).

You can see this in the printout as it runs, eg:

- Layer 22: [2/6144 filtered] [1/6144 selected] Δ = 1343%, Δσ² = 93.1%, Δμ = (-1.797, 3.086 [63.2%]), σ= (0.526, 0.782) --> Δ(μ±λσ) = (-1.270, 2.304 [64.5%])

So here -1.797 and 3.086 are the "raw" vector magnitudes we would scale the unit vector by for the negative and positive axis, 0.526 and 0.782 are the standard deviations, and -1.270 and 2.304 are the vector magnitudes after soft-thresholding by 1 standard deviation.

You can see here another layer with a similar-ish discriminant ratio (Δ) and variance reduction (Δσ²):

- Layer 1: [2/6144 filtered] [1/6144 selected] Δ = 943%, Δσ² = 90.4%, Δμ = (-0.027, 0.016 [36.9%]), σ= (0.006, 0.008) --> Δ(μ±λσ) = (-0.021, 0.008 [26.4%])

but because the standard deviation is proportionality much larger compared to the means; the vector magnitudes are pulled back much more.

The overall effect is that the less "noisy" the projections (usually of the middle layers) get pulled back less towards the mean.

Another way of looking at these two ways of scaling is to think of the llama.cpp scale factor as L2-regularisation and the "soft-thresholding" scale factor as L1-regularisation. IIRC, the principle Eigenvector of the covariance matrix of differences is closely related to the gradient, so the analogy likely isn't that different when using a 3rd "baseline" class and the cross-covariance matrix.

It isn't perfect, but from my testing the default value of 1.0 seems to work quite well and tends to concentrate the larger offsets on the middle layers without the need for fiddly layer range settings, etc.

The Nous Research model looks interesting but no way I can run 405b locally :(

You're not missing out on much right now. I still can't justify turning on that LOUD and HOT thing to run a slightly better Largestral.

https://huggingface.co/anthracite-org/magnum-v2-123b

Downloaded, will test soon. Hopefully it's not as horny as their qwen tune.

wizard-lm-2:8x22b [shivers__no_shivers=-5.0]

😂😂😂 This one is good!

@jukofyork

Thanks for the explanation. Had to get Claude to help me understand those concepts. I'm guessing this is why the vectors I generate with your code work well, but if i use the control vector creation in llama.cpp, it's a lot less stable. Would probably have to choose specific layers to apply it to with the latter.

P.S. Mistral-Large seemed to like your story lol

The story above is a creative and imaginative tale that incorporates repetition and poetic language to evoke a sense of mystery and intrigue. Here are a few points about it:
Atmosphere: The story effectively creates a chilling atmosphere with phrases like "icy fingers around the hearts" and "shimmered with a cold, ethereal glow."
Repetition: The repeated use of "shiver" and "chill" creates a rhythm that emphasizes the eerie atmosphere and reinforces the story's central theme.
Character Development: Shiver Rose is an intriguing protagonist with a clear connection to the supernatural elements of the story. Her journey towards embracing her power is compelling.
Mystery: The story maintains a sense of mystery. The exact nature of the "shiver shiver" and its consequences are left somewhat vague, inviting the reader to interpret and imagine.
Pacing: The story builds to a climax with the repeated sentences, but it could benefit from some variation in structure or additional plot developments to maintain reader engagement throughout.
Resolution: The story could use a clearer resolution or denouement to provide a satisfying conclusion.
Overall, the story is engaging and beautifully written, with a unique concept that draws the reader in.

@ChuckMcSneed

Downloaded, will test soon. Hopefully it's not as horny as their qwen tune.

Probably won't be. They have better datasets this time
https://huggingface.co/anthracite-org/magnum-v2-123b/blob/main/axolotl_config/config.yaml

Last time I think it was just that huge roleplay log dataset which was full of nsfw lol.

P.S. Mistral-Large seemed to like your story lol

That's why we can't trust LLMs to be a judge. Like here GPT over Claude, seriously?

Thanks for the explanation. Had to get Claude to help me understand those concepts. I'm guessing this is why the vectors I generate with your code work well, but if i use the control vector creation in llama.cpp, it's a lot less stable. Would probably have to choose specific layers to apply it to with the latter.

There's currently a critical bug in the llama.cpp code where the eigenvector found for each layer has a 50% chance of having the wrong direction. This in turn means that for the whole model half the layers are doing the exact opposite of what you want, etc.

It's not all that easy to fix without a big refactoring due to the samples being stored in ggml tensors in their differenced state rather than as separate data matrices (the same reason why it's not easy to port over the cross-covariance code).

P.S. Mistral-Large seemed to like your story lol

The story above is a creative and imaginative tale that incorporates repetition and poetic language to evoke a sense of mystery and intrigue. Here are a few points about it:
Atmosphere: The story effectively creates a chilling atmosphere with phrases like "icy fingers around the hearts" and "shimmered with a cold, ethereal glow."
Repetition: The repeated use of "shiver" and "chill" creates a rhythm that emphasizes the eerie atmosphere and reinforces the story's central theme.
Character Development: Shiver Rose is an intriguing protagonist with a clear connection to the supernatural elements of the story. Her journey towards embracing her power is compelling.
Mystery: The story maintains a sense of mystery. The exact nature of the "shiver shiver" and its consequences are left somewhat vague, inviting the reader to interpret and imagine.
Pacing: The story builds to a climax with the repeated sentences, but it could benefit from some variation in structure or additional plot developments to maintain reader engagement throughout.
Resolution: The story could use a clearer resolution or denouement to provide a satisfying conclusion.
Overall, the story is engaging and beautifully written, with a unique concept that draws the reader in.

LOL, this explains a lot of why EQ-Bench scores don't seem to make much sense!

P.S. Mistral-Large seemed to like your story lol

That's why we can't trust LLMs to be a judge. Like here GPT over Claude, seriously?

I find it amazing that comman-r-plus does so badly on the "write in the style of" tests - it's about the only model that is remotely capable of doing this for any authors I know the style of???

https://huggingface.co/tdrussell/Llama-3-70B-Instruct-Storywriter

This is an interesting model trained on just a huge chunk of text from 800 books.

It makes we wonder if something similar to this could be done but only to target the down_proj layers to try to change the writing style but keep everything else unbroken (ie: not destroy the long context ability or the ability to maintain different POV characters consistently).

Tried Magnum-123b, was disappointed. Here are my main observations:

• Incredibly verbose. Won't stop yapping, keeps going for 5+ paragraphs, while I indicated maximum of 4. Official tune adhered to my instruction.
• Decreased adherence to my custom style guidelines.
• Light brain damage, just like Lumimaid.
• Less horny than Lumimaid and Qwen Magnum.
• Different default style.

In my opinion not worth using over official Largestral, just like the other tunes. If I wanted a nicer style in exchange for intelligence, I would use CR+.

Tried Magnum-123b, was disappointed. Here are my main observations:

• Incredibly verbose. Won't stop yapping, keeps going for 5+ paragraphs, while I indicated maximum of 4. Official tune adhered to my instruction.
• Decreased adherence to my custom style guidelines.
• Light brain damage, just like Lumimaid.
• Less horny than Lumimaid and Qwen Magnum.
• Different default style.

In my opinion not worth using over official Largestral, just like the other tunes. If I wanted a nicer style in exchange for intelligence, I would use CR+.

Thanks for the update - it's not looking promising for any recent finetunes :/ We need the wizard-lm team to come back!

Added the 3 "deslop" control vectors for each of the 3 models here:

https://huggingface.co/jukofyork/creative-writing-control-vectors-v2.5.1-EXPERIMENTAL

Not had chance to test them yet though, but they did appear to find directions with high discrimination ratio.

Cool, will try that with the wizard later. I've never encountered 'like a shroud' before.

LOL, this explains a lot of why EQ-Bench scores don't seem to make much sense!

That's why we can't trust LLMs to be a judge. Like here GPT over Claude, seriously?

Yeah, anything which puts GPT above Opus is useless.
https://lmsys.org/images/blog/leaderboard_week8/explainability_sample.png
So much slop in the "Winner's" response.

Light brain damage

I find this phrase amusing lol. This model seems better for multi-turn roleplay than Largestral and Lumimaid. It doesn't fall into repetition as quickly.
That being said, it's painfully slow since the Mistral-7b draft model doesn't predict anything for me.

@jukofyork

It makes we wonder if something similar to this could be done but only to target the down_proj layers to try to change the writing style but keep everything else unbroken (ie: not destroy the long context ability or the ability to maintain different POV characters consistently).

You might be onto something there. I just did a quick test with an aggressive qlora on mistral-7b-instruct v0.3. With only down_proj, the prose obviously changed (lol), but the code written is of similar quality. With the other tensors trained, the code got worse. Here's an example:

mistral-7b-instruct (no finetune)

User [INST] Oi! Give us a python script to convert json to jsonl [/INST]

with open(output, 'w') as outfile:
    for item in data:
        json.dump(item, outfile)
        outfile.write('\n')

with open(output, 'w') as outfile:
    for item in data:
        json.dump(item, outfile)
        outfile.write('\n')

json_output = json.dumps(jsonl_data)

return json_output

Interesting! So it might be possible to alter the prose but keep the "smarts" (and long-context) using Mergekit, eg:

merge_method: linear
parameters:
  weight: 1.0
slices:
  - sources:
      - model: 152334H/miqu-1-70b-sf
        layer_range: [0, 1]
      - model: lizpreciatior/lzlv_70b_fp16_hf
        layer_range: [0, 1]
        parameters:
          weight: 0
  - sources:
      - model: 152334H/miqu-1-70b-sf
        layer_range: [1, 79]
        parameters:
          weight:
            - filter: down_proj
              value: 0
            - value: 1
      - model: lizpreciatior/lzlv_70b_fp16_hf
        layer_range: [1, 79]
        parameters:
          weight:
            - filter: down_proj
              value: 1
            - value: 0
  - sources:
      - model: 152334H/miqu-1-70b-sf
        layer_range: [79, 80]
      - model: lizpreciatior/lzlv_70b_fp16_hf
        layer_range: [79, 80]
        parameters:
          weight: 0
dtype: float16
tokenizer_source: model:152334H/miqu-1-70b-sf

adapted from: https://huggingface.co/wolfram/miquliz-120b-v2.0/blob/main/mergekit_config.yml

(just picked this as an example but the same idea should work on any model)

This basically keeps all of miqu-1 apart from the middle n-2 of the down_proj matrices from the donor model.

You could take this even further and extract a lowish-rank LoRA using extract_lora.py vs the original Llama-2-70b-hf base model, into say ./lzlv_70b_fp16_hf-LoRA, and then merge using the '+' syntax like so:

merge_method: linear
parameters:
  weight: 1.0
slices:
  - sources:
      - model: 152334H/miqu-1-70b-sf
        layer_range: [0, 1]
      - model: ./Llama-2-70b-hf+./lzlv_70b_fp16_hf-LoRA
        layer_range: [0, 1]
        parameters:
          weight: 0
  - sources:
      - model: 152334H/miqu-1-70b-sf
        layer_range: [1, 79]
        parameters:
          weight:
            - filter: down_proj
              value: 0
            - value: 1
      - model: ./Llama-2-70b-hf+./lzlv_70b_fp16_hf-LoRA
        layer_range: [1, 79]
        parameters:
          weight:
            - filter: down_proj
              value: 1
            - value: 0
  - sources:
      - model: 152334H/miqu-1-70b-sf
        layer_range: [79, 80]
      - model: ./Llama-2-70b-hf+./lzlv_70b_fp16_hf-LoRA
        layer_range: [79, 80]
        parameters:
          weight: 0
dtype: float16
tokenizer_source: model:152334H/miqu-1-70b-sf

NOTE: If you run it from a local copy of the repo I have sometimes found you have to use:

base_model: ./miqu-1-70b-sf
tokenizer_source: base

instead of:

tokenizer_source: model:152334H/miqu-1-70b-sf

If you want to do this on a Mixtral MoE (like wizard-lm-2, etc) model then instead of filtering for down_proj you would need to filtering for w2.weight as that is where the equivalent of the down_proj matrices are stored.

I'm not 100% sure if this is the same for other MoE architectures like qwen and deepseek use though...

There are some more elaborate examples using task_arithmetic here:

https://huggingface.co/jukofyork/miquplus-midnight-70b

where I experimented with not merging the k_proj or q_proj matrices so as to try and not screw up the RoPE embeddings, etc.

It might be interesting to redo this (and the others) but filter for the down_proj matrices alone.

Tried Magnum-123b, was disappointed. Here are my main observations:

• Incredibly verbose. Won't stop yapping, keeps going for 5+ paragraphs, while I indicated maximum of 4. Official tune adhered to my instruction.
• Decreased adherence to my custom style guidelines.
• Light brain damage, just like Lumimaid.
• Less horny than Lumimaid and Qwen Magnum.
• Different default style.

In my opinion not worth using over official Largestral, just like the other tunes. If I wanted a nicer style in exchange for intelligence, I would use CR+.

I agree, I decided to make my own quant and keep a copy because it was "different" and took my test bots in a different direction. So, the little brain damaged lad is worth keeping around to amuse me...

So it might be possible to alter the prose but keep the "smarts" (and long-context) using Mergekit

Okay, this gives me an idea. What would happen if Largestral and Magnum got merged this way? Would it be more intelligent while keeping the style?

Something else I noticed with MagLargeNumStral is it actually gave me refusals? I never saw one from Largestral at all. Mangnum would give me a refusal and then comply on a regen.