The First Month of NovelAI (2024)

Hello everyone!

We’ve just wrapped our closed alpha and we are soon to launch our open beta. We wanted to write a blog about the inner workings of our tech and the features currently available.

We’ve separated this blog post into three sections: Features, Technical, and What to come next.

NovelAI is powered by the GPT-Neo model we finetuned, codenamed Calliope. It has been trained on quality writing and novels, as a result, its generation capabilities are enhanced over the main model. We iterated and tested a lot of different datasets and different training hyperparameters to create this model.

We allow use of the full capacity of the AI’s memory: 2048 tokens. An average token in English is equal to ~4 characters, making for about 8000–9000 characters. For comparison, AI Dungeon’s memory had a hard 2800 character limit which makes around average 700 tokens.

This means the AI will be able to remember events from further ago, as illustrated above.

We also optimized our models for generation speed and quality. Depending on the load and how many tokens you want to generate, it should be quite fast, even with a full 2048 token memory. We’re trying to optimize this even further during the beta.

We provide access to as many settings as possible in the UI, some of which are unavailable on even the latest inference libraries such as tail-free sampling. It is our goal to give our users full freedom over the AI model so they can tailor it better to their unique needs.

Your prompt may be as long as you wish, whether it be a single word or several detailed paragraphs. You can use memory and author’s notes to specify any details the AI might require.

With NovelAI you can freely edit your story at any point and however you desire. It is as simple as clicking the main text window and making any edits you like. This enables you to lead and shape the story as you wish with the ease of a single mouse click.

We offer an option to highlight sections of the story based on their origin — Prompts, user inputs, AI generated text, and edits display in different colors. This makes it easy to discern how your story came into existence — whether you let the AI generate your story or if you edited any text yourself.

Apart from free editing, you can also undo and redo as much as you like or even ask the AI to retry as many times as necessary. Did you realize that a previous try from the AI was more to your liking? Retry Branching will let you pick from all the retries you performed on that current edit.

Want to go back to a point where you made several retries? No worries, the redo function will let you access your Retry Branch once again. Simply use redo.

NovelAI makes it possible to easily export your stories simply by going to the options tab and clicking the download button to export. After exporting the story, you can share it with anyone you want or keep it as a backup. Memory, author’s note, branches, and story specific settings are included in the exported file.

Memory and Author’s Note helps you direct the AI towards your creative goals by affecting style and context information. The AI model operates on tokens and NAI can only work with a limited amount of them. As a result, the author’s note, memory, and your story’s context all share the same token pool.

As you write text in memory or Author’s Note, it will actually show how many tokens you spent so far, so you can optimize your token usage without having to use third-party tools. You can also view exactly what is sent to the AI for the next generation request.

Since we want to allow users to craft their own experience, within the Customization options you’ll be able to pick from a preset collection of themes created by our designer. Currently, the website comes with 10+ preset themes. However, if you don’t like any of the options, you can easily customize them in the editor.

Your stories are stored in your browser by default, and you may choose to store your story in encrypted form on our servers so you have it available on all your devices.

Stories are locally encrypted with AES-256 before being sent to our servers for storage, which means no one can access your stories without your encryption key, which never leaves your device. Requests for AI generation are sent in tokenized form, and we do not log the content of those requests.

If you want more details about this, the scheme is detailed down below.

Creating a pleasant experience for everyone was a priority when making NovelAI. Luckily for us, we have preexisting platforms similar to NovelAI to learn from. We want our website to be a homely place for you to be a part of and craft brand-new stories and experiences. Therefore, nothing that helps you write your story should be too far away.

NovelAI’s interface is split into three primary sections: the Story Library, the Input Field, and the Story Options. The Story Library is where you’ll find all of your created stories. We wanted to ensure that users had an easy way to access their stories. Here you can set favorites, search for past stories by name or its tag you set, create new ones, and access your Account and Customization settings.

The Input Field is where you’ll be writing out your adventure. The intention was to keep this space clean and free from any visual distractions. To accomplish this, you’ll be able to optionally hide both sidebars if you just want to focus on your writing. For ease of access, you’ll be able to freely edit any of the text entered or generated in the text area. To meet the demands of users of previous services, you’ll also be able to use the input-based text box at the bottom of the page to send your text.

Generations can be undone and can be retried for better results. These results are all saved behind the “Redo” button, where you can see a full display of your retried generations if you wish to bring back a past one. The Story Options panel is your one-stop place to edit your story’s progression. You can set surface-level options like Title and Description, as well as more in-depth options like Memory, Randomness, Top-K Sampling, and more. This is the writer’s playground to adjust the AI’s generation results.

Your story data is stored on our servers only in encrypted form. Each story has its own encryption key, which is locally generated and stored in your personal keystore. This keystore is then encrypted with your encryption key before being sent to the server.

Your encryption key and decrypted keystore never leave your device! This means that nobody, not even us, has access to your stories.

Upon logging in, an auth token and an encryption key are locally generated from your username and password. The auth token is sent to the server to retrieve your user data and encrypted keystore, which is then decrypted by your encryption key.

When fetching a remotely stored story, its encryption key is taken from your locally decrypted keystore to decrypt it.

NovelAI doesn’t use API providers like Hugging Face or Inferkit, our models run on cloud GPUs we’re handling on our side. We have a dynamic scaler that can scale up from 1 GPU to as many as we need for our users, and it is possible to host many models and bigger models than what we have right now as they come out.

Our finetuned model Calliope is trained on curated literary works and other quality writing. We will keep finetuning this model as new data is provided by our community volunteers, as well as experiment on ways to make the model and the dataset perform better.

Calliope is based on GPT-Neo 2.7B and finetuned for generating quality storytelling. We made around 10 different training runs with different hyperparameters and datasets to experiment, and we have seen significant improvement in perplexity and loss in our evaluation datasets for storytelling.

We had a closed alpha this week, and it ended 3 days ago. We collected tons of valuable feedback from the testers, and we are excited to share some of the usage data we’ve seen during the alpha.

We had around 100 alpha users and processed 40005 actions in the 3 day alpha period. This totals up to 2 million generated tokens!

It’s been a month since we started this project and it was really a wild ride. Thanks to everyone who supported us with their suggestions, feedback, or simply by being here with us and believing in the project.

We’re going to release the pricing and release date of the open beta early this week with feature roadmaps. Stay tuned for the updates by joining our subreddit, Discord, following our Twitter, or checking our website for news!

Discord: https://discord.gg/novelai

Twitter: https://twitter.com/novelaiofficial

Reddit: https://www.reddit.com/r/NovelAi/

Website: https://novelai.net/