# That One Time I Almost F**ked Up inko.cat

This is too crazy that I have to document it. It is Nov 7th, 2023, at 3:33AM. I almost shat my own pants just now.

I have been working on this personal project, inko.cat, since the end of August with my good friend Guo. It is a task management app, a feel-good planner app. Two days ago, my sister wanted to give my app to her coworker, Ling, to give it a try. Afterwards, she had some pretty good constructive criticisms, so I took the chance and reached out to figure out more about how to improve inko.cat.

Inko Cat right now has around 10 users, yet most of them (me included) are loyal customers to this app, having more than 100+ tasks tracked/tracking on the platform. Ling had around 10 tasks, and she said she would give inko a try in the next couple days and let me know how it goes. To further sell inko to her, I told her that there was this AI feature that allowed natural language to be turned into task items.

Simultaneously telling her that, I was working on encrypting all task titles. I've put off this task for a while but I don't know why I thought it was a good idea to knock it out before bed. It wasn't too hard, and I deployed it without much thought. The thing is, in the AI code where natural language is processed, there's this one part of the code that would encrypt the title first, and then decrypt it, and then save it. I know it's complicated, but the result of it was that after Ling gave this feature a try, even though most of the database was encrypted, Ling's new task items weren't. This contamination seemed easy to fix, all I had to do was remove all the encryption markers on ALL tasks, and then encrypt them all including Ling's new ones... or so I thought.

I quickly modified the original migration script, removing all `__enc_title` markers, and encrypting every tasks again. I deployed, and I was about to go to bed. The result? Fucking horrendous


> The attached image is a recreated screenshot. I was too busy shitting my pants, didn't think about taking any screenshots.

Basically, what happened was, by removing all the markers, I essentially told my code that ALL task titles are unencrypted. 98% of the originally encrypted task titles were now all marked as unencrypted. Next, I encrypted every task titles again. What this did, was that those "080f6349c76b..." supposed-to-be encrypted titles are now encrypted again. And, Ling's new task titles are also encrypted, but only once. Seeing a bunch of random charachers on your own damn app is something I've never imagined to be so scary. Especially, on production, I shat my pants.

I panicked, quickly ran "mongodump blablabla" to backup the database once again before I fuck things even further. Stupid of me again, I didn't change the filepath to the mongodump results, what this did was mongodump OVERWRITTEN THE LATEST BACKUP I DID TWO DAYS AGO...

Now comes the best part, in midst of my panic, I shut off the deployment, changed the MONGO URI to prod on my development environment so I can see the production tasks on my local environment, and started writing a migration script. The plan I came up with was quite clever in my opinion:

- Decrypt all tasks (undo the latest encrypt)
- Decrypt again (but in a try/catch to catch those that can't be decrypted, this undoes all tasks other than Ling's new ones)
- Encrypt task titles again

To an experienced developer, this may be nothing. But to my shitless brain 20 minutes ago, this was genius. Repeatedly I ran console.log to make sure everything was OK, and then I uncommented the `save` to update them all with the right data. Railway database scared me again by throttling my updates, but I patiently waited, and eventually everything was a OK. Took about 5 minutes.

## Key Takeaways

Seriously, back up your database. That makes you feel so much better when you fucked up. I just added a couple more action items to my own inko.cat:

- Write a mongodb script to incrementally backup prod database
- Document how to use `mongoose-field-encryption` to avoid this type of situation again
- Back up the database again


This is my first time deploying something that people are using, and this is also my first time fucking up something that people actually use. I'm glad that inko is still small. Imagine with 1000+ people using it globally, fucking it up would really mean I'm fucked. I know that there are better ways of handling situations like this, and I'll do some research tomorrow.

It is now 3:53AM, I'm going to bed.

## `step22Temp.ts`

This saved me...

import { getDb } from '@/server/models/mongoose';
import { TaskModel } from '@/server/models/schemas/task';

export async function up() {
  const db = getDb();

  const taskIdToEncryptedBlocksJsonMap: Record<string, string> = {};
  const taskIdToUpdatedTaskDataMap: Record<string, any> = {};

  for await (const task of db.collection('tasks').find()) {
    // @ts-ignore
    taskIdToEncryptedBlocksJsonMap[task._id] = task.blocksJson;
  }

  for (const task of await TaskModel.find({})) {
    // @ts-ignore
    task.decryptFieldsSync();
    task['__enc_title'] = true;

    try {
      // @ts-ignore
      task.decryptFieldsSync();
    } catch (e) {}

    task['__enc_title'] = false;
    // task['__enc_blocksJson'] = false;

    // @ts-ignore
    task.encryptFieldsSync();

    const taskBlocksJson = taskIdToEncryptedBlocksJsonMap[task.id];

    task['blocksJson'] = taskIdToEncryptedBlocksJsonMap[task.id];

    const updatedTaskData = {
      __enc_title: true,
      __enc_blocksJson: true,
      blocksJson: taskBlocksJson,
      title: task.title,
    };

    console.log('updatedTaskData', updatedTaskData);

    taskIdToUpdatedTaskDataMap[task.id] = updatedTaskData;
  }

  for await (const task of db.collection('tasks').find()) {
    await db.collection('tasks').updateOne(
      {
        _id: task._id,
      },
      {
        // @ts-ignore
        $set: taskIdToUpdatedTaskDataMap[task._id],
      }
    );
  }
}