Bento: Self hosted documents - A postmortem

When I started working on bento it was to meet an need that I felt wasn't being met with what is currently out there. This is a pretty common feeling for me with my softwares, I can always find an issue with what already exists. There isn't anything exacly like bento but I learned a lesson about finding the correct search terms to establish this.

What is Bento?

Bento was designed to be an ultra light weight document editor for personal or family use. It wouldn't have LDAP integration, it wouldn't have SSO, it wouldn't even be encrypted. I had a handful of guiding principals that I was sticking to:

  1. Cross platform - Both a web app and a native mobile app (Never got to the latter)
  2. Stable - My wife will be using this so I want it to not be down all the time (unlike plex...)
  3. Light - This will be running on a Raspberry Pi. It should be very light so as to fit inside each environment easily.
  4. Scriptable - I want to connect automations to this so it needs to be scriptable via REST

But the heart of Bento was it's diff based document editing. I wanted a fancy collaberative editing experience. I knew I was merely mortal so trying to rival Google docs for example (which Microsoft seems to be struggling to do) was out of my reach. I decided to depend on existing technology in the form of the "unified patch" system that linux developers have been using for decades. When a user makes a change, not the entire document but rather just the patch of what changed is sent.

But most importantly bento was designed for private use. By an individual or a family. It's name comes from the japanese Bento Box, which is effectively a lunch box, but most importantly a lunch box for a single person. Not a picnic, not a buffet, a single meal.

Why not these other options

There already exist plenty of options out there for self hosted documents, so why make another one? I'll give a quick glance at the options I found while I searched and why I didn't decide to do them

Collebra

Crypt PAD

Only Office

SmartOffice

Etherpad

What went right

I created a functional collaberative editor. My scheme of using diffs to keep things updated worked. It didn't work perfectly but it did work. While I didn't ever get around to having a mobile app, I was planning on doing a progressive web app which would allow me to effectively turn the web app into a mobile app, so I was aiming to make the web app mobile friendly to begin with and then just have te ability to install that on your phone.

So what were the problems?

Diff based updating

This was the clever trick I built Bento on, and I just said it worked, but it wasn't perfect. That ended up being a bigger deal than I anticipated. There were plenty of instances of the document jittering. Times that I would type something and (exactly) two seconds later it would erase it as it thought it was out of sync with the server. Some times it wouldn't realize it was out of sync with the server and never update. This mattered because I was using bento as an inter-computer clip board. So if I pasted something into bento and it didn't show up on the other computer it was defeating it's purpose.

I decided to use existing code for creating and applying the patches. But this ended up meaning I was using two different programs for one process. One written in python, one written in javascript. Any slight differences in their implementations lead to headaches and complications. If I had to do it again I would write my own diffing code, even though it would take extra effort. The reason being is I would have a more through understanding of the code and could debug it better.

This also lead to issues with the "undo" function. I was going to have undo be built into the program via reversing diffs, but it turns out that was going to be very difficult to do if I just use off the shelf editors that have their own undo features built into them.

And so, if I were to do it again I wouldn't include this feature. I would just depend on a more traditional "Save" button and push the full document. If there was a collision it would create a copy. That would be much simpler. I ended up getting kinda bogged down in this collaberative trickery.

Patch work of hacks

Here's the plan: I create a way of communicating between existing editors, like TiniMCE, and a custom server back end. I don't have to make the interface, just the server, nice.

Yea, not so much. Turns out there is a lot more to the front end of Bento than just an editor. Setting aside the file browser that was necessary to create and focusing on just the editor screen for a moment: There is of course the editor on the screen, but there is also the title of the document, how do you edit that? There are the menu buttons for saving, downloading, closing the document. There are a bunch of random little details like this that made it challenging to implement even the basic text editor let alone my objective of a spread sheet editor.

Realize how your scope has changed

I started this project thinking I was making a self hosted document editor. And it is, but it is more accurately a self hosted markdown editor. The important thing here is that there are already a lot of projects out there that are very similar to what I was making. Enough so that I could have potentially accomplished my objective if I had started with one of them as a base and added the features I needed for my environment. So I may come back to Bento, but I'm going to start by, for example, taking Hedge Doc and adding a spread sheet editor to it rather than creating an entire project from scratch.