Own Data, Own Software

Summary

In this article I talk about "local-first", a paradigm for building software that runs on your device, stores data locally, and syncs across devices without depending on cloud servers. I contrast it with the "cloud-first" approach we're all accustomed to, highlighting the benefits of local-first for the user.

I then share my vision for Superego, a product I'm developing that combines a personal, general-purpose database with a platform for building local-first apps, aiming to create an ecosystem that champions the ideals of data and software ownership.

For the past couple of years I’ve been keeping an eye on a software trend that I find very inspiring: local-first software.

The essay that coined the term is a lengthy-but-worthy read, but, in essence, it characterizes a piece of software as local-first when:

  • It runs entirely on your device, where it stores the data it operates on (i.e., your data).
  • It syncs that data with your other devices.
  • It allows real-time collaboration (where it makes sense).
  • It’s yours forever: you’ll be able to use it in 20-years time, even if its developers are no longer around.

This contrasts the most common form of software we’re accustomed to nowadays - cloud software - which mostly runs on some company’s servers, keeps your data there, and is inextricably tied to that company.

I’m excited by the local-first approach, in particular by the idea to own and control my data and the apps that operate on it; they stay on my device and I always have access to them.

I’m a software developer, an obsessive tracker, and a nerd for stats. I very often wish that the data from the various apps that I use were somewhere on my computer, easily accessible, so I could write all sorts of scripts to prod it and reveal some curious hidden trend in my life.

Additionally, from a privacy point of view, thinking that my data is currently in the hands of so many companies makes me uncomfortable. It’s also preposterous that so many of them make me jump through hoops just to get a copy of it. Not to mention that, if one of those companies goes out of business, poof - my data with them is essentially gone, as - even if I do have a copy - I can’t do much with it without the company’s software.

Since I first heard about the concept, I’ve been mulling over the idea of a local-first software world: what would it have looked like had cloud apps never taken off in the mid‑2000s and had everything been local‑first instead?

Take my Garmin watch as an example. It continuously tracks my heart rate, my sleep, and I use it to record all my workouts. The watch connects to the Garmin mobile app on my phone, which gets the data out of the watch and uploads it to Garmin’s servers. When I want to look at this data, I must use the Garmin app (or their web app), which requests it every time from those servers. Of course I must also have an internet connection to do so.

Saving and accessing data in a cloud-first world.

In my local-first alternate timeline, instead, the Garmin app - after getting the data out of the watch - saves it in a standardized local data store that syncs with my other devices through some encrypted cloud service of my choice. When I look at the data, the mobile app (or the web app) fetches it from that local store. No Garmin server is ever involved.

Saving and accessing data in a local-first world.

In this scenario, Garmin’s mobile and web apps are local-first: in order to work they only need access to the local store. They don’t need me to “log in”; they don’t phone home to check if I can use them. I can always use them, as long as I have them installed. Even if Garmin shut up shop, my workflow would be unaffected.

Furthermore, since that local data store is standardized, it’s easy for me to access its data if I want to do something else with it. Say I want to check whether, during my daily run, my heart rate stays lower on those days where I got lots of sleep. It’s easy for me to write a script to run some stats. Maybe, since Garmin’s data format is - necessarily - publicly known, some other developer has already written an app to do that.

A few months back I started envisioning a product that could make my local-first dream a reality. Or at least a smaller, personal version of it.

I emphasize personal. The local-first crowd is very focused on real-time collaboration, but I actually don’t care that much about it; for most apps that I use in my private life, outside of work, I rarely need to collaborate with anyone. What I do sorely need (well, want) is that standardized store I was imagining where I can keep all my data. It could support collaboration, I guess, but if that feature adds a lot of complexity, I can do without it.

I identified six essential desiderata for my product:

  1. I want a local database, where I can put any data I want, in a transparent format that I can control.
  2. That data should be automatically synced between my devices and automatically backed up.
  3. The product should allow me to sync data from other apps and services I use.
  4. There should be a basic - but functional - UI for CRUD-ing the data.
  5. I should be able to easily extend the UI by writing snippets of code (e.g., a React component) that provide custom views of the data, or custom forms to edit it, to improve my user experience.
  6. The product should be open source.

Note easily in point 5: I want something hackable, like Emacs, that I can adapt to my needs with relatively little effort. Of course, this should still be a rare occurrence, so I want the default UI to be good enough for most uses.

Also, regarding data syncing and backups, I don’t want my data to go through some obscure or questionable service. I want it to be stored in something proven and widely used, like Dropbox or Google Drive.

Having found no existing product that matches all the requirements above, I naturally set out to build one myself: Superego.

I’m hard at work on it, and a first alpha is due in the coming weeks. In the meantime, though, I want to share something about it and see if there’s anyone else thinking “this is actually a great idea!”

To start, what exactly is Superego? In a nutshell, two things:

  1. A personal, general-purpose database.
  2. A platform for local-first apps that build on top of that database.

Let’s dive into the details.

In a way, you can think of Superego as an Airtable clone. Its interface allows you to define collections of documents - each collection having its own schema - and performing the usual CRUD operations on them.

Mockup of a collection view in Superego.
Mockup of a collection view in Superego.

Compared to Airtable and its many clones, however, Superego is designed for personal use, to keep data that is only accessed by you. Its overall UX is optimized for that use case. For example:

  • It’s just a desktop app. You download and launch it; that’s it. No hassle of self-hosting.
  • Its only user is you. You don’t need to “create an account” to use it.
  • It stores your data on your computer, so you can use Superego whether you’re online or not.
  • It’s fast, because again, data comes from your local disk, not through the network.
  • It stores your data as simple “files in a folder”, which you can sync and back up with any file-syncing service like Dropbox, iCloud, etc.

The personal database is the foundation of Superego and it’s a useful product on its own; after all, many apps are little more than CRUD interfaces on a database and Superego’s default interface can actually get you pretty far, offering you a few powerful widgets for viewing and editing data.

Example of Superego's default UI.
Example of Superego's default UI.

I expect the default UI to be good enough in most cases, but I also want Superego to be hackable and extensible, and that is done with superapps.

Superapps are bits of JavaScript that run inside Superego and extend its functionalities. They can, among other things:

  • Provide alternative interfaces to view and edit a collection or a document.
  • Run background jobs whenever a document is created, modified, or deleted.
  • Sync data from external services.

Two characteristics make them local-first:

  1. They run “on device” and most of their functionalities don’t depend on anything non-local. (Superapps that sync from external services are the obvious exception.)
  2. When you install them, they get saved in Superego’s database, so you can use them for as long as you keep them installed.

Suppose you have a collection of foods and their nutrition facts, which you use to track your calorie intake. The schema for the collection looks something like this:

Food:
  type: Struct
  properties:
    name: String
    servingSize: Quantity # Quantity type is defined elsewhere
    nutritionFacts:
      type: Struct
      properties:
        energy: Quantity
        protein: Quantity

According to the schema, Superego generates the following interface to create a Food:

Food document form generated by Superego.

It’s perfectly functional, but a bit clunky: it takes a lot of vertical space, so it’s difficult to read and fill out. Lucky us, in this hypothetical scenario someone made the 🥑 Superfoods app, which markedly improves the UX:

Food document form provided by the Superfoods superapp.

The app has been associated to the Foods collection, so - when you open a Food document - its logo appears in the app switcher on the top-right corner of Superego, which allows you to switch back and forth between the default view and the Superfoods one.

As I mentioned above, superapps are bits of JavaScript, so you can use any language that compiles to JavaScript (or WASM) to write them.

In the spirit of making hacking on it as easy as possible, Superego comes with high-level SDKs that expedite development. Without going into detail, here’s a couple of TypeScript examples to give you an idea of what the code of a superapp looks like:

// EXAMPLE: custom view for `Food` documents.
import { DocumentView } from "@superego/react-sdk";

export default DocumentView("foods", ({ food }) => (
  // The food prop is fully typed according to the collection schema.
  <div>{`Name: ${food.content.name}`}</div>
));
// EXAMPLE: background job that runs image classification on `Photo` documents.
import { OnDocumentCreated } from "@superego/background-jobs-sdk";
import { extractFeatures } from "./image-classifier.wasm";

export default OnDocumentCreated("photos", async (photo, superego) => {
  const features = await extractFeatures(photo.content.original);
  // Updates the photo document with the features extracted by the classifier.
  await superego.documents.createNewContentVersion(
    "photos",
    photo.id,
    photo.contentVersionId,
    { ...photo.content, features },
  );
});

I’m building Superego according to my wishes, to be the product that I want, but of course my hope and my ambition is that it will also fit the needs of others.

My first target audience is software developers like myself, who might be drawn to Superego for its idealistic goals (as a group, we are generally more sensitive to data ownership issues) and for its hackability.

In my vision, Superego’s success will stem from these early adopters. They’ll start writing small apps for themselves, to scratch personal itches. Some of those apps will be shared with the rest of the community and slowly an ecosystem of useful tools will develop, which will eventually make the platform appealing to a wider audience of data enthusiasts, privacy concerned, and open source lovers.

But, even in the best-case scenario in which Superego becomes somewhat popular, I don’t actually see it as a product for the general public, so that’s not a really goal and Superego’s design choices are not influenced by it.

Speaking of non-goals, here are some other things that Superego doesn’t strive to become:

  • Collaborative. It turns out, collaboration does indeed add a ton of complexity and I want my efforts to go towards solving other problems.
  • A platform for inherently-online apps, like a chat or a flight tracker. It just doesn’t make sense.
  • A platform for non-data-centric apps, like a game or a calculator. These apps can live happily elsewhere.
  • A platform for complex specialized apps, like an IDE or a video editor. Though they might operate on personal data to a certain extent, they require foundations that Superego can’t provide.

I see Superego as the ideal platform for relatively-simple apps like personal trackers of all sorts (meals, periods, expenses, etc.), task boards, calendars, note takers, and virtual whiteboards.

Not that we don’t have plenty of those already, but they’re usually either proprietary cloud services or they require self-hosting. Also, from a data point of view, they’re all silos. With Superego all their data - your data - would be collected in one place, always at your fingertips. Yours, or perhaps the ones of a local AI assistant, but that’s a musing for another day.