Search IconIcon to open search
     . _
._   \/ @
@ \._/

Rhizome Proposal

Last updated May 26, 2022

A proposal on building infrastructure for collaborative local-first applications.

Perhaps the current episteme is best rendered as a rhizome: a subterranean plant stem that can shoot out roots that grow, hydralike, even when snipped in two… a system without beginning or end, “always in the middle, between things, interbeing, intermezzo.” –Claire Webb in Noema

Privacy and security in this world mostly means “which private company do you trust with your safety?” The answer often coincides with who has the largest walls and deepest moats.

Commercial software, especially aggregators, are incentivized to resist interoperability. To be composable is to be commoditized. Interoperability means you no longer have a data lock-in moat, or a privileged hub position in the network.

When you have a universal API for composition each additional tool increases the number of possible workflow combinations by n * (n - 1).

Unix philosophy: Expect the output of every program to become the input to another, as yet unknown, program.

For example, lego rigorously defines the interface between pieces (“the dot”). If they didn’t, each piece would have to make up its own mind about the kind of connector it should present.

# Motivation and Values

Imagine a web where you can bring all of your data from Roam to Obsidian or Google Drive to Dropbox without needing to worry about how to make the file types work or massage the output of one API into another.

Imagine a web where your applications don’t just stop working when you lose internet connection or some company intern accidentally takes down production.

Imagine a web where it is easy and normal to create vast and rich collaborative spaces that allow you co-browse the internet and collectively digital garden with friends.

Imagine a web where your digital spaces feel like portable universes and community gardens.

Rhizome is an attempt at infrastructure for a world where these are possible.

Collaboration at scale while keeping local conditions in mind

Below are properties that Rhizome will optimize for:

I’ve written about these in greater detail regarding Rhizome’s Philosophy.

# Existing problems with peer-to-peer protocols

Many peer-to-peer protocols already exist today, claiming to give people the ability to own their own data. Yet, none of them have seen large-scale adoption with the exception of a few social media platforms.

Reflecting on this, there seems to be 3 major hurdles that no single protocol/system has been able to overcome:

# Availability + Durability

In most P2P apps nowadays, closing your device or disconnecting it from the internet means the end of a session and whatever resource your peers were also connected to is no longer available.

If you need to send someone a file or message, both devices need to be online at the same point. If you need to download a really obscure file through a torrent, the chances that someone is currently seeding it are extremely slim.

Large companies get around this by storing the state of a user’s documents on one of their many servers who make it available on your behalf but P2P apps do not have this luxury. The problem is that most people do not have a device that is “always-on” like a server is.

This poses a large problem for emulating those smooth ‘always available’ experiences that we’ve grown accustomed to in modern web apps like Google Docs.

# Connectivity

The current Internet, with its NAT routers, firewalls and VPNs, are hostile to P2P connections. Even the best techniques we have to establish direct peer-to-peer connections with other hosts work only about ~85% of the time (see notes on hole-punching efficacy). Approaches like DHTs are promising but no one has got it to work consistently in a browser yet.

Peer-to-peer connectivity is hard without an intermediary.

Of course we can always fallback to a trusted server to act as a proxy but this comes at the same price of decentralization. Lots of protocols provide their own signalling and rendez-vous servers you can run but people don’t want to run/host/maintain their own servers either!

# Identity

Most P2P protocols don’t have good primitives for identity. Most users are identified by whatever session they are in along with a sequence of random numbers or codes.

Once that session is terminated, that notion of identity dissolves. Often, there is no way to identify a user across applications. Sometimes this is useful but not for vast majority of use cases.

Most apps require some notion of persistent ‘identity’ in order to function properly.

# Why not Blockchain?

At this point, some critics may be screaming “why not just use blockchain??”

I admit that it is true that blockchain actually solves most of these problems. Blockchain approaches have great approaches to solving both identity and availability through a combination of wallet addresses and token incentive mechanisms. Yet, they solve it in a way that leaves much to be desired.

Blockchain causes a whole new set of problems that makes it quite cumbersome to build on top of it – partially why I suspect there has yet to be a widely-adopted real-world application of web3 yet. Some of the core problems that I have personally seen are:

All of these make it incredibly unfeasible for data-intensive or real-time applications (e.g. file sharing, games, collaborative text editing) without aggressive application of blockchain scaling ideas. Of course, there are certain applications that benefit from the unique properties that blockchains possess (namely strong guarantees about consistency and message ordering) that make it worthwhile for certain applications like cryptocurrencies, but for most applications these tradeoffs make no sense and eventual consistency in a fair-loss crash recovery system model is more than good enough.

Blockchain is suitable for a very small subset of use-cases. Is there a more general purpose technology that still addresses these main problems?

# So, what is Rhizome?

Rhizome aims to be a data-persistence and identity layer for the distributed web.

It is an abstraction on top of IPFS (specifically IPLD), Filecoin, and the Raft consensus protocol.

The goal of Rhizome is to build out the model of the personal cloud. Think iCloud or Dropbox but distributed and extensible. It is the one database that is the source of truth for all other applications you use. You control it, nobody can take it down because it runs primarily on devices you own.

In this model,

At this basic level, Rhizome is a local-first data replication and synchronization service much like iCloud/Dropbox. This on its own is already a promising idea but where it gets interesting is how application data is managed.

Of course, this then brings up a question about how users are going to be able to provide the necessary compute and storage and uptime? Nobody has a set of devices they keep on all the time.

We can solve this with an always-available cloud peer, a companion add-on to the sometimes-available personal devices we have. A cloud peer is not a hosting provider, it is rather a different type of a personal device. It does not have a screen, but it is capable in a different way, it complements our personal devices with its high availability, storage, and compute.

Rough architecture diagram as of May 6th

The properties work together to make a solid foundation for peer-to-peer applications to exist and thrive in the future.

# Differentiation from existing work

# Output

# Research artifacts

Blog posts explaining distributed systems concepts as I learn and become more familiar with them

# Root

The data replication and identity part of Rhizome

# Trunk

The application-level event log management and collaboration

The goal is to finish the major research artifacts and Root by end of summer.

You can find the ongoing research log here.

Interactive Graph