Building real-time applications is today standard. At Crisp, we used Firebase in production over 9 month, starting from day one. From the dream to the nightmare, here is our experience.

Note that this article covers the production aspect of using Firebase Realtime Database, and we still think that this solution still great to build MVPs, Hackathon Projects, or if you have your own reasons.

Edit: Firebase reached us and was very receptive to the issues we encountered. We are glad to see such an engaged team for their product. Our case is special as we have very specific needs for which we got to the limits of Firebase (which was by the way the pre-2016 Google I/O version). Nonetheless, this is an excellent service if all you need is storing user data and dealing with real-time issues.

The Project

Our Initial (Good) Reasons To Use Firebase

Crisp is a a dead-simple live chat focused on user-experience. We wanted to make a cost-effective solution so we decided to use a Socket.IO backend to deal with live-chat on websites and in the same time, using Firebase for our dashboard, all synced with AMQP on the backend-side.

This hybrid architecture allowed us to build the MVP in only 3 weeks, both the live chat and the dashboard, used by separate people in the same time, and get it to communicate with micro-services. This server-less MVP dashboard allowed us to focus on UI and UX rather than server code, and this was actually a good thing at that point.

We bootstrapped the product quickly and got to market early. Exactly what you'd want with an MVP.

Then Came The Storm

First few months went great, and then boom. We got featured on ProductHunt. We had a great growth, but with it came our first scale issues.

From a huge asset, Firebase became a nightmare, slowing down our execution. Building new features was deal-breaker and we had to think twice before releasing new things in production.

We discovered Firebase technical limits too, as we encountered performances issues when we scaled to 100GB+ Firebase traffic every month.

10 Reasons To Switch

1. Spaghetti code

Server-less, doesn't mean code-less! Using Firebase means that all your server logic is now running right in your web or mobile client.

In most of apps, you have to send welcome emails, process images (avatars, etc), deal with payments, and build your business-core features. You really don't want all those to be done on the client, as this can range from impossible to dangerous for your business.

These things may still be "hacked" using Firebase, but it means that you will have to add even more code to your web-app, and it could be a nightmare to maintain if you have a mobile app too.

Think about distribution: any database logic change results in client app updates. How do you deal with clients who didn't update? Is it a correct thing to do for your users to deactivate older clients to force update?

2. Integrating Firebase with micro-services is infernous

At Crisp, we use micro-services, and in many-cases you have to query a database, to get user information, IDs, etc.

Firebase can be both used from the client (eg: Web, mobile apps), but also from the backend (eg: NodeJS). You may query directly Firebase over the network, right from your backend, but you should avoid doing this because it is really slow at scale.

We used Redis to cache all these operations, meaning that we had to synchronize all the data back-and-forth. The micro-service doing this job (in other words - connecting to Firebase) had memory issues because Firebase caches data in memory and don't seem to release all unused references (we used the Firebase NodeJS library).

3. Pricing

Server-less doesn't mean cost-less. No, no, no: one day you will have to pay for your laziness. Indeed, we had to when we received this email from Firebase:

Firebase paid tiers come with a usage quota, plus overage fees if you exceed the plan limit. Your Firebase, crisp-<..>, has been exceeding its plan’s limits, and, due to an error on our end, you have not been charged overage fees. We will be correcting this beginning next week, which will result in your account being charged for any excess usage.

[..]

Thanks so much for being a Firebase customer!

Thank you for the mail, Firebase!

Paying $100 per month for something you can run on a $5 DigitalOcean droplet is something that get you to think twice when dealing with server-less code.

With your own server code, you will gain maintainability and productivity and you will have a cost-effective code-base.

4. Firebase downloads all subtrees on load

Assuming that you are building a Slack-like app, you will have to download all channels data on app load.

Some people will say that it could be something improved with pagination, but with Firebase, you cannot paginate because you cannot get query array length, you cannot paginate ordered arrays, etc...

5. You might have inconsistencies

Firebase supports offline operations. It's works like "git commits", but the main issue is that if you client goes offline and then online and you have concurrency on some input data (for instance, a shared notepad), you might have inconsistencies. Pretty much like Git merge conflicts.

6. The problem of data migration

With Firebase, you can't deal easily with data-migration like you can do with a simple SQL database, an ORM or ODM.

This means that you will have to do things like such:

if (user && user.new_subdocument && user.new_subdocument.new_property) {
  // Do stuff.
}

Resulting in safety conditions, everywhere.

7. Relations are marvelous

Dealing with relations with NoSQL is hard, dealing with relations with Firebase is pain in the ass.

For instance: an user belongs to a team, and a team has users.

User:

{
    name : "John Doe",
	team_ids : [...]
}

Team:

{
    name : "Acme Inc",
	user_ids : [...]
}

It means that you user has to watch team_ids, and then populate on your own teams and your team has to watch user_ids and populate users. This example is simple. Picture the whole thing with more relations, and you get a spaghetti logic.

8. Queues are buggy

To deal with server code and micro-services, Firebase introduced queues to share operations between server and prevent concurrency (eg: to avoid sending an email twice).

The NodeJS library implementing Firebase queues is named: firebase-queues.

This feature was not well maintained by Firebase teams, and we had several bugs, including synchronization issues, locks, etc.

Besides, queues were a scale bottleneck as queue items can be inserted (stacked) quickly, but consumed (unstacked) very slowly. If your connected clients insert more than your backend is able to process (which is seems to be limited by the firebase-queue library timers, and not your CPU or network), this will result in huge processing delays that are not acceptable when dealing with real-time applications. Furthermore, I've just described a potential DOS attack vector: connect to a client and flood the queue with records, and the whole queue tasks will be greatly delayed, such that the service becomes unusable.

So yeah, avoid Firebase queues.

9. You don't own your data

Beside the fact that your data is hosted on servers that you don't own, it's not possible to export your user data. You can't export emails, and user accounts are not recoverable (you cannot export user accounts with passwords).

Plus, it was not possible to export our data when we had hundreds megabytes hosted. We had to contact Firebase by email.

Notice: exporting data email/password data is possible by contacting Firebase Team, but not from Dashboard.

10. Complex queries are impossible

It's still impossible to query your database to find fields with some properties.

For instance, you want to build a Slack-like app: you cannot count unread messages, even if you have the proper structure that would allow to do so (eg: a read = <true|false> tag). The hack is to count all unread messages client-side, after having retrieved the whole dataset. It's a huge performance issue.

You can't perform operations to get active users, or doing batch operations to update documents having some fields.

Bonus. Want to build an API for your product? It's impossible.

Today, most apps expose a developer API. This is impossible to do with Firebase.

Of course, you could still remotely query your Firebase database upon receiving API requests from clients, but this would be extremely slow as the data is hosted remotely and the Firebase libraries are leaking memory (see above).

The second solution is to make an HTTP wrapper. All that said, in this case the best option is build your API with standard storage backend you host, using your own database systems (relational or non-relational). This is what we did.

Migration

Architecture

We decided to migrate to a REST API. To make things simple, we chose to use an old-fashioned SQL Database, which overs 90% of our data storage needs. It has relations, models, and it's perfect to make a bulletproof, simple, and maintainable API. You can read our API docs here

For the 10% part, we decided to use MongoDB to store messages and conversations at scale. We used UUIDv4 as primary keys, both in MongoDB (this is the de-facto standard in MongoDB) and into our SQL DB (this is not the standard in the SQL world) to make it transparent for everybody.

All CRUD (Create/Read/Update/Delete) operations are performed over the REST HTTP layer and some asynchronous replies (after some API requests) are send back over a Websocket layer (we call it our Realtime API).

Updates are internally forwarded over AMQP and then synced with a Socket.IO micro-service, and an authentication layer filters ressources that you are subscribed to.

Simple, effective, and sustainable. We are now free to switch from Socket.IO to another engine, or even migrating from MongoDB to another data store.

Schema

Let's Code

Started to build the API from Paw (a Postman-like app, for Mac), by mocking API routes and simulating things. Then, we made it working with real code. We used NodeJS but you could use PHP, Ruby or every language that you and your team are happy to work with.

At the same time, we removed all Firebase logic from our apps to remove all logic from the client and plugged all network operations over our API. The complexity should be implemented in the server.

These two parts took around three weeks, then came our migration scripts, migration tests, and production migration.

Let's Breathe Again

This migration allowed us to focus again on execution and several things were improved into our product:

Web app loading time were reduced from 20 seconds for some users with many conversations to 2.5 seconds (constant time, regardless of how many chats you have).
Features like search, pagination, and more are now possible because: MongoDB.
The code of our apps became simple again.
Making mobile apps became simple, too.
Migrations are now easy thanks to schemas and ORM.
We can query our data-bases to update so inputs when needed (applying a discount, etc).
We now have a developer API.

Thank you for reading, and now, think twice before using Firebase!