Given their start point (a product unfit for public consumption) that is the absolute minimum they need to do.
I at least will never trust Mongo for anything but a toy project. There are so many better options out there, options whose technical capabilities are as good as Mongo's marketing.
I think if you look back objectively, there are very few database platforms that were absolutely "fit for public consumption" right out of the box. Look at all the SQL Server shops out there (mine included) that won't even roll out a new version of SQL Server until it hits SP 1 at a minimum... For MongoDB, If you look forward based on what they are doing now rather than at how early adopters may have had a sub-optimal experience way back when, you'll see a mature product that is consistently improving and is demonstrably reliable. Can you give an example of another option you are referring to?
Right out of the box? Mongodb has been trying to get it right for 10 years now. Kyle says the storage engine they've used for most of that lifetime is fundamentally flawed, and they've only now, a decade on, managed to write something without known bugs to replace it. And maybe this time it's ok. Maybe this time there aren't any more layers of buggy crap in mongo yet to be found and fixed.
Maybe. But you'd have lost that bet if you made it any day in the last 10 years. And in those 10 years mongodb has demonstrated again and again that they aren't up to the task of writing a reliable database. Even with their new storage engine they couldn't find the bugs alone.
I think using mongo today for any mission critical data is an irresponsible choice. I'd seriously question the judgement of any senior engineer who picks it for a new project over rethinkdb or Postgres.
>"Kyle says the storage engine they've used for most of that lifetime is fundamentally flawed, and they've only now, a decade on, managed to write something without known bugs to replace it"
Didn't WiredTiger Inc write the new WiredTiger storage engine before they were acquired by MongoDB Inc?
Do you think MongoDB is a good choice (given how easy it is to use) when you only care that 99.999% of your data that you insert should end up in the database? That's my use case. Best-effort integrity. I mostly just want a DB can insert and query fast for documents and am not really concerned if I lose a few documents here and there.
It's about making tradeoffs. If MongoDB works for you (I actually enjoy using it tremendously) then I have to ask myself am I ok with its non-perfect integrity. For my use cases this isn't a problem. I'm not working with customer data or anything where losing a few records would make any difference at all.
How do you expect to check the end result? The article's Jepsen analysis shows that both the v0 and v1 replication protocols (excepting the very latest version of v1 that appears to be in response to this) can result in acknowledged writes being lost. I.e., the DB tells you, for a write sent with a majority, that the write was successful — to a majority! Subsequently (and, if I understand the article, possibly not immediately), the write can be lost.
Given a small cluster of reliable nodes on a reliable network, these errors will occur extremely rarely. So rarely, in fact, that they'll be written off as "user error" by support.
If you're a startup building a system which has to quickly and reliably scale from 3 > 3000 nodes in a year then the whole thing is likely to explode in your face. Twitter style.
Now, if MongoDB was so superior that it was truly platform which would even enable that kind of scaling, then the decision is simple: just go for it.
The thing is, this isn't how the world works. When systems are built, very few people consider (or are capable of considering) the growth of the system. Frameworks and database are, by the rule, chosen arbitrarily. When scaling happens, the question is more "how can we scale what we have whilst having everything kind of work" than "how do we design a system which works correctly at scale".
Mongo's whole strategy is based around this. Make Mongo the default choice for the current generation of developers.
Fantastic market strategy, but it's still snake oil they're selling.
When you talk about growing, the biggest value in Open Source has been that you can start with something free but shit, and then as you make money then you can spend it on customizing that Open Source in a way that benefits you.
However there exist commercial offerings that are (and were) faster and better at MongoDB than MongoDB was: KDB could've handled Twitter, we never would've seen a fail whale, and it is a whole hell of a lot cheaper than the developers and the customizers, and the headache, and the fact that you're making something open source which ultimately benefits your competition.
Another way to think about it is by thinking about experts: If you've got a great startup idea, why would you want to make your odds 10% worse by introducing the possibility it'll fail, by using the cheapest hacky hack thing that has 10% chance of losing your data? Ask experts with data, and be honest with your budget and you'll do a lot better.
I have some actual experience with KDB and MongoDB so I'm going to have to call bullshit.
How does KDB handle replication and failover? Or even high insert/update rates to datasets that exceed the size of memory? How do you shard KDB?
KDB doesn't support unicode text. Do you plan to only have English speaking users?
Yes, KDB excels at its relatively well defined niche of transforming and aggregating "smallish" (say 10 TB or less) numerical time series data. It would be a horrible choice for the backing store of a high throughput CRUD application...
What is it with KDB zealots thinking that KDB is the best database for every task? I swear, KDB is the Scientology of databases.
With -r and (in my case) SO_REUSEPORT. Most people use a dedicated gateway (have seen custom tomcat stuff and haproxy).
Meanwhile, MongoDB doesn't actually replicate reliably (acking then losing anyway) and failover can crash cascade in the naïve configuration.
> Or even high insert/update rates to datasets that exceed the size of memory?
This is literally the KDB tickerplant model. Have an RDB that flushes out regularly (daily) to an HDB.
You can also just write to a log `:log upsert ...
> How do you shard KDB?
Same way you shard anything else? By picking a key and directing the query to the appropriate server. h[(first md5 k) mod count h] "query..."
> KDB doesn't support unicode text.
UTF8 is fine.
The number of times I've needed the first 5 code points (and not the first 5 bytes or the first 5 characters) in my life is zero. All that half-baked Unicode support in various languages (like MongoDB) just makes people think that they've solved a problem that they really haven't.
> Yes, KDB excels at its relatively well defined niche of transforming and aggregating "smallish" (say 10 TB or less) numerical time series data. It would be a horrible choice for the backing store of a high throughput CRUD application...
I use it in one of those big CRUD databases (digital marketing and tele-lead tracking).
> What is it with KDB zealots thinking that KDB is the best database for every task? I swear, KDB is the Scientology of databases.
Because it solves problems they have.
Even when I don't use KDB I use a similar architecture because it's the correct architecture, because I've had these problems for a lot longer than I've had KDB.
If it doesn't solve every problem I have, that's because I have work to do, not because it isn't great at the problems it does solve, and I don't shout at my hammer because it isn't a spoon.
However MongoDB doesn't solve any problem I've ever had: I've never needed a bag of objects/filesytem that loses data, or a binary blob that I cannot query. It's so famously "web scalable" it has made a joke of the very idea of being scalable.
> With -r and (in my case) SO_REUSEPORT. Most people use a dedicated gateway (have seen custom tomcat stuff and haproxy).
So as a KDB user you need to implement your own HA solution. That is strictly worse than MongoDB replication, even with its now-fixed bugs. Do you really think your homemade multi-master KDB system would pass Jepsen?
> This is literally the KDB tickerplant model. Have an RDB that flushes out regularly (daily) to an HDB.
Wat? That only works if data is immutable once written. Tweets are liked/deleted/etc. You could store an immutable log of user actions, but then you would have to reconstruct the current snapshot every time someone loads a timeline. It's entirely possible for someone to like/delete/RT an old tweet. Financial data is naturally partitioned because the order book clears at the end of every trading day - this doesn't apply to CRUD apps.
> UTF8 is fine
I think you misunderstand what I mean by unicode support. Does KDB support locale specific collations? Does it support normalization/canonicalization? Being able to index by code point is about 1% of the needed solution to build an i18n-proof product. Obviously that doesn't matter when you are dealing with normal KDB datasets like market data where e.g. asian names are represented with numbers.
> I use it in one of those big CRUD databases (digital marketing and tele-lead tracking).
Were you using it to store clickstream data? Or some other kind of immutable stream of events? That isn't really applicable to general CRUD applications.
Like I said - KDB is great for analyzing immutable streams of events. It's not a general purpose database for building CRUD applications. MongoDB tries to be a reasonable enough solution for many use cases, while KDB focuses on excelling at a small number. Both are valid approaches to building a database...
> So as a KDB user you need to implement your own HA solution. That is strictly worse than MongoDB replication, even with its now-fixed bugs.
You simply cannot take MongoDB in its (near) default configuration, put it on AWS, and handle Twitter volumes.
I think "this broken tool is better than your working tool" represents a certain kind of madness that I can't argue with.
> Wat? That only works if data is immutable once written. Tweets are liked/deleted/etc. You could store an immutable log of user actions, but then you would have to reconstruct the current snapshot every time someone loads a timeline. It's entirely possible for someone to like/delete/RT an old tweet. Financial data is naturally partitioned because the order book clears at the end of every trading day - this doesn't apply to CRUD apps.
I don't know what your experience level is, but Financial data typically has many subscribers to that tickerplant, and build-up indexes an views representing the queries that consumers are actually going to be interested in. This is covered in the most basic of KDB tutorials[1]
When a user loads a timeline, ideally you want to hit a single machine on a single query to a machine near the viewing user. Processes representing tweet consumers subscribe to the ticker plant, and build-up the indexes of what information they're going to need to publish. You're also going to need to have a fast index-by-publisher as well, so that when a subscriber wants to follow someone, we don't need a replay -- again, more indexes, but at least these can be "centrally" located.
This isn't even a remotely difficult problem to solve with the right tools.
> I think you misunderstand what I mean by unicode support. Does KDB support locale specific collations? Does it support normalization/canonicalization? Being able to index by code point is about 1% of the needed solution to build an i18n-proof product. Obviously that doesn't matter when you are dealing with normal KDB datasets like market data where e.g. asian names are represented with numbers.
If I misunderstand you, it is because you are unclear.
JavaScript, C and C++ don't actually support "locale specific collations" even though there are well-maintained and well-distributed collation and localisation libraries that people can use.
That "iasc" doesn't know the difference between Chinese and American spellings for a word is irrelevant. I can solve the problems I have with my tools, and building sort keys on my symbol tables for each locale means that the user-visible aspects of sorting remain instantaneous, instead of being tricked into doing stupid shit like x.toLocaleString(user.getLocale()) which is slow at Twitter scale.
This is part of what I mean by "engineered correctly": The tools that are available to us can trick us into thinking certain problems are solved when they aren't.
> Were you using it to store clickstream data? Or some other kind of immutable stream of events? That isn't really applicable to general CRUD applications.
Tele-lead means (outbound) phone calls for the purpose of lead-generation, so I have phone calls and the results of those calls in KDB. It's not "big data" by any stretch of the imagination.
> Like I said - KDB is great for analyzing immutable streams of events. It's not a general purpose database for building CRUD applications. MongoDB tries to be a reasonable enough solution for many use cases, while KDB focuses on excelling at a small number. Both are valid approaches to building a database...
MongoDB is not a valid approach full stop: Build dogshit and then try to pepper over the bad press with "the new version isn't dogshit anymore" every few years is negligent at best, and pays dividends with the fact that it makes it easy to identify inexperienced engineers.
That KDB is not as accessible as MongoDB is Kx's problem, and not KDB's problem.
I replied to a sibling with more details. Indexing by code point is only the smallest (and easiest to solve) part of the problem of dealing with non ASCII text.
You check the result with getLastError which, as you described, can be used to ensure a majority agrees with the write. But you normally don't use getLastError that way. Because a majority might not even be concerned with that particular write. They are, after all, shards. Instead you check if primary got the write. If primary disconnects while you are checking, you catch the exception and try checking until a new primary is decided. And if your check result is not ok, you try inserting again. That's as reliable as it gets when inserting to any database including SQL databases that support transactions.
> I'd seriously question the judgement of any senior engineer who picks it for a new project over rethinkdb or Postgres.
... you mean RethinkDB, whose future is still uncertain? Regardless of technical merits, the currently unstable future of RethinkDB means a senior engineer should be extremely cautious about selecting it for a significant project.
To be fair, choosing a scalable database even for a senior engineer still requires quite a bit of very specialized knowledge in distributed systems that most simply don't have. So they have to rely on what "feels right" anyway, rather than making an engineering decision, and are very susceptible to all the marketing and PR and authoritative opinions. There are no right choices for them. Although if in doubt everyone should probably default to a dynamo-style db, as it forces you to think about and organize your data in a certain future-proof way, which actually excludes all of the mentioned databases.
Not the OP, but RethinkDB is superior in many ways including stability, integrity and the feature set for pretty much every use case you would consider using MongoDB.
But with Jespen tests MongoDB can finally be considered a contender. Its not like competent teams were using it in production. Right?
Do you not consider Stripe to be a competent team?
Please, name some F500 companies using RethinkDB to power critical infrastructure. There are many using MongoDB. While Rethink is widely renowned among the HN set it is nonexistent in comparison when looking at actual deployments.
The reigning HN view of MongoDB being a buggy mess is outdated. Yes, they overmarketed a buggy project in 2009. It didn't matter, because they built a product that developers loved (and continue to love) to use. RethinkDB didn't aggressively market itself, and look where it is now - defunct. Mongo used that momentum to raise money and hire an incredible engineering team, including Keith Bostic, one of the fathers of Unix, and Michael Cahill, the inventor of the transaction isolation mechanism used in Postgres. Sometimes you need to employ aggressive business tactics to get to a point where you have the engineering resources to build a world class project. Moreso when you need to catch up to millions of man hours spent building Oracle and MSSQL.
I should note that I work for a multi-national gaming company and we use software that is ABSOLUTELY not fit for purpose, but once you have a hard dependency on something and the cost of muddling through is _less_ than the cost of a rewrite then you're going to be stuck supporting it.
I don't think it should be taken as given that there's a correlation between competency and the size of an organization that a team exists within, and I don't think such a correlation, when combined with large organizations' usage of MongoDB would challenge the assertion that there exists an anticorrelation between team competency and use of MongoDB.
Looking at the numbers, larger organizations straight-forwardly seem like they should be more likely to eventually hire mediocre talent, survive despite having done so, and more likely to have adopted any given tool.
I think you're looking at it wrong. It's not a popularity contest; I've seen billion dollar companies use fucking stupid tooling as well, but they still have the right processes where they don't lose data.
In non-tech centric large orgs, it seems you frequently do not have the talent required to be be both risk averse and productive at the same time, so "IT" becomes a risk averse and non-productive political structure from which springs an "alternative IT" rebellion group (if the initiative is lead from above) or many little cowboy teams (if the initiative is driven from below) and these will be "productive" at the expense of having no processes for avoiding stupid risks that, amongst other things, can lead to data loss.
The point? How do you draw a line from "Fortune 500s use Mongo" to "Mongo is used by competent teams"?
I specifically challenge that you can do so, given a model of a large organization as being necessarily more diverse (regressing to the mean of general competency, more likely to have facts like "org Y uses tech X" being true) and more robust to survive failures as it grows.
We don't have data, but we can still model (if nothing else, to think about what data we would need).
> Its not like competent teams were using it in production. Right?
I've heard that about 1000 times a day for 6 years. Usually the person stating it is snickering as if they are clued into some unknown secret.
Mongo does work in production at many shops, and in many forms. Sometimes it's used as the main database, sometimes it's used to house specific slices of data, etc.
As a sysadmin, I got tired of the devs constantly ragging on Mongodb (the same folks that selected it before I was hired). I eventually got fed up and said: "why do we use it if you all hate it so much. Let's replace it. What do you want to use instead, it's easy for me to set up something new". Cue everyone going "ah, it's not so bad, really..."
MongoDB is the Nickelback of databases: a reasonable act that's not going to blow your socks off, but one where saying "OMG I hate it!" somehow signals membership to some cool clique of connoisseurs.
alternative anecdota : I rewrote a backend that was using mongo, moved it to postgresql+postgis. Solved an ever-expanding RAM issue and is still blindingly fast [ on SSD hosting ]
The main win was not server stability, it was having general tools to manage data .. including the inbuilt geo-algorithms that come with postGIS. eg. I could make our data set 9x smaller by smoothing map paths.
I still love the Mongo api, but I just cant risk it with data on projects that people are paying for, or that I need to support.
[ I think the real sweet spot will be deep integration of javascript and json into postgres - so I can write stored procs in js, get db events in js, wrangle json fluidly.. all of which is improving. ]
>OMG I hate it!" somehow signals membership to some cool clique of connoisseurs
You do realize you are the one dragging identity into the mix.
I dont understand how this is a big ego debate. MongoDB isnt that relevant and it isnt Nickelback, its a homeopathetic database. That is: if you use it for something else than caching (store data you cant afford to loose, or load balance and use the database as main mutex to deal with all concurrency issues) that would arguably be a very irresponsible choice.
Hell i hate most databases, because its hard to get right yet some have interesting trade-offs (ElasticSearch, Cassandra, CockroachDB).
And its not a subjective or even analog discussion where databases are more or less consistent or more or less durable. They fsync or they dont. They use raft with majority consensus or they dont.
If you as a sysadmin judge these emperical facts based on your prejudices about the sort of people that would agree or disagree with you than you are much more like the cool clique of connoisseurs than the people at the other end of your finger.
Its engineering, not wine tasting. The shape of the world isnt a subjective thing anymore than the durability of a database that doesnt fsync.
Well, I'm a member of the MongoDB is worth quitting over club. I replaced it at a former employer with postgres. Eng waved bye to an endless stream of operational issues, and customers saw better uptimes and much much faster responses.
Rethinkdb is still the new database on the block, and never really found it's feet. Look up Postgres, Cassandra, Kafka, Riak, MySQL/Mariadb or MSSQL. (Or dare I say it, oracle). All of those tools have a long history of reliability and solid engineering.
You're comparing ISAM/MySAM (storage engine) to the MongoDB replication protocol. As a more relevant parallel MongoDB also replaced its original storage engine with one acquired from WiredTiger (BerkeleyDB founders).
One big difference from a corporate strategy perspective is that MySQL let the replacement storage engine (InnoDB) fall in to the hands of Oracle. MongoDB was smart enough to make sure that they were the acquirer, which puts them in control of their own destiny.
If MongoDB is heading along the path of MySQL, that's a pretty good path to be on considering that MySQL is used as the store of record at Facebook, Twitter and some parts of Google.
> You're comparing ISAM/MySAM (storage engine) to the MongoDB replication protocol. As a more relevant parallel MongoDB also replaced its original storage engine with one acquired from WiredTiger (BerkeleyDB founders).
My bad, MMAPv1 vs WiredTiger although I think it was obvious what I meant.
> One big difference from a corporate strategy perspective is that MySQL let the replacement storage engine (InnoDB) fall in to the hands of Oracle. MongoDB was smart enough to make sure that they were the acquirer, which puts them in control of their own destiny.
Not sure if that's relevant though, since whole MySQL became property of Oracle (after they acquired Sun).
Yes... its very relevant... If MySQL had acquired InnoDB instead of Oracle there is a good chance they would still exist as an independent entity. That's what I meant by MongoDB's acquisition of WT putting them in control of their own destiny.
> Can you give an example of another option you are referring to?
That depends on the data.
What type of data you have and what you want to do with it.
MongoDB isn't data specific and it claim to fame is flexible data structure.
If you want fast write and look up with very little relation Cassandra is good.
If you want searchable text document then anything that is base on Lucene is good (ES, Solr, Raven).
If you want time series there are few out there but it's a niche.
Likewise if you want graph data then there are NodeJS, Titan, etc..
MongoDB at most company I worked with was use because they don't think about what type of data it is and what performance they want. They want to store unstructure data cause it's easy.
I personally think it's a cop out, especially as a statistician/programmer.
What if I want filtering by several criteria (on a table with 1k columns) and simple aggregations, but I don't need full-text search* ? I'm still looking :(
* I only want starts-with and contains on strings.
SQL Server and Oracle are hardly comparable. The issues you see there are more to do with backwards compatibility or performance impact of big new features than they are with the core stability.
> As far as I can ascertain, RethinkDB’s safety claims are accurate. You can lose updates if you write with anything less than majority, and see assorted read anomalies with single or outdated reads, but majority/majority appears linearizable.
The safety claims are accurate but like all databases, its never 100% perfect as there are tradeoffs.
> That reordering was only possible because of a special workaround for a bug in an older version of RethinkDB.
> What are the risks to users? The RethinkDB team and I suspect it’s unlikely this bug will occur outside of stress testing. Cluster reconfiguration is typically infrequent, and users would need a specific series of network failures or other message delays which happen to cut the replicas apart—in a way which allows both network components to find independent majorities for their respective table configurations. In Jepsen tests, it usually takes tens to hundreds of partition/reconfigure rounds to trigger this bug.
This isn't really an architectural failure (as you see repeatedly with MongoDB) but an implementation bug.
Compared to the failures in MongoDB that are frequently architectural (rather than implementation bugs), RethinkDB passed easily and quickly fixed the implementation bugs.
Rethink and Cassandra both fare far better in Jepsen.
Depending on the exact requirements Postgres (in the right configuration) and Redis are better options.
I'd argue that for 90% of MongoDB users - the group whose requirements were met by MySQL/MyISAM - would be better served by Postgres or MySQL with a proper storage engine.
Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully. There were also a few implementation issues around LWT (Paxos) but at that time Paxos was very new. There were no architectural / fundamental changes needed to correct these issues. Also, even before fixing, Cassandra was losing only a tiny fraction of data than Mongo was losing. So, yes, this is what I call "far better", even if not initially perfect.
Pretty sure Cassandra still uses local timestamps and last-write-wins, which means you can't do safe updates. Also pretty sure they didn't fix the timestamp-collision row-isolation issue I talked about, because it was on HN's front page a few months ago. ;-)
Using timestamps and last write wins is a well known and well documented behavior, so you must assume updates can be sometimes applied in a different order than they were submitted. However, reordering updates is not the same as rolling back writes as was in case of Mongo. Cassandra does not promise linearizability in this mode of operation (if not using LWTs), and this is a tradeoff to get better availability. There is no better way than last write wins if you want high availability and partition tolerance and don't want to pay the performance and availability price for a consensus algorithm like Paxos or Raft. And no, vector clocks do not solve this problem in practice at all.
As for the post on HN from a few months ago, I remember only one guy who mixed LWT and non LWT updates and was surprised with lack of linearizability. Not a Cassandra fault if somebody doesn't know what he's doing.
> assume updates can be sometimes applied in a different order than they were submitted
I suggest you re-read the analysis. Some databases can offer safe, generalized commutative updates; e.g. Riak. Cassandra can't: updates, in general, can be lost through reordering.
> There is no better way than last write wins if you want high availability and partition tolerance and don't want to pay the performance and availability price for a consensus algorithm like Paxos or Raft.
You can have safe updates through clustering columns or if you really insist on destructive updates - through LWTs. With clustering columns you can easily achieve whatever is possible with vector clocks.
As for the research you posted, there is no free lunch. Each of these strategies come with their own set of drawbacks. That's why Cassandra offers choice at a query level.
Clustering columns do not make Cassandra updates any safer: they only reduce the scope of conflicts. It's still last-write-wins--the approach that Cassandra's own blog recognizes as "a high potential for data loss".
That's why Cassandra offers choice at a query level.
It doesn't give you a choice: you get last-write-wins, or some limited merge functions with CQL types. You can't get generalized CRDTs in Cassandra, because they won't expose conflicts to the user layer. Cassandra gives up on a whole class of safe AP datatypes as a consequence of this restriction.
Now you are arguing about a lack of a particular builtin feature. Riak is last-write-wins by default with optional vector clocks and builtin CRDTs. Cassandra is last-write-wins by default with clustering columns allowing to implement an equivalent of Riak vector clocks and CRDTs in the application. If you include a client id in the primary key and use client-side timestamps, you essentially are doing vector clocks and there are no conflicts guaranteed - users of Cassandra have been doing it for years.
We built examiner.com on MongoDB in 2010. Yes, that was before the journal was introduced. Would you call a then Quantcast Top 100 site a "toy project"? It served well into 2016.
I said that I would only use it for a toy project. Or at least a project for whose requirements resulted in MongoDB being a good choice. For example a content site where losing some data doesn't matter and whose hot datasets would never expand above the memory available by my budget.
Given the projects I chose to work on, MongoDB would be a bad choice. So I would only use it for a toy project right now. In 3-5 years time I expect that it will be a very good choice, given that they've pretty much won and can hire all of the good engineering talent.
The parallels with MySQL are really astounding; I just hope that they don't end up being sold to Oracle or Microsoft.
I think you're going to end up on the wrong side of this one, if you're not willing to change your opinion based on new info. It's a big deal, his second to last sentence in the intro:
> While the v0 protocol remains broken, patches for v1 are available in MongoDB 3.2.12 and 3.4.0, and now pass the expanded Jepsen test suite.
I'll change my opinion once they've actually fixed it. The same way my opinion of MySQL changed when it finally matured (around 5.0 when they added proper transactions).
I at least will never trust Mongo for anything but a toy project. There are so many better options out there, options whose technical capabilities are as good as Mongo's marketing.