I've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services.
I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries.
Fanning out one big service in parallel with a matching scalable DB is by far the most sane way to build things.
Right, but the thing that makes Linux actually useful isn't really the kernel is it? I would say what makes it useful is all the various small, targeted programs (some might call them microservices) it lets you interact with to solve real world problems.
If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise? Sounds like a nightmare to me.
One example I'm familiar with that sounds like microservices is the Robot Operating System (ROS). At its heart it's just a framework for pub/sub over IP, it just happens to be targeted towards robotics. A ROS system comprises of 'nodes' for each logical operation e.g. image acquisition -> camera calibration -> analysis -> output.
The system is defined by a graph of these nodes, since you can pipe messages wherever they're needed; all nodes can be many-many. Each node is a self-contained application which communicates to a master node via tcp/ip (like most pub/sub systems a master node is requried to tell nodes where to send messages). So you can do cool stuff like have lots of seprate networked computers all talking to each other (fairly) easily.
It works pretty well and once you've got a particular node stable - e.g. the node that acquires images - you don't need to touch it. If you need to refactor or bugfix, you only edit that code. If you need to test new things, you can just drop them into an existing system because there's separation between the code (e.g. you just tell the system what your new node will publish/subscribe and it'll do the rest).
There is definitely a feeling of duct tape and glue, since you're often using nodes made by lots of different people, some of which are maintainted, others aren't, different naming conventions, etc. However, I think that's just because ROS is designed to be as generic as possible, rather than a side effect of it running like a microservice.
IMO, the main benefit of ROS is the message definitions. Having exactly one definition of "point cloud" that everybody accepts means everyone's code will be compatible. That isn't normally the case in library ecosystems. If ROS was replaced by <robotics_types.hpp> I think we'd get 90% of the benefit.
There's actually no reason you can't architect microservices like this. You can put RabbitMQ or some AMQP service as a comm layer between services. But then you have to architect your system to be event-driven. It's not a bad approach.
I've worked on a system that used Kafka in-between. It makes consistency issues a ton worse because everything is async. At least with HTTP you can do synchronous calls
This is just not true. Google uses almost entirely immediately consistent databases and gRPC internally. Eventual consistency is hardly ever required for REST type calls even at massive scale.
gRPC has no queuing and the connection is held open until the call returns. All of Google's cloud databases are immediately consistent for most operations
My understanding is that Spanner is optimistically immediately consistent, with very clever logic for deciding on whether to bounce the attempted transaction (TrueTime).
But strictly speaking, even inside a single multicore CPU, there is no such thing as immediate consistency. The universe doesn't allow you to update information in two places simultaneously. You can only propagate at the speed of light.
Oh, and the concept of "simultaneous" is suspect too.
Our hardware cousins have striven mightily and mostly successfully for decades to create the illusion that code runs in a Newtonian universe. But it is very much a relativistic one.
ROS is not a particularly good implementation of this concept though. And it tends to encourage extremely fine splitting of jobs, which basically just turns your robotics problem into a distributed robotics problem, making it much harder in the process (as well as less efficient).
> If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise?
> Same with BSD; the userspace programs are not part of the kernel, and developed separately by the same entity.
they're part of the same repo and built at the same time than the kernel. Run a big "make universe" here : https://github.com/freebsd/freebsd and see for yourself. That they are different binaries does not matter a lot, it's just a question of user interface. See for instance busybox where all the userspace is in a single binary.
I would argue that the fact that they're in the same repo does not make it a monolith at all. All of google and facebook is in a single repo and built at the same time, but it doesn't make them monoliths.
My understanding is that OpenBSD is constructed this way, and I have also heard that their code is well-organized and easy to follow.
There is the "base" system which is the OS itself and common packages (all the ones you mentioned), then there is the "ports" repo which contains many open-source applications with patches to make them work with OpenBSD.
I think OpenBSD has reaped many of the same advantages described by Segment with their monorepo approach, such as easily being able to add pledge[1] to many ports relatively quickly.
The whole thing is released and deployed together, which is the usual distinction between microservices and not. I don't think anyone's advocating not using modules and folders to structure your code in your repository (certainly I hope not), but a single deployment artifact makes life a lot easier.
But there is a big difference. These small targeted programs are invoked in user land, usually by the user. Microservices get invoked directly by the user when debugging is going on. Otherwise they are expected to automagically talk to each other and depending on the abstraction even discovery each other automatically.
Also I can pipe these tools together from the same terminal session, like
tail -f foo | grep something | awk ...
You don't have that in general with Microservices. Unix tools are Lego, Microservices aren't. They are Domino at best.
Probably one could come up with an abstraction to do Lego with Microservices but we're not there yet.
And they can be a right mess to wrangle because upstream gets into some artistic frenzy, and shouts down anyone that worries about the ensuing breakages as luddites and haters.
And still nobody would argue to pack everything together into one executable because of these issues. As everything in software engineering, it is about finding the right trade-off.
The big problems with microservices come from distributed transactions being difficult and clocks getting out of sync. Multiple services on a single machine don't have that problem.
With web apps the main concern is data consistency between relations. On the OS level you have these same concerns with memory and disk, and there's database-like systems in the kernel and drivers to handle it. Essentially all these utilities are running within the same "database" which is disk and memory management handled by the kernel. Usually microservices have their own databases, which is where consistency hell begins
Unless you mean "their own database tables", not "database servers". But that's just the same as having multiple directories and files in a Unix filesystem.
To be blunt, its news to a lot of people, but it also isn't wrong. Microservices really shouldn't share a database, and if they do then they aren't "microservices".
(a) a prototype
(b) a set of applications that share a database.
You would have to have an oddly disconnected schema if modifications to the program don't result in programs accessing parts of the database that other programs are already accessing. If this isn't a problem it means you're using your database as nature intended and letting it provide a language-neutral, shared repository with transactional and consistency guarantees.
so maybe not microservices, but fine nonetheless.
EDIT: two more comments:
- this is exactly what relational databases were designed for. If people can't do this with their micro-services, maybe their choice of database is the issue.
- "micro-service" as the original post suggests, is not synonymous with good. "monolith" is only synonymous with bad because it got run-over by the hype-train. If you have something that works well, be happy. Most people don't.
I think you are exactly right. Microservice architectures are definitely not automatically good, and there is nothing wrong with a well architected "monolith".
If you're going to go microservices, you want service A to use service B's public API (MQ or RPC or whatever), not to quietly depend on the schema B happened to choose. And sharing a database server instance turns overloads into cascading failures, unless the stack is very good at enforcing resource limits on noisy neighbors.
That architecture is commonly known as a distributed monolith. If you put two services on the same DB you can guarantee that someone will be joining onto a table they shouldn't have before the week is out.
If you can afford all your components sharing the same database without creating a big dependency hell, then your problem is _too small_ for microservices.
If your problem is so large that you have to split it up to manage its complexity, start considering microservices (it might still not be the right option for you).
Yes, they do fine as examples. I have worked both on decades old booking systems and banking software. Both were dependency hells. Refactoring was impossible. Everybody was super-careful even with tiny changes because the risk to break something was just too high.
If it could be avoided, these systems were not touched anymore. Instead, other applications where attached to the front and sides.
I don't think so. For example, the Linux kernel is old, but still quite well maintainable.
So I would say: it applies to systems where proper modularization was neglected. In the anecdotical cases I referred to, one major element of this deficiency was a complex database, shared across the whole system.
Nah, just use a database that can scale with your app and run a ton of instances. You'll be hard pressed to find any size app that doesn't fit in RAM these days.
While that's true. It also means you are only one grant away from sharing database tables. Maybe with good discipline you will be ok but all it take is for one dev to take that one shortcut.
Agreed. I've only seen proper user separation happen when legal gets involved. Usually it's an all-access free-for-all, and after all, why shouldn't it be? It's a lot easier to pull data and logs from adjacent services yourself rather than taking a ticket and hoping the team still exists for a response.
If you have different database servers then you are only a "connection string" away from breaking your separation. Using a different login and schema is perfectly fine in a lot of architectures.
> You push data consistency concerns out of the database and between service boundaries.
Sing that from the rooftops. That is exactly my observation as well. All the vanilla "track some resource"-style webapps I've worked on were never designed to cope with a consistency boundary that spans across service boundaries. Turning a monolith into distributed services is hard for that reason - you have to redesign your data access to ensure that consistency boundaries don't span across multiple services. If you don't do that, then you have to learn to cope with eventual consistency; in my experience, most people just don't think that way. I know I have trouble with it. Surely I'm not the only one.
It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path.
I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. I think it's one of those things that gives the illusion of productivity - "I did all this work, and now I have left-pad-as-a-service running! Look at the little green status light on the cool dashboard we spent the last couple months building!"
Programmers get excited about little happily running services, these are "real" to them. Customers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" that comb the database for various inconsistencies that inevitably show up.
You can't have consistent microservices without distributed transactions. If a service gets called, and inside that call, it calls 3 others, you need to have a roll back mechanism that handles any of them failing in any order.
If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent.
Worse, this "undo state" needs to be kept transactionally consistent in case it's your service that dies after the first call.
In reality, nobody does this, so they're always one service crash away from the whole system corrupting the hell out of itself. Since the state is distributed, good luck making everything right again.
Microservices are insane. Nobody that knows database concepts well should go near them
Data consistency is one of those things that sounds like it matters but often doesn't. There's not much in Netflix's platform that screams data consistency is an ultra high priority. Any application that deploys active/active multi-region is by definition going to encounter scenarios where data loss is possible. There's just no way around CAP.
I'd venture a guess that most applications have all sorts of race conditions that could cause data corruption. The fact of the matter is that almost nobody notices or even cares.
A little birdie who worked at Grab once told me that they pretty much don't use transactions anywhere. So... maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them. The next time you're standing in a monsoon waiting for the "coming" car that never comes, remember this post :-)
Aside from that, I've found that even in non-mission-critical scenarios ("it's just porn!") it's incredibly convenient to have a limited number of states the system can be in. It makes debugging easier and reduces the number of edge cases ("why is this null??") you have to handle.
> maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them.
I think you'd be surprised/alarmed at how little transactions actually get used in the software world. Not just on small systems where it doesn't matter but I've seen a complete absence of them in big financial ones handling billions of dollars worth transactions (the real world kind) a day. Some senior, highly paid people even defend this practice for performance reasons because they don't realize the performance cost of implicit transactions. And this is just the in process stuff where transactions are totally feasible, it get's even worse when you look at how much is moved around via csv files to FTP and excel sheets attached to emails. I've spent the last 2 weeks being paid to fix data consistency issues that should never have been issues in the first place.
Maybe when we're teaching database theory we shouldn't start at select/join but at begin transaction/commit/rollback?
Having spent a good amount of time fixing bugs related to code that fetches data with a consistency level of READ_UNCOMMITTED for "performance" reasons I can appreciate what transactions give you as an application developer.
However, I would argue that transactions are overkill. What's the worst case scenario if I book a ride for Grab and my request gets corrupted? I'm guessing I'll see an error message and I'll have to re-request my ride.
Re: I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" [to fix "bad" transactions]
If your business model is to be cheap with high volume sales, then corrupting say 1 out of 10,000 customer transactions may be worth it. If you give customers a good price and/or they have no viable alternative, you can live with such hiccups, and the shortcuts/sacrifices may even make the total system cheaper. You are like a veterinarian instead of a doctor: you can take shortcuts and bork up an occasional spleen without getting your pants sued off. But most domains are NOT like that.
>It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path.
If two microservices have to share databases, they shouldn't be microservices.
One microservice should have write access to one database and preferably, all read requests run through that microservice for exactly the reason you mentioned.
>I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better.
If you're running microservices between regions and communicating with each other outside of the network it is living in, you're probably doing it wrong.
Microservices shouldn't have to incur the cost of going from SF to China and back. If one lives in SF, all should and you can co-locate the entire ecosystem (+1 for "only huge companies with big requirements should do microservices")
>ustomers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
Again, that is an example of microservices gone wrong. You'll have the same amount of changes even in a monolith and I'd argue adding new features is safer in microservices (No worries of causing side effects, etc).
I will give you +1 on that anyway because I designed a "microservice" that ended up being 3 microservices because of dumb requirements. It probably could've been a monolith quite happily.
My problem with microservices is the word 'micro'. It should just be services.
Problem domains (along with organizational structures) inherently create natural architectural boundaries... certain bits of data, computation, transactional logic, and programming skill just naturally "clump" together. Microservices ignore this natural order. The main driving architectural principle seems to be "I'm having trouble with my event-driven dynamically-typed metaprogrammed ball-of-mud, so we need more services!".
> Problem domains (along with organizational structures) inherently create natural architectural boundaries
The "natural" order is very often bad for reliability, speed and efficiency. It forces the "critical path" of a requests to jump through a number of different services.
In well built SOA you often find that the problem space is segmented by criticality and failure domains, not by logical function.
It took me longer than it probably should have to realize that the term microservices was made by analogy to the term microkernel. I think that part of my main issue with the "microservices" is that it conflates highly technical semantics with word forms that are a bit more wishy washy in meaning (i.e., a service is something formed more of human perception, not rooted directly in operating system abstractions).
I wonder if microservices will have a similar evolution as microkernels. While some microkernels almost closed the performance gap to monolith kernels, the communication overhead killed them. It turned out that the kernel's complexity could be reduced by moving some functions to the applications ("library OS"). Linux kernel drivers are only accepted when the functionality can't be done in user space. The extreme version is running single purpose, highly specialized unikernels on a hypervisor.
Yes! But it’s worse again. Because some bright spark comes along and says “hey there’s some commonality between these n services. I’ll make a shared library that will remove that duplicated code.
Then before you know it there are a dozen more shared libraries and you have the distributed monolith.
Then either have to Stand up every micro service every time integration test or make changes and hope for the best.
When I was designing a microservices architecture for a former employee, the justification for it was security. We needed separation of the components in the system because we were going to be handling money. It used ZeroMQ for communication, because I wanted something more lightweight and falut-resistant for message passing (in such a sensitive application, we didn't want to trust the network). Although it was of course fun to design, microservices weren't my first choice. It just made sense to us to use it in that particular scenario.
Sometimes, I've seen a lack of regard for data consistency within a monolith.
That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions. On the other hand, I've seen some pretty flagrant disregard for it just because there are not atomicity guarantees.
> That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions
I'd argue that's on the developer, if he was the one to choose a database that doesn't support transactions, and then didn't implement application-level transactions (which is very hard to do correctly).
Txns are just hard in general but mature RDBMS have spent a huge number of man-years on getting it right. After the time I've spent on Couchbase, Mongo, Elasticsearch, I doubt I'll ever use something non-transactional for anything OLTP or OLTP-adjacent. If Postgres or MySQL can't scale up to handle it, make them...or get a new job. Scale is a PITA.
I've seen it on monoliths too. We run a pretty large app just fine with the "nuclear option". Database transaction isolation level serial. Makes it impossible to have inconsistencies on the database level, and on some RDBMS like PostGres the performance impact is small
I remember using BEA Tuxedo 15 years ago and I can't help wondering if theoretically one could do microservices using it or some OLTP to achieve consistency when needed. Has it progressed over the years? Is there any free alternative to it? Or is it dead tech? When I used it it was hard and too much extra work, but it did the job successfully.
"Need to" and "sane" are among my favourite subjective terms!
(Further below, I'll go into in which contexts I'd agree with your assessment and why. But for now the other side of the coin.)
In the real world, current-day, why do many enterprises and IT departments and SME shops go for µservice designs, even though they're not multimillion-user-scale? Not for Google/Netflix/Facebook scale, not (primarily/openly) for hipness, but they do like among other reasons:
- that µs auto-forces certain level of discipline in areas that would be harder-to-enforce/easier-to-preempt by devs in other approaches --- modularity is auto-enforced, separation of concerns, separation of interfaces and implementations, or what some call (applicably-or-not) "unix philosophy"
- they can evolve the building blocks of systems less disruptively (keep interfaces, change underlyings), swap out parts, rewrites, plug in new features to the system etc
- allows for bring-your-own-language/tech-stack (thx to containers + wire-interop) which for one brings insights over time as to which techs win for which areas, but also attracts & helps retain talent, and again allows evolving the system with ongoing developments rather than letting the monolith degrade into legacy because things out there change faster than it could be rewritten
I'd prefer your approach for intimately small teams though. Should be much more productive. If you sit 3-5 equally talented, same-tech/stack and superbly-proficient-in-it devs in a garage/basement/lab for a few months, they'll probably achieve much more & more productively if they forgoe all the modern µservices / dev-ops byzantine-rabbithole-labyrinths and churn out their packages / modules together in an intimate tight fast-paced co-located self-reinforcing collab flow. No contest!
Just doesn't exist often in the wild, where either remote distributed web-dev teams or dispersed enterprise IT departments needing to "integrate", rule the roost.
(Update/edit: I'm mostly describing current beliefs and hopes "out there", not that they'll magically hold true even for the most inept of teams at-the-end-of-the-day! We all know: people easily can, and many will, 'screw up somewhat' or even fail in any architecture, any language, any methodology..)
In my experience if your developers were going to make choices that lead to tight coupling in a monolith, they’re going to make the same choices in a distributed architecture. Only now you’ve injected a bunch of network faults and latency that wouldn’t have been there otherwise.
In this case it sounds like they started with a microservice architecture, but CI/CD automation necessary for robust testing and auto-scaling was not in place. The problem of queues getting backed up might have been addressed by adding a circuit breaker, but instead they chose to introduce shared libraries (again, without necessary testing and deployment), which resulted in very tight coupling of the so-called microservices.
Do they actually force a discipline? Do people actually find swapping languages easier with RPC/messaging than other ffi tooling? And do they really attract talent?!
You make some amazing claims that I have seen no evidence of, and would love to see it.
In my experience, there's a lot of cargo culting around microservices. The benefits are conferred by having a strong team that pays attention to architecture and good engineering practices.
Regardless of whether you are a monolith or a large zoo of services, it works when the team is rigorous about separation of concerns and carefully testing both the happy path and the failure modes.
Where I've seen monoliths fail, it was developers not being rigorous/conscientious/intentional enough at the module boundaries. With microservices... same thing.
Also, having a solid architectural guideline that is followed across the company in several places (both in infrastructure and application landscapes) makes up the major bulk of insuring stability and usability.
The disadvantage is obviously that creating such's a 'perfect architecture' is hard to do because of different concerns by different parties within the company/organisation.
> The disadvantage is obviously that creating such's a 'perfect architecture' is hard to do because of different concerns by different parties within the company/organisation.
I think you get at two very good points. One is that realistically you will never have enough time to actually get it really right. The other is that once you take real-world tradeoffs into account, you'll have to make compromises that make things messier.
But I'd respond that most organizations I see leave a lot of room for improvement on the table before time/tradeoff limitations really become the limiting factor. I've seen architects unable to resolve arguments, engineers getting distracted by sexy technologies/methodologies (microservices), bad requirements gathering, business team originated feature thrashing, technical decisions with obvious anticipated problems...
> "You make some amazing claims that I have seen no evidence of"
I'm just relaying what I hear from real teams out there, not intending to sell the architecture. So these are the beliefs I find on the ground, how honest and how based-in-reality they are are harder to tell and only slowly over time at any one individual team.
A lot of this is indeed about hiring though, I feel, at least as regards the enterprise spheres. Whether you can as a hire really in-effect "bring your own language" or not remains to be seen, but by deciding on µs architecture for in-house you can certainly more credibly make that pitch to applicants, don't you think?
Remember, there are many teams that have suffered for years-to-decades from the shortcomings and pitfalls of (their effectively own interpretation of / approach to) "monoliths" and so they're naturally eagerly "all ears". Maybe they "did it wrong" with monoliths (or waterfall), and maybe they'll again "do it wrong" (as far as outsiders/gurus/pundits/coachsultants assess) with µs (or agile) today or tomorrow. The latter possibility/danger doesn't change the former certainties/realities =)
Pure anecdote so I know it is meaningless but I have rewritten/refactored old services with new code, or even new languages twice with little problem because the interface was well defined. We had hundreds of devs working in other areas and we were all on different release cycles because changes were easy and decoupled. We let any team submit bug reports when we either weren't complying with our interface or we had a bug somewhere.
The only teams I had to spend time on were the ones which were on a common DB before we moved off of it.
I think this is probably true for larger or more distributed corporate environments, but I think a modular monolith is going to be a more productive and flexible architecture for most teams, and should be the default option for most startups (many of whom are doing microservices from day 1 it seems).
1. Is auto-enforced modularity, separation of concerns, etc actually better than enforcing these things through development practices like code review? Why are you paying people a 6 figure salary if they can't write modular software?
2. Is the flexibility you gain from this loose coupling worth the additional costs and overhead you incur? And is it really more flexible than a modular system in the first place? And how does their flexibility differ? With an API boundary breaking changes are often not an option. In a modular codebase they can easily be made in a single commit as requirements change.
3. Is bring-your-own-language actually a good idea for most businesses? Is there a net benefit for most people beyond attracting and retaining talent? What about the ability to move developers across teams and between different business functions? Having many different tech stacks is going to increase the cost of doing this.
I do see the appeal of some of these things, but IMO the pros outweigh the cons for a smaller number of businesses than you've mentioned. And the above is only a small sample of that. Most things are just more difficult with a distributed system. It's going to depend on the problem space of course, but most backend web software could easily be written in a single language in a single codebase, and beyond that modularization via libraries can solve a lot of the same problems as microservices. I'm very skeptical of the idea that microservices are somehow going to improve reliability or development speed unless you have a large team.
These are great observations. For anyone interested in going more in depth on the topic, I highly recommend the book Building Evolutionary Architectures
Except the language argument, don't you get all that by just having modules in your code (assuming a statically typed language since the boundaries are type checked)?
Not really. For example, it's easier to mock a microservice, than a module, for testing purposes. Let's say you have component A and component B, A depends on B (dependency implemented via runtime sync or async call), B is computationally intensive or has certain requirements on resources that make it harder or impossible to test on developer's machine. You may want to test only A: with monolithic architecture you'll have to produce another build of the application, that contains mock of B (or you need something like OSGi for runtime module discovery). When both components are implemented as microservices, you can start a container with mock of B instead of real B.
Running E2E blackbox test is equally simple for all kinds of architectures, especially today, when it's so easy to create a clean test environment with multiple containers even on developer's machine. It may be harder to automate this process for a distributed system, but, frankly speaking, I don't see a big difference between a docker-compose file or a launch script for monolith - I've been writing such tests for distributed systems casually for several years and from my personal experience it's much easier to process the test output and debug the microservices than monolithic applications.
> it's much easier to process the test output and debug the microservices than monolithic applications.
You easier to debug end-to-end tests of a microservice architecture that monolith? That's not my experience. How do you manage to put side by side all the events when they are in dozen of files?
Using only files for logging is the last thing I would do in 2018.
I use Serilog for structured logging. Depending on the log destination, your logs are either stored in an RDMS (I wouldn’t recommend it) or created as JSON with name value pairs that can be sent directly to a JSON data store like ElasticSearch or Mongo where you can do adhoc queries.
I just don't use files for anything (services should be designed with the assumption that container can be destroyed any time, so files are simply not an option here). If you are talking about the logs, there are solutions like Graylog to aggregate and analyze them.
Easy until you have 100,000 of them anyway, in which case it's expensive and slow to run it for every dev. (At that point you have enough devs that microservices 100% make sense, though)
A dependency injection framework where you use flags at the composition root to determine whether the “real” implementation class or the mock class is used based on the environment.
You will end up with something like OSGi. That can be the right choice, but is also a quite 'heavyweight' architecture.
For a certain class of applications and organizational constraints, I also would prefer it. But it requires a much tighter alignment of implementation than microservices (e.g., you can't just release a new version of a component, you always have to release the whole application).
For a certain class of applications and organizational constraints, I also would prefer it. But it requires a much tighter alignment of implementation than microservices (e.g., you can't just release a new version of a component, you always have to release the whole application).
Why is that an issue with modern CI/CD tools? It’s easier to just press a button and have your application go to all of your servers based on a deployment group.
With a monolith, with a statically typed language, refactoring becomes a whole lot easier. You can easily tell which classes are being used, do globally guaranteed safe renames, and when your refactoring breaks something, you know st compile time or with the correct tooling even before you compile.
> It’s easier to just press a button and have your application go to all of your servers based on a deployment group.
It's not so much about the deployment process itself (I agree with you that this can be easily automated), but rather about the deployment granularity. In a large system, your features (provided by either components or by independent microservices) usually have very different SLAs. For example, credit card transactions need to work 24x7, but generating the monthly account statement for these credit cards is not time-critical. Now suppose one of the changes in a less critical component requires a database migration which will take a minute. With separate microservices and databases, you could just pause that microservice. With one application and one database, all teams need to be aware of the highest SLA requirements when doing their respective deployments, and design for it. It is certainly doable, but requires a higher level of alignment between the development teams.
I agree with your remark about refactoring. In addition, when doing a refactoring in a microservice, you always need a migration strategy, because you can't switch all your microservices to the refactored version at once.
With separate microservices and databases, you could just pause that microservice. With one application and one database, all teams need to be aware of the highest SLA requirements when doing their respective deployments, and design for it. It is certainly doable, but requires a higher level of alignment between the development teams.
That’s easily accomplished with a Blue-Green deployment. As far as the database, you’re going to usually have a replication set up anyway. So your data is going to live in multiple databases anyway.
Once you are comfortable that your “blue” environment is good, you can slowly start moving traffic over. I know you can gradually move x% of traffic every y hours with AWS. I am assuming on prem load balancers can do something similar.
If your database is a cluster, then it is still conceptually one database with one schema. You can't migrate one node of your cluster to a new schema version and then move your traffic to it.
If you have real replicas, then still all writes need to go to the same instance (cf. my example of credit card transactions). So I also don't understand how your migration strategy would look like.
blue-green is great for stateless stuff, but I fail to see how to apply it to a datastore.
Do you realize that this is actually an anti-pattern, that adds unnecessary complexity and potential security problems to your app? Test code must be separated from production code - something, that should know every developer.
It’s basically a feature flag. I don’t like feature flags but it is a thing.
But if you are testing an artifact, why isn’t the artifact testing part of your CI process? What you want to do is no more or less an anti pattern than swapping out mock services to test a microservice.
I’m assuming the use of a service discovery tool to determine what gets run. Either way, you could screw it up by it being misconfigured.
First of all it is the test code, no matter whether it's implemented as a feature flag or in any other way. Test code and test data shall not be mixed with the production one for many well-documented and well-known reasons: security, additional points of failure, additional memory requirements, impact on architecture etc.
>But if you are testing an artifact, why isn’t the artifact testing part of your CI process?
It is and it shall be part of the CI process. Commit gets assigned build number in tag, artifact gets the version and build number in it's name and metadata, deployment to CI environment is performed, tests are executed against specific artifact, so every time you deploy to production you have a proof, that the exact binary that is being deployed has been verified in its production configuration.
>I’m assuming the use of a service discovery tool to determine what gets run.
Service discovery is irrelevant to this problem. Substitution of mock can be done with or without it.
If you are testing a single microservice and don’t want to test the dependent microservice - if you are trying to do a unit test and not an integration test, you are going to run against mock services.
If you are testing a monolith you are going to create separate test assemblies/modules that call your subject under test with mock dependencies.
They are both going to be part of your CI process then and either way you aren’t going to publish the artifacts until the tests pass.
Your deployment pipeline either way would be some type of deployment pipeline with some combination of manual and automated approvals with the same artifacts.
The whole discussion about which is easier is moot.
Edit: I just realized why this conversation is going sideways. Your initial assumptions were incorrect.
you may want to test only A: with monolithic architecture you'll have to produce another build of the application, that contains mock of B (or you need something like OSGi for runtime module discovery).
> What exactly are you trying to accomplish?
Good test must verify the contract on the system boundaries: in case of the API, it's verification done by calling the API. We are discussing two options here: integrated application, hosting multiple APIs, and microservice architecture. Verification on the system boundaries means running the app, not running a unit test (unit tests are good, but serve different purpose). Feature flags make it only worse, because testing with them covers only non-production branches of your code.
> Your initial assumptions were incorrect.
With nearly 20 years of engineering and management experience, I know very well how modern testing is done. :)
Verification on the system boundaries means running the app, not running a unit test
What is an app at the system boundaries if not a piece of code with dependencies?
If you have a microservice - FooService that calls BarService. The "system boundary" you are trying to test is FooService using a fake BarService. I'm assuming that you're calling FooService via HTTP using a test runner like Newman and test results.
In a monolithic application you have class FooModule that depends on BarModule that implements IBarModule. In your production application you use create FooModule:
var x = FooModule(new BarModule)
y = x.Baz(5);
In your Unit tests, you create your FooModuleL
var x = FooModule(new FakeBarModule)
actual= x.Baz(5)
Assert.AreEqual(10,actual)
And run your tests with a runner like NUnit.
There is no functional difference.
Of course FooModule can be at whatever level of the stack you are trying to test - even the Controller.
I was doing this with COM twenty years ago. It had the same advantages of modularity and language independence but without the unnecessary headaches of a distributed system.
I take your point, but it saddens me that there aren't better ways of achieving this modularity nowadays.
I was too, of course a distributed system could also be built with DCOM and MTC (later COM+), and the DTC (Distributed Transaction Coordinator) could be used when you needed a transaction across services (or DBs, or MQs). Obviously the DTC was developed in recognition of the fact that distributed transactional service calls were a real requirement - something that current microservice architectures over HTTP REST don't seem to support.
Are you telling me you chose micro services just to enforce coding standards and allow devs to be more comfortable?
The legacy concerns I don’t see being true, as it’s mainly a requirements/documentation problem and you can achieve the same effect with feature toggles.
The key to a large code base is a hierarchical design. People are looking for a technical solution to a design problem. No matter what technology you use if you don't maintain the hierarchical layers your code will turn to mush. Micro services can provide a really strict boundary but if you draw the line in the sand at the wrong place it isn't going to matter.
Teams have a lot to do with microservice use. If you have a small team a monolith can work well. If you have a large team, microservices have some significant advantages. And in some cases, you might have a microservice in Swift when you need high performance, but other less intensive services might be in Ruby or some other language, etc.
Right took for the job should be the goal as opposed to chasing fashion. Microservices are definitely overused, but they do have many legitimate use cases. Your CRUD web application probably doesn’t need microservices, but complex build systems might.
Imagine the complexity of Amazon.com or Netflix were a monolith. But something like Basecamp is probably better as a monolith.
There is also the issue of scale. An ML processor might need more (and different) hardware than a user with system.
It depends but in general I'd disagree. If the small balls of mud have all kinds of implicit dependencies, but you have to find them by searching across codebases (and languages) -- does that sound easier than finding them all in the same codebase and language? Overall, its comparing bad design to bad design. I think the main argument I'd make here is, micro-services doesn't actually solve the big ball of mud problem, it solves a completely different problem.
You can step-through your big ball of mud in a debugger, but you can't do the same with your small balls of mud. Not easily at least. That alone makes a huge difference.
They're both bad. It feels like a reverse Sophie's Choice to have to pick one.
The real friction in the system is always in the boundaries between systems. With microservices it's all boundaries. Instead of a Ball of Mud you have Trees and No Forest. Refactoring is a bloody nightmare. Perf Analysis is a game of finger pointing that you can't defuse.
> 100 web applications in the world that need to be split into multiple services.
Services are not micro services. Most large scale applications can and should be split into multiple services. However, when approaching a new problem you should work within the monolith resisting the service until you absolutely can't any longer. Ideally this will make your services true services, that could capture an entire business unit. When it's all said and done you should be able to sell off the service as a business.
The other use case, which should be obvious, is compliance. If you are thinking about implementing anything that would require PCI or SOX you should do that in a service to shield the rest of the dev org from the complexities. So, any webapp that takes payment and interacts directly a payment processor.
That said, you're correct in that you should not be rolling out a new service to avoid sharding.
Microservices aren't some magic bullet for scaling. If anything, they conform to Conway's Law [1]. I'd agree, though, if a single engineer is singularly responsible for 2+ microservices that are only supporting a single product...you're doing it wrong.
I think that's possibly wrong as a key advantage of micro service over monolith is scaling. We have a 'single' product built as 2 micro services which is basically an external facing orchestration service that calls the compute heavy backend and it allows the external facing service to serve all the incoming traffic out of (on average) 4 containers whilst the backend computes with 32 containers and scales independently - very small changes to the incoming traffic can have large effects on the volume of traffic going to the compute service.
> I've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services.
This is apples and oranges. The production profile of operating system kernels and web applications are so dissimilar that the analogy is not useful. It may be true that most web applications don't need to be split into multiple services, but the Linux kernel provides no evidence either way.
Yes, but "monolithic" web applications can be built in the same way. It might not be a 100% accurate usage of the term, but microservice/SOA advocates love to call modular applications monliths anyways, to the point that it's something that most people seem to do.
The kernel can't be categorized fully as monolith: Many kernel subsystems are developed individually, by different teams and merged into the main branch by subsystem.
This is really missing the point. Microservices are not about code organisation - they are about runtime separation.
And even kernels have kernel threads, which are basically local microservices. Anything which needs to scale beyond a single system is more deserving of microservices than a kernel.
"I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries."
This exactly. Me too. Data consistency concerns in sufficiently large real world projects can be practically dealt with only 2 ways IMO: transactions or spec changes.
I generally agree that for the vast majority of web applications KiiS holds true and use monolith.
In terms of the common code divergence why not just use private NPM and enforce latest? Have a hard-fast rule that all services must always use the latest version of common.
I've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services.
Similarly it’s made with make. If anyone has a project more complex than the Linux kernel or GCC I’ll gladly listen to why they need some exotic build system... never met anyone yet...
I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries.
Fanning out one big service in parallel with a matching scalable DB is by far the most sane way to build things.