Skip to main content

My2Cents - Eight things leads to developing catastrophic cloud native microservices system



More of my two cents, just my thoughts. A quick fun read, not too deep, but worth noting :).

1. Setting the domain boundary wrong


This is a job guarantee tactics, it's endless looping in development and testing for everyone involved in the project without making the service to production! First everything starts simple and gradually find more and more functions, business logic gets added into the microservice, at the end, one even have to rename the whole damn thing. 

Symptoms and side effects 

  • A growing microservices becomes too fat, or every single microservices in the domain calls your microservice. (Sometime core microservice has the same behavior, but you should not see many of this type of services in a single domain.) This violates the microservices principle of easy, maintainable and agile.
  • Duplication of microservices/code everywhere. Where you find bits and pieces of duplicate code or microservices being copied and deployed into other domains. 


If you get into the endless implementation and testing hell, take a step back and look at how you separate the domains. Did you acc
identally place a context from other domains or mixing different concepts int
o one? Maybe it's worthwhile to go back to the design phase and think about the boundaries. To avoid duplication everywhere, make sure there are proper documented catalogs such as API portals using the Open API Standard Doc available between domains.


2. Mixing responsibilities of microservices 

In the mood for some Spaghetti, this is the way to go, and want some meatball to go on the Spaghetti to make it tastier? Mixing some stateful processes in there will get you plenty! 

Mixing

  •   Composing microservices
  •   Functional core business
  •   Stateless integration
  •   Stateful process


Symptoms and side effects 

  • Client calls becomes complex, if you don't provide a composed view of particular service, the client may require to compose and possibly handle some sort of core business login in it. 
  • Spaghetti relations between services, and it's really easy to lose track of how the flow of data and business logic spread around microservices.  
  • Too much coupling, having stateless and stateful is depended on the storage solution and scaling down could also have potential problems.


I've done a bunch of blogs on this topic, the main concept is always the separation of concerns, making sure there your cloud native microservices are clean, lean and agile meaning you need to make sure the business composition and the actual business functions needs to be separated into different instance and deployed individually for better structured and more modularized. Making sure the composition/orchestrations of microservices are stateless for scalability and flexibility. Set aside other long running process on other instance with dedicated proper storage solution to keep the states.

3. Too much coupling with external factors/demands

Being patient is always good, if you want to practice your ability to be extra patient while being calm, by coupling deeply with legacy system will train you to become the patient master, and become better at code management as you need juggle between agile demands and slow legacy update cycles.

Symptoms and side effects

  • Constant updates to core business to the core business system outside of better control. Having external system decide and make change request and short reaction time could cause issues. 
  • Once contract are establish between parties, will need extra care for version updates. 
The deployment cycle varies from system to system, but brown field application will have a longer application lifecycle compare to more agile and dynamic cloud native application. To move things faster, always consider to have a shield protection between your brown/green field application. So it accommodates to the s
lower moving parts as well as help green field cycle to continue on it's agile development. In your core domain microservices, avoid add anything that is dependent or specific for external consumers, as it might leads to too much customization that breaks the definite context boundary. Make sure you have other vendor/external system facing module that does the customization and hide the presentation complexity from internal domain models.

4. Duplicating and replicate non-biz related code in microservices

As a developer, it's nice to be depended on, you feel the satisfaction of being needed. There is a reason why there are 24 hours each day, so you get to work 20 hours per day and you are so important when ever there is a deployment, you needed to be there. 

Symptoms and side effects

  • None centralized control, all retry, reroute, versioning and deployment strategy are store in each individual instance.
  • Responsibility not clear between DEV/DEVOPS, again, the networking strategy is better for party that has better access to monitoring stats of how the environment is doing. 

Setting retry and routing policy can allows application to be more robust to failure. To have a more centralized view of how these policies are handled instead of spreading them all over individual microservices. Release the burden of developer handling everything in the application and better management and monitoring.


5. Service in mesh, connecting all with APIs

Making API calls are super easy, and service mesh is what everyone is doing today. Every technology has an library to handle REST/JSON. Let's connect all components in the system with APIs! 

Symptoms and side effects
  • Slow to react, since request and response are how most people uses it. And the wait could take longer and more locks for longer process. That becomes the performance bottleneck. 
  • Not utilizing cloud infrastructure. 

When microservice first came out, people wants to get away from ESB therefore many moved away from messaging broker and use API as the only connection method between services, as the Open API standard allows you to build a nice catalog among what is available. But event driven is there for a reason, to achieve true scalability, better decoupling, you need to stop making sticky dependency between calls. And event driven allows you to populate events to related parties, and because it's asynchronies, no resource are wasted when waiting for the reply.

6. Un-sorted events

Blasting the system with events, notify every corner to any events, will get your system super reactive! By the way, why not stream everything and store all the event! For better traceability!

Symptoms and side effects

  • Events everywhere, you realized you have to listen to insane amount of events to react up on it. 
  • Un-needed code to filter out events.
  • Confusion between status change or action should taken
  • Don't know what is better to preserve or lose the event messages

There are various types of events, if not carefully where and how the events are distributed, it's very likely to end up with unwanted events floating around consuming unnecessary resources. Focusing on some of the common ones like events containing data, status and commands. You want to minimize the number of data messages, as it take longer to process, more work to store. Striping down, filter the data could be more efficient. Handling retries of status events and rolling back command events should also be part of the event strategies.

7. Data Silos

One mircroservice one datasource, that is what experts says. 

Symptoms and side effects

  • Data not consistent among the datasources, or slow. 
  • Un-needed microservices created just for making sure the data are consistent.
  • Events everywhere to update status.

When you have isolated datasources there are risk of creating data silos, and because of the nature in microservice being distribute and more complex datastore scenario, it has become more difficult to mange the data consistency. You can start looking into how to capture the data change with many existing solutions such as streaming event changes as it happens during process or listen to changes from the main status store.


8. Limited automation

Manual is much more flexible, beside that was how it was done before with the JavaEE application on the servers. Automation, it takes too much time to setup, Lets deal with it when we have time?  

Symptoms and side effects

  • Slow updates
  • Long painful steps to productions

The first thing that needs to be in the planning is definitely automation. How to apply automated CI/CD pipelines and the deployment strategy, this how cloud native microservices system achieve the agility. I've done a couple of CI/CD automation with Jenkins in the past. See Github Link.

Alright, these are just my two cents, let me know if you think there are any other things that could lead to terrible cloud native microservices development? Love to know!


Comments

TurboTax Customer Service Number: +1-844-801-6775. For over 25 years, TurboTax products have been continuously ranked as the #1 best-selling tax software. TurboTax boasts that they are always up-to-date with the latest tax laws, including health care. TurboTax Customer Service Phone Number +1-844-801-6775.

Quicken Support Number | Sage Software Support
Smith Watson said…
Norton.com/setup – Learn how to get Norton security on your device. Also, find steps for installation and activation of Norton setup. Login to your Norton account at www.norton.com/setup.
Rajani said…
Thanks for sharing nice information and nice article and very useful information.....
Microservices Online Training
Microservices Training in Hyderabad
Edward said…
Thank you so much for this nice information. Hope so many people will get aware of this and useful as well. And please keep update like this.

Big Data Services

Data Lake Services

Advanced Analytics Services

Full Stack Development Services

Popular posts from this blog

My 2cents on the future of Integration - With Service Mesh/Istio and Serverless/KNative

It's been a year and half since I blogged about "Agile Integration architecture" (Gosh, time just flies). With the "microservices" and "cloud-native" hype, I was especially curious on how all these new concept and technology affect us on how to architect the integration systems. If you ever pay close attention to all the latest and greatest news from the Kubernetes community, I am sure you will hear a lot about the new "Service Mesh". And rumor has it that this is how integration can/should be done in cloud native world, but, is that so? Anyone who has ever worked on an integration project would tell you, it's a LOT more COMPLEX and can get worst overtime. I did a talk with Christian Posta in Red Hat Tech Exchange coming from a more comprehensive view of how different Red Hat technologies are applied under different patterns when building integration solutions. In fact he also did a great blog about it.


Since then, another topics has be…

JBoss EAP 6 - 效能調校 (一) DataSource 的 Connection Pool

效能沒有什麼Best Practice, 反正能調整的就那些。 通常,一個程式的效能大概有70-80% 都跟程式怎麼寫的其實比較有關係。

最近我最疼愛的小貓Puji 因為膀胱結石開刀的時候過世了,心情很差請原諒我的口氣沒有很好,也沒有心情寫部落格。

Puji R.I.P.

=======================正文=======================

這個題目很多人叫我寫,可是這題目好大,這分明就是整死我咩~
所以我會分幾段慢慢寫。

JBoss 的 SubsystemDatasource WebWeb Service EJB Hibernate JMSJCAJVM 調校OS (作業系統)

先來看一下 DataSource Subsystem, DataSource 的部分主要是針對Connection Pool 做調校。

通常,程式都會需要跟資料庫界接,電腦在本機,尤其是在記憶體的運算很快,但是一旦要外部的資源連接,就是會非常的耗資源。所以現在的應用程式伺服器都會有個Pool 放一些先連接好的 資料庫connection,當程式有需要的時候就可以馬上提供,而不用花那些多餘的資源去連接資料庫。

這就是為什麼要針對Connection Pool 去做調校。

以下會討論到的參數,都是跟效能比較有關係,Datasource 還有很多參數,像是檢核connection 是否正確的,我都不會提到。如果你追求的是非常快速的效能,那我建議你一個檢核都不要加。當然,這樣就會為伺服器上面執行的程式帶來風險。這就是你要在效能與正確,安全性上面的取捨了。 (套句我朋友說的話,不可能又要馬兒好,又要馬兒不吃草的..)

最重要的調校參數就是 Connection 的 Pool 數量。(也就是那個Pool 裡面要放幾條的connection.) 這個參數是每一個應用程式都不一樣的。

min-pool-size 

Connection Pool 最少會存留的connection 數量

max-pool-size 

Connection Pool 最多可以開啓的 connection 數量

prefill

事先將connection pool 裡面建立好min-pool-size 的connection.

我的建議是觀察一下平常程式要用到的量設定為 min-pool-size 。
加上…

Red Hat Fuse - Announcing Fuse 7 Tech preview 3 release.

Red Hat Fuse 7.0 technical preview three is out today! On the pathway to become one of the best cloud-native integration platform, Fuse gives developer freedom to choose how they want to develop the integration solution, where they want to deploy it and capabilities to address new integration personas that do not have development experience.
By supporting the three major runtime, developer is free to work on the runtime of their choice.By supporting standalone and cloud deployment, it simplifies the complexity to distinguish between these environments, allowing application to deploy freely among the environment of your choice. All levels of developers are welcome, you can either dive deep into creating customize complex integration logic, or using the new low code platform to quickly build a simple integration. In this Tech Preview release you get it all.
Fuse StandaloneSpring-boot for microserviceKaraf 4 for OSGi loverJBoss EAP for JavaEE developersFuse on OpenShiftPlugins for easy co…