Making and Breaking down a Monolith

Making and Breaking down a Monolith

In 2016 I founded ImperialPlugins. ImperialPlugins is an eCommerce marketplace for game server plugins. It was also my first website. Yes, my first website was a eCommerce site made from scratch.

Making an eCommerce site from scratch

Before founding ImperialPlugins I was selling custom Unturned plugins, but with each sale it was getting harder and harder to track customers and to distribute updates. I was also manually selling the plugins via Steam and Discord. It was clear that a website would be needed to solve these issues.

Now you might wonder why I did not use something like Shopify, Magento or WooCommerce and maybe it would have been indeed the better choice. But there were some factors to consider that made me make my own solution:

  • I wanted to get into web programming and explore new technologies
  • ImperialPlugins has a lot of complex business logic like whitelisting, updating and licensing. It would take much time to develop extensions for an existing solution
  • I do not like PHP and I do not want to use it to develop extensions

The first iteration only had a single MVC project and it was like that for long time. Today the backend of ImperialPlugins consists of 12 microservices, 113 projects and 652 classes.

Separating layers

Like mentioned earlier, the first iteration of ImperialPlugins was a single ASP.NET Core “MVC” project. I put “MVC” in quotes because this wasn’t really MVC at all. Forget about layers, not even Entities and DTOs were separated. Views processed business logic in them and controllers only were there to render the views directly. It was a nightmare architecture wise, but what can you expect from the first website I made.

The second iteration of ImperialPlugins was about separating layers. The project was rewritten from scratch, the solution now consisted of a “Web”, “Infrastructure” and “Core” layers. It was still garbage architecture and bad code-wise, but at least some parts we separated now.

Using Angular for an eCommerce application

With the separation of layers, the frontend moved to Angular and the backend was processing and returning data now. It turns out that making an eCommerce application with Angular was very difficult. There were a lot of issues:

  • Most importantly, SEO isn’t that great with Angular (this was solved by using Angular Universal)
  • The site is slow and initial load takes a lot of time
  • Some JS libraries did not have Angular bindings, finding libraries could be difficult
  • Good user experience is more difficult especially when lazy loading content

The difficulties of making an eCommerce application with Angular are out of scope of this article, so maybe I will address them in a separate article in the future.

Migrating data

With the rewrite the entities had changed completely. The changes were too deep to do them in a way using EntityFrameworkCore migrations, so I had to create a separate migrator application. The migrator imported the old and new entities and then migrated the entities by recreating, mapping and inserting them again. One important thing was the  order of the migrations. For example, you can not migrate ProductFiles before Products, because ProductFiles has a Foreign Key on Products.

Learning more about software architecture

In December 2017 I applied to Kayten Technologies, my first job application ever. After showing the source code of ImperialPlugins, which wasn’t much impressive back then, I still got accepted for a 2 months internship followed by a fixed part time job. During this time, thanks to our project lead and software architect Rashiddudin Yoldash, I learnt a lot more about software architecture and especifically domain-driven design.

Domain-driven design and ASP.NET Boilerplate

While developing a dashboard for proximity solutions at Kayten, we were using domain-driven design in combination with ASP.NET Boilerplate (ABP). It took a couple of months to understand the code and architecture of ABP, but it was worth it. If you are interested in using DDD with ASP.NET (Core), the easiest and fastest way is likely by using ABP. There is really no other comparable DDD framework for .NET based applications.

ABP Framework vs. ASP.NET Boilerplate

After working with ASP.NET Boilerplate at Kayten I decided that I wanted to use it for ImperialPlugins as well. After researching a bit, I learnt that there is a new project by the team behind ASP.NET Boilerplate, the ABP Framework. The new framework was very promising: it had an updated architecture which was made with microservices in mind. The older ASP.NET Boilerplate was made back in legacy ASP.NET MVC times, when ASP.NET did not even offer builtin dependency injection. The new ABP Framework is directly integrated with ASP.NET Core. You can learn more about the ABP Framework in the ABP blog.

Migrating to ABP Framework and using Domain-driven design

Migrating to ABP Framework wasn’t straightforward. It needed another rewrite, but this time ImperialPlugins was much bigger. The backend consisted of ~200 classes at that time. Unlike the first rewrite, this time it took a couple of months to complete the rewrite. ABP Framework was also not ready for production yet and often contained bugs which I had to wait for to be fixed. The backend was now consisting of 7 projects:

  • Domain.Shared (shared domain constants, like enums)
  • Application.Contracts (DTOs and service interface definitions)
  • Domain (business logic)
  • Infrastructure (EntityFrameworkCore & repositories)
  • Application (app services, mapping to DTOs, etc.)
  • HttpApi (Defining http endpoints via controllers)
  • HttpApi.Host (The ASP.NET Core web app project)
  • and a standalone database migrator application
Migrating data again

With the migration to ABP Framework the entities have fundamentally changed again. It was time to reuse the old entity migrator code mentioned earlier. I used the same code to migrate from ASP.NET Core entities to ABP Framework (like users, external logins, etc.). Since I had to do such an migration anyway, I also redesigned a lot of entities and split some as well.

Migration code was like this:

var aspNetUsers = await aspnetCoreDbContext.Users.ToListAsync(); // Imperial doesnt have too many users
foreach(var aspNetUser in aspNetUsers)
{
    var abpUser = new AbpUser();
    // map properties
    MapAspNetUser(aspNetUser, abpUser);
    await abpDbContext.Users.InsertAsync(abpUser);
}
// save changes

Dockerization

Before migrating to ABP Framework, deployments were done primitively: I compiled the project and directly uploaded the dll files to my linux server. This was very problematic since it caused a lot of downtimes when things did not go well, reverting to an earlier version was also very difficult.

After migrating to ABP Framework, I completely changed the deployment process. I created Dockerfiles for the frontend, the backend and for the standalone database migrator, installed and integrated Teamcity for CI/CD and deployed a Portainer instance for easier docker managment. TeamCity now automatically builds staging and production images on each git commit and then pushes them to ImperialPlugins’ private docker registry. Deployment is then done manually with Portainer.

Modularization

This was the first step to breaking down the monolith.

After migrating to ABP Framework, it was time to modularize the application. Modules are a specific set of classes with a shared context. ABP Framework has a first class support for modules and actively encourages to use them. The framework itself is also modular. In practical terms, ABP Framework modules are used to define and register dependencies of an assembly. Good modularization reduces coupling, increases cohesion and helps separating concerns. The modularization of the ImperialPlugins backend resulted in 13 different modules like “products”, “merchants” and “webhooks”. Each module has it’s own DDD layer projects like Domain, Infrastructure, Application, etc.

During the modularization it was important to have well defined dependencies and to avoid circular dependencies, which took some time. I had to plan well of what classes a module should consist of, which turned out to be not too difficult. Figuring out the dependencies and resolving problems related to that took much more time.

From modules to microservices

Converting modules to microservices was very easy. It is one of the areas where the new ABP Framework and DDD shines. Since modules are already very isolated and made with microservices in mind, it was easier than expected to convert them to standalone microservices. Of course some deeper code changes were needed too, especially for some events, but the ABP Framework provides a simple distributed eventbus abstraction that was very easy to use.

Each service has it’s own modules, layers, build configurations, version and docker image. Messaging between modules, like publishing product update notifications on Discord, is done via RabbitMQ.

Mono Repository

It would have taken too much effort to have a different repository for each microservice. I would have to set up a private NuGet feed and had to deploy updates for each single change I wanted to do and test. Having a mono repository where all services are in a single repository was a good solution. The only problem here was the integration to the build pipeline. At first each commit started a build for *all* projects, which, of course, was very inefficient. Thankfully TeamCity has an option to trigger a build for a project only if changes were done in a specific directory. This is still not the best solution, since a change in one module could affect how a different module works, but would not automatically rebuild it. I would have to add all dependencies to the trigger folders list in TeamCity for each project. It would be very difficult to maintain such a list and I have not found a solution to this problem yet.

Visual Studio becomes slow

With 113 projects in a single solution Visual Studio has become slow, even worse when using Resharper. I plan to split the solution, where each microservice would have it’s own solution. Unloading unused projects is a workaround for now.

API Gateway

The first microservices had their own subdomains, like xxxx-api.imperialplugins.com while the monolith backend remained at api.imperialplugins.com. This was not an issue until more microservices were added. Creating a subdomain for each service is a slow process and requires maintenance. Using an API gateway would allow me to host all services under the same subdomain and to load balance the services in the future. ImperialPlugins is currently hosted on a single machine, so I chose Ocelot as API gateway. It is very easy to use and setting it up only took some minutes.

Next up

The breaking down part is still not done. Around 50% of ImperialPlugins’ backend has been separated to it’s own microservice and new services are always added as microservices. So currently the backend consists of a mix of a modular monolith and some standalone microservices.

There is still a lot to do, which hopefully will be covered in Part II when they are done:

  • Removing shared access to tables between services (a private HTTP API should be used instead)
  • Database migrations for microservices
  • Kubernetes integration for container orchestration
  • ELK / seq integration for log management
  • Prometheus integration for monitoring
  • Istio integration for service mesh
  • Automatic deployments with TeamCity
  • Unit Tests (yes, ImperialPlugins still does not have them)
  • Green and Blue deployments
  • Getting rid of local storage

No Comments

Add your comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.