Portbase's Faster Application Development

Portbase is the spider in the web of Dutch logistics. Founded in 2009, the non-profit organisation connects all parties in the logistics chains of the Dutch ports. Portbase facilitates data sharing between companies and information exchange with governments to work faster, more efficiently and at lower costs.

Like so many organisations, Portbase's IT architecture was until recently a real monolith: one jumble of services that were dependent on each other. This jungle meant that development took a long time and that many people had to pull the cart at the same time, with joint releases occurring at most once every fortnight. Portbase wanted to change this with autonomous teams responsible for certain functionalities without depending on other teams, other infrastructure and the monolith.

As Lead Software Engineer, René de Waele set to work in 2018 with his team (and later with other teams) to realise this transition. "The starting point here was that we not only made the software scalable but also that each department could focus on domain aspects, or business rules instead of purely dealing with technology in the code. In short, in your code only write things down the way you want them."

The basis for the solution was a tool De Waele had previously developed: Flux Capacitor. "Swapping a monolith for microservices is attractive, but communication between the services is quite a challenge," he explains. "Thanks to Flux Capacitor, exchanging messages between applications and launching new services is a lot easier. Once you have launched a service, it can communicate with any other service that is connected."

How did René and his team achieve this? "If you go back to basics and look at what an application actually does, you can distinguish three things. Commands are the requests to the application, for example: I want to register a new visit by a ship. Queries are requests, in this case for example: give me all the visits of this one ship. Then there are Side effects, for example notifying the port or customs that the ship has been notified. In a dynamic environment where these kinds of messages are exchanged in large numbers and where reliability is essential, the traditional way works cumbersome. For instance, services are often called multiple times, requiring a load balancer to keep things running smoothly. Because you have more and more end points, the infrastructure becomes needlessly complicated, with all those end points also needing to be secured. A specialised team of cloud engineers has to deal with these issues, and then you still have to launch your product."

This can be avoided by having the queries and commands handled by a central message broker. The various microservices are not tied together and none of them are directly connected to the internet. "They are not applications you can go to in your browser. That also means they are less vulnerable to hacker attacks. Moreover, the message broker does not understand anything about the messages at all, but only stores which queries and commands are handled. The message broker can also take care of load balancing. Moreover, we can track exactly what happens. Not just the events, but also all incoming API requests, all messages coming in, which browser was used and so on. At Portbase, we have now created dozens of microservices that read the logs completely independently."

Testing, too, is now much easier and produces much better results. "If you want to test different services, it is often a cumbersome process. You have to create a test environment with a database, different services and a network connection. Because this is cumbersome and error-prone, in practice the functional behaviour of the application as a whole is rarely tested. In testing, we are interested in the behaviour of the whole application. We want to test that constantly, without needing to know how the application manages it. That's how we wrote all our tests."

Meanwhile, the main applications are completely separate from the monolith. "The transition took no more than six months per application and could be done with a tenth of the number of lines of code compared to the old situation. The bottom line is that 90 per cent of the application consisted purely of technology. Instead of going live with a new version twice a month, we can now do it five to 10 times a day. In fact, we now go live with different teams about a hundred times a day, always with real features." Despite putting so many new things live, there are far fewer problems. "Before we started, the team had a hefty backlog and it was growing faster than it was decreasing. After the six-month transition, the backlog was almost completely empty."