# Web Application Development at Muze

At Muze (muze.nl) we've been building websites and web applications since 1998. In that time we've tried about everything at least once. But it seems we've landed on a style, or filosofy of web application design and development that is unique. Or at least it seems so to me.

So here I try to show how we approach building new applications for the web. And hopefully also why we do it this way. This is written mainly for our own use, but if you somehow find it somewhere online, welcome.

Before I start, let me explain some of our constraints and context. Muze is a small independent web development agency. We focus solely on web development, because we love the web. We still remember the world before the web, and it is almost impossible to explain to those young enough how the world worked or not worked before then. We believe the web is one of the greatest inventions of the 20th century. Yet we think it can be better. That is our aim, help to make the web a bit better.

Because we are a small company, with mostly smaller-sized clients, we can't focus large stretches of time on a single project. At best we manage to divide the work so we can work on a single project each day. This means that a lot of traditional wisdom about agile software development doesn't fit our workload. But the agile core does. We believe that the core idea about agile is to reduce the feedback loop. By which we mean: deliver a testable product as soon as possible and get meaningful feedback on it.

The meaningful feedback part has been the difficult part. Through the years we found that the only reliable feedback is when we deliver the final version. Only then the product is really tested, when users try to work with it. And that is when the bug reports come in. So these days we try to create as small and simple a 'minimum viable product' as we can. We deliver this as soon as we can. And then we start on the next, improved version. This idea is from the startup world, where it has become accepted wisdom. It also works for us and our clients.

The big problem is to convince the client to partition the work like this. Often a client has a grand vision, pie in the sky dream. And wants us to deliver a plan, with a budget that delivers that in total. As anyone with any software development experience will tell you, that is impossible. That is one of the reasons agile was invented. Unfortunately, most agile development systems rely on a smart, engaged client who can deliver meaningful feedback consistently during the development cycles. We found that our customers usually do not have the required knowledge to be able to do this. However, they do have all the knowledge needed to verify that a useful and usable product fits their need. So our main goal is to get to a MVP as quick as we can.

Because of the diversity of clients, and therefor problem domains. We have to accept that when we start a new project, we don't yet know enough to make all the right decisions. The first few attempt will miss the mark. But if we do the process right, our clients will tell us and we should be able to correct our mistakes without much wasted time and money. The first few releases are all about exploring the problem domain and possible solutions to problems.

This means that we cannot invest in complex and/or deep architecture upfront. We need to deliver a visible, usable product in just a few weeks. So we've created our own approach which we call 'frontend first'. This is an approach that works well when you are starting a new project. One where both the developer and the client still need to finetune what the problem is and how to solve it.

## Frontend First

This approach stems from our observation that quality feedback starts when an application is actually used in production. Design documents, paper or actual prototypes don't get the attention that they need.

So to get quality feedback as soon as possible, we start building a frontend application with the absolute minimum backend, or even no backend. We skip all considerations of security, reliability, scaling. What we do focus on is a useful and usable subset of the application and create a working application for that.

The frontend is build using a rapid application development methodology. We re-use standard designs, components and workflows. Maybe add a few design touches to make visually clear that the application is built with quality in mind.

If we need to store data in a backend, we implement this in as minimal a way as possible, with as much freedom to change as possible. In effect we usually create a freeform database or object store. The frontend application is in full control of the shape of the data.

The frontend application starts out without any fixed data model. We've created an approach with SimplyEdit where the actual visual html template defines how the data is structured. So if requirements change, all we need to do is to change a form or template. Everything else follows from that.

Then after we hit a version that actually matches the expectations of people using it, we start filling in the behind-the-scenes parts. We may add validation and security. Or we may focus on scalability and upgrade the backend. But always focused on a real need. If performance is not an issue, we don't spend time on that. Every cycle we focus on the most important thing to improve and deliver a new version with that improvement.

When the application is in a state that is good enough for the current needs, then we start to look ahead. What are the issues we expect we will need to handle in the coming months or year. Then set pragmatic targets for that. E.g. the dataset is now about 5.000 items. We expect it may grow to 10.000 in the next year. So lets set a target of 100.000, on order of magnitude greater, so if we hit that, we know we can let that issue rest for a while and focus on other issues. There is no need to think about handling 1.000.000 items, we don't expect to build the next twitter, web-scale is not a thing.

Now, any experienced developer will tell you that when you design a software architecture to handle 10.000 items, it is unlikely that you will be able to adapt it to handle 10.000.000 items. It is much more likely that you will need to throw away the old code and start from scratch. Frontend first tries to embrace this idea. The application should be designed in such a way that it contains loosely coupled components. Each component should be simple enough, with a small enough API, that it can be removed and replaced, without the need for a large refactor in other parts of the code.

This is the key for growing good software, make it so that you can 'prune' your code with ease.

## Boundaries

To make it easy to replace components, they must align with both technical and domain boundaries. This is a rather vague statement, so I will try to make this more clear.

A domain boundary is the border around a problem space, where inside this border a certain concept or term is clearly defined to mean one thing and one thing only. The smallest domain is around a single such concept. Whenever you design software, look for those boundaries and make sure you do not cross them in a single part or component of your software. Inside a single component, each concept should mean one thing.

Take for example the concept of a form. As a webdeveloper, a form to me is a set of html codes that together build a form in a webbrowser. But for a business person it may be a set of webforms over multiple pages. Another developer may include the validations and handling code. And someone else may mean the actual printed out form they're used to working with.

It is important to make sure that concepts and terms are understood by all parties involved in the exact same sense and on the same level of abstraction. Wherever there is a conflict, some parties use the same term, but mean different things, there is a boundary. Make sure this boundary is reflected in your code. Either by explicitly handling only one meaning, or by seperating the different meanings over different components.

Then if you've seperated you components by the business or problem domain, then you may further seperate it by technical domains. But make sure that you've handle the problem domain boundaries first.

On clear technical domain that we've landed on is to always seperate the frontend and backend. This means, we don't write frontend code in the backend, or vice-versa. The backend is usually a simple RPC (remote procedure call) API over HTTP that accepts and returns json, more commonly and incorrectly called a REST API.

In the old days, we used to write PHP code that printed HTML. Then we made template code to do that. These days all the HTML is just that, HTML. Then we make a javascript application that turns json responses into HTML in the client.

A good component has these characteristics:
- The code is short, aim for less than 1000 lines of code
- All API methods are at the same level of abstraction
- The API methods have clear semantics
- You don't need to create, use or transport abstractions (e.g. classes or types) from inside the component to outside or the other way around. (It may be an option though.)
- The component has no knowledge of things outside its own control. Where you would need this, that must be injected through objects or functions as a parameter to general purpose code.
- Datatypes are all as simple as possible. If there are domain specific datatypes that must cross technical boundaries, make sure these are defined in a seperate component.

Some examples we're using now:

- ARC is a set of components for backend applications (https://github.com/ariadne-cms/arc-arc). They are derived from common patterns extracted from Ariadne (https://ariadne-cms.org/), an application development platform in use since 1999. ARC/path is a component that abstracts the concept of a file path. It is both a set of functions and a datatype. The functions can and will operate on a string, but the datatype is a 'value class', which provides certain guarantees about the path. You can use strings and Path objects interchangeably. This concept of a path is used throughout the rest of the ARC components, so it has its own component.
- SimplyView is a set of components for frontend applications (https://github.com/simplyedit/simplyview). These are derived from solutions we've come up building web applications since about 2015. One component is simply.include. This allows you to embed a piece of html on a different page, perhaps a different domain. The main API here is just a standard HTML tag: `<link rel="simply-include" href="https://example.org/my-app">`. This will include the html at the referenced URL, and add all the stylesheets and javascript files, and run them in normal order according to the HTML specification. There is no need to call a javascript API. The `link` tag can be inserted in the DOM at any time. By making the DOM the API boundary, any other javascript code can just manipulate the DOM and has no need to even be aware of the existence of a simply-include link. A simple example is in [SimplyEdit](https://simplyedit.io/), where we can use the `<template>` HTML element to insert or remove HTML content. SimplyEdit has no knowledge of simply-include, and you can still use it. This is the advantage of finding the correct technical boundary, this is the most loose coupling you can get.

## Clean code

Writing 'clean' code is a good thing. But at Muze we found that it is much more important to create a clean architecture. If you write your components to be small and easily replaced, the code inside that component becomes much less important. It is still a good idea to write code that is easy to read and easy to test. But clean code in a dirty architecture is much worse than the other way around.

## Testing

Unit tests are a gods end... for the developer. They allow you to make radical changes and feel confident about your code. The unit tests form a sort of contract. "This is the functionality this component provides". You should write unit tests. You should design your code so that testing it is easy. Unit tests should be fast, because you use them while writing code to validate that you didn't forget something.

Unit tests aren't a goal in itself. 100% code coverage is not a goal. Even with 100% code coverage, you will not have tests for every eventuality. The best you say about it is that at least every line of code is tested at least once. This is not as meaningful as you may think. If you use this to make sure that there are no syntax errors in your code, you are missing out on a load of static analysis tools that do that job much better and with a lot less work.

Unit tests are also not integration tests. Integration tests are part of the validation tests. Validation tests are tests that check if the application fulfills the needs of the client. If the application is fit for purpose. These tests may and will run longer. They will test functionality that crosses boundaries. And you don't run these while refactoring a piece of code. Therefor the context and focus is on different things. It is important to test whether all components are wired together correctly, not if each component is adhering to its 'contract'. So these tests don't have to be exhaustive.

## Strong types

At muze we're working mostly with PHP and Javascript. This is historical, not necessarily by design. However there are good reasons to stick with these two languages, even though they're both maligned quite a bit online.

PHP is growing ever more towards an enterprise style language, unfortunately. We learned to love PHP for the ease with which you could integrate it with Apache. But more importantly at Muze we believe that the best parts of PHP are:
- Share nothing architecture
- Very limited variable scope
- Ducktyping
- Magic methods (or better: invisible encapsulation)

And for javascript, the best feature is that it is universally available in all webbrowsers.

For both we also like the development workflow, where no compilation is needed. You can change a line of code and see the result immediately. More so in javascript, but PHP today even has a REPL option (Read-Evaluate-Print Loop.)

But the community is moving more and more towards strong typing, both in javascript, through typescript, and in PHP. And we're not following along. Here's why.

Using strict or strong types, you give up quite a bit of flexibility in how you design your sofware. You will need to write more code than you would need when you use ducktyping. The promise is that you get that back in a more reliable code base, where the compiler will tell you upfront when you make a mistake.

However, to use this, you will need a compiler. Typescript provides one, so that works, but PHP doesn't. In the case of PHP you will need an IDE or a seperate static analysis tool. Otherwise you are back to runtime errors.

And this need for a seperate step between writing code and seeing the results, creates more distance between you as a programmer and the code you are working on. The added code and complexity also work to create more distance. It is much less easy to work with and reason about such code.

There is certainly an advantage to strict typing, but we believe these advantages in PHP are seldom reached. The biggest advantage in PHP lies in the use of value objects. There you can specify not only the meaning of a parameter, but also be assured that its value is valid. So in complex algorithms or places where the validity of data is very important, strict typing can be very useful. Where it is utterly useless is when you add a type hint for a basic type like `int` or `string` or even worse `array`. At that moment you don't add any value to your code, you only make it more difficult to use.

A concrete example comes from our ARC library. The ARC\cache component defines a class that allows you to cache calls to any other object, no matter what class. It does so by creating a proxy object. You inject the object you want to cache in the constructor of the cache (using a factory method for a simpler API.) Then you can use the cache proxy anywhere where you would use the original object. This works because we don't use strict types. The cache proxy object clearly has a different class then the proxies object. We can't define an interface that we could add dynamically to the cache proxy. We can't create a trait that the proxied object could dynamically use. The only option you might have is to dynamically create a new class, which PHP does support today. But you would need to use `eval()` (or alike) to create one that implements the correct interface.

This is where ducktyping shines. Whenever you create an API function that expects a parameter to be a `duck`, you write an assertion that tests the specific functionality that you require. You can check if it has a `quack` method, you can even call its `walk` method and check that it returns `like a duck`. If that is all you need, you are good to go. Now anyone using your code can create their own duck. You may use an interface, or a trait, but you are not required. You can create a proxy, like our cache example, or create a completely new duck from scratch.

Back to typescript. I really want to like it, and use it. But every time I try, something makes me give up. It is one part the need for the compile step. Their are scripts that do this automatically whenver a file changes, but they suffer from configuration complexity. And usually they want to be in control of the browser and the webserver. But most importantly, the typescript compiler uses Babel. Babel is impressive. But it is also unstable and unwieldy and overly complex. Babel tries to do everything and ends up being almost unusable as a result. And by that I mean that the ecosystem around Babel changes so often so much, that a project that I've not looked at for a few months, suddenly stops working when I set it up again. Some dependency somewhere has made a breaking change, without the common courtesy of a major release. Now you have to go digging again on the internet to find out what the cause is and how to solve it.

We're not the only ones decrying the deplorable state of modern frontend development with javascript (or typescript). The toolchain complexity here is through the roof. The tools themselves start out sensible and usable, and then they add another feature, and another. And then you need to add a configuration file, which becomes its own javascript program, with less than stellar documentation. And then there is a breaking change. And then I give up, and go back to plain vanilla javascript running directly in the browser and the world is great again.