1. Code

At the code level, agentic coding replaces procedural control with intent-driven collaboration. Developers increasingly express goals, constraints, and patterns of reasoning, while autonomous coding agents interpret and act on these prompts.

As developers adapt to this new approach the focus increasingly becomes around writing concise specifications. Instead of working line-by-line, developers use specifications to orchestrate AI agents that:

• Generate and refactor code according to localised patterns and style rules.
• Rely on inbuilt tooling and pipelines to enforce coding standards and preferences.
• Review implementations based on industry best practice for non-functional requirements (e.g security and performance).

As agents become responsible for generating code and humans become responsible for writing the specification, the code becomes an artifact. The specifications, guardrails and tooling built into the repository are the inputs into the LLM which produces the code. Similar to the way a compiler produces machine code from source-code, albeit less deterministic.

Working with non-deterministic outputs from an LLM can be difficult, but using the specification it is possible to verify that the source-code generated meets the original requirements either through testing or using other LLMs for verification.

As generating code becomes “cheap”, the code becomes a disposable artifact that can easily be regenerated. That changes the balance of trade-offs that have long been discussed in the industry.

Frameworks vs Libraries

Discussion around using a library vs a framework goes back to as early as 2005 when Martin Fowler wrote in his post about Inversion of Control:

“Inversion of Control is a key part of what makes a framework different to a library. A library is essentially a set of functions that you can call, these days usually organized into classes…

A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes. The framework’s code then calls your code at these points.”

Martin Fowler doesn’t express a preference for libraries or frameworks in his post but the debate quickly grew. David Heinemeier Hansson’s Ruby on Rails Doctrine supporting frameworks and Adrian Holovaty of Django cautioning against frameworks.

Over time it’s become clear that there are trade-offs to both:

• Frameworks offer speed and ease of use but become restrictive and can quickly become a form of vendor lock in.
• Libraries offer more power and flexibility but require more set up and wiring from the developer.

This can also be framed as Speed vs Power.

Agentic coding changes this trade-off by increasing the speed of development when using libraries. The initial cost of set up when composing libraries is offset by using a coding agent to do the work. Given access to curated collections of libraries or components through an MCP server, agents can correctly assemble individual components efficiently.

That’s not to say that agents don’t work well with frameworks, the strict guardrails often help, but they do not solve the fundamental downsides of a framework. Using agents with libraries allows you to have the power and flexibility libraries already bring, but the speed and ease of use that is typically associated with frameworks.

2. Architecture

Large organisations tend to follow one of two philosophies when designing their macro-architecture: Configuration Driven or Code Driven. The choice often comes down to how best to achieve re-use across different services in the organisation.

Configuration Driven Architecture

The Configuration Driven (or Low Code) architectural approach opts for feature-rich centralised services that provide a lot of reusable functionality but require lots of configuration. The premise is that you can write the functionality that lots of services need in one place, allowing other services to re-use it by providing a configuration.

Over time Configuration Driven Architecture often falls prey to the Inner-platform effect where the configuration becomes increasingly complex in order to support all the requirements of the users as the system evolves.

There are parallels between Configuration Driven Architecture and frameworks. Configuration can allow you to move quickly if you paint within the lines, but it quickly becomes difficult if you want to paint outside them.

The complexity of the configuration quickly grows as the system matures and custom DSLs or configuration languages often become necessary. You then face a form of vendor lock-in where developers are forced to learn organisation specific tooling, rather than industry-wide programming languages and libraries.

Code Driven Architecture

Code Driven Architecture is a more engineering led approach where there are centralised services but as with using libraries, the services are just reusable functionality to be called upon by consuming services via API call. The set up and orchestration are done by the consumer, which comes at a cost but also allows a greater degree of flexibility and autonomy.

As Martin Fowler’s 2014 page on Microservices states that “The microservice community favours an alternative approach: smart endpoints and dumb pipe”, similarly, Sam Newman’s book Building Microservices warns against heavily centralised architectures: “Avoid approaches like enterprise service bus or orchestration systems, which can lead to centralization of business logic”.

As with libraries and frameworks, the trade-off is Speed vs Power. Configuration Driven Architecture promising speed and Code Driven Architecture power and flexibility.

Using agentic coding we can again adjust that trade-off. Teams are able to quickly compose reusable pieces of architecture in a way that works for them and mitigate the high set up cost without the constraints of a custom DSL or configuration language.

Common patterns and approaches from across the organisation can be distilled into AGENTS.md guidance, centralised prompt libraries or even dedicated sub-agents to call upon, all making it easier for the agent to build services in a way that is compatible with the ecosystem it is living in. Artifacts like ADRs stored in other codebases are invaluable resources to help agents understand the historical decision making that has led to the organisation’s current approach.

So instead of large, centralised orchestration layers, intelligence moves closer to the edge:

• Systems are composed of smaller, self-sufficient services that encapsulate their own decision-making and processing logic.
• Configuration-heavy or low-code abstractions give way to clear, composable components that can evolve independently.
• Shared libraries and infrastructure still provide coherence, but not command. They serve as enablers, not bottlenecks.

This decentralisation empowers individual service teams to innovate rapidly without waiting on central gatekeepers. Architecture becomes a distributed ecosystem where composability replaces centralisation, and agility replaces dependency.

3. Team Topology

Centralising functionality behind configuration based services comes with another drawback: the teams that run the centralised services quickly become a bottleneck, as the aggregates all of the demand for new features from every one of their consumers.

Martin Skelton and Manuel Pais’ book Team Topologies outlines the problems in terms of value-stream alignment. If the delivery teams are not empowered to deliver value for their users or stakeholders, their requirements are pushed on to the team that can deliver it. This results in complex cross-team coordination for developing and releasing functionality.

In essence, it’s important not to confuse the technical platform that developers use to build their solution with the solution itself. As Martin and Manuel state:

“The most important part of the platform is that it is built for developers.”

Agentic coding reshapes team topology from centralised delivery functions into value-stream-aligned teams where value is delivered “at the edge”. No longer dependent on a central platform or integration groups to deliver functionality, teams own the full lifecycle of their services.

• Teams align around products or domains rather than technical specialisms.
• Dependencies on centralised delivery teams are reduced as teams become more autonomous.
• Engineers operate within cross-functional squads that combine development, testing, and deployment capabilities, supported—not controlled—by platform services.

This value-stream alignment mirrors the autonomy seen in the code and architecture layers. Teams become self-contained, adaptive units capable of shaping their own pipelines and responding directly to user needs.

Ultimately Conway’s Law takes effect and the organisation decentralises to the same degree as its software.

The Emerging Pattern: Empowerment Through Decentralisation

The common thread through all these layers is decentralisation and empowerment “at the edge”. As agentic coding is introduced at the very bottom of the stack, the ramifications rise to the top. Developers and development teams become more empowered to deliver functionality unencumbered by centralised systems holding them back, and they do it with the speed and ease of use typically offered by the centralised systems.

The rise of agentic coding (or vibe coding) has generated lots of debate within the industry and opinion is split on whether it’s a net negative or positive. Proponents argue that there are massive productivity gains when using agentic coding, but opponents argue that there are risks around security, code quality and governance. What both sides agree on is that agents can generate a lot of code, and they can do it quickly.

As the industry matures it’s becoming clearer that the risks can be mitigated with appropriate guardrails and tooling. It takes time to distill and embed the organisational knowledge, practices and engineering standards but doing so not only makes developers more productive, but also allows organisations to move from centralised control to empowered, autonomous teams.

HMCTS, the courts and tribunals service within the Ministry of Justice, handles millions of Civil, Family and Tribunal cases each year. All of those cases flow through a case management system called CCD. CCD has its quirks - it’s configured via Excel spreadsheet and it implements business logic through an intricate system of callbacks into each team’s APIs - but most of the time it just works. Every now and then, though, case data goes missing, which leads to… distress.

I was working on one of the smaller services at HMCTS where an issue was reported on a case. After triggering an event, most of the case data simply vanished. Not all of it (that would have been too easy); specifically the caseworker’s notes on the case had gone missing.

The audit trail showed nothing sinister: CCD’s callback came in, our Spring Boot service answered 200 OK, and everybody went back to work… until someone opened the case and found most of the data was missing.

The only clue: a lonely warning

Buried in the logs there was a single line:

Response already committed. Ignoring: org.springframework.http.converter.HttpMessageNotWritableException

That warning was our only breadcrumb. No stack-trace, no 500, no “request failed” banner on CCD - just a shrug from Spring saying too late, the headers are out the door.

A getter with a secret

With a debugger hooked into the callback we discovered the culprit: during the Jackson serialization a getter sometimes threw a NullPointerException. You would assume that throwing an exception during serialization would raise a 500, but clearly this wasn’t happening. We took a closer look at how Spring handles exceptions to find our log message:

if (request instanceof ServletWebRequest servletWebRequest) {
    HttpServletResponse response = servletWebRequest.getResponse();
    if (response != null && response.isCommitted()) {
        logger.warn("Response already committed. Ignoring: " + ex);
        return null;
    }
}
return createResponseEntity(body, headers, statusCode, request);

For context, HTTP responses happen in two parts: first the status and headers are sent, then the body starts streaming. Tomcat keeps those headers in an internal buffer until it fills up or you explicitly flush. When that happens the response is marked committed (HttpServletResponse#isCommitted() returns true). Before that moment Spring can still change its mind and turn the call into a 500 error; after it, the status code is already on the wire and cannot be rewritten.

So in our world:

• If Jackson explodes before the headers are committed, Spring’s error handler tears down the partial response and returns a clean 500 Internal Server Error.
• If the exception happens after the commit, Spring can no longer touch the status code. All it can do is log “response already committed” and keep streaming - which leaves the client holding a valid but half‑filled JSON document.

Re-creating the issue

At first we couldn’t reproduce the bug locally; every attempt gave the expected 500. The breakthrough came when we realised two things mattered:

1. Size - the JSON had to be big enough to flush the buffer early.
2. Order - the “big” field had to appear before the naughty getter.

So we inflated an innocent field (in our case the applicantName) until the payload passed the buffer threshold. Now the exception triggered after the response was committed, and boom - we hit the same “data-eating” path Spring had taken in prod.

CCD received a perfectly valid but truncated document. Fields up to, but not beyond, the failing getter survived; everything else evaporated.

Why the JSON stayed valid

Jackson streams objects field-by-field. When the getter exploded, the serializer simply abandoned the remaining properties but still wrote the closing braces it had already buffered. The result was syntactically correct JSON - just missing half its content. Because the transport never broke, CCD had no reason to reject it.

Takeaways

1. Never put real logic in a getter. Jackson will call it during serialisation; if it can throw, it will someday.
2. Buffer thresholds matter. A 5 kB object and a 50 kB object can follow completely different exception paths.
3. Monitor warnings, not just errors. When Spring says “response already committed” it’s really saying “I gave up”.

We’ve patched the offending getter, cleaned up our DTOs, and set up an App Insights alert for that warning so the next silent 200 won’t slip through the cracks.

CSA refresher

The connection scan algorithm has been covered on this blog before and there is a reference implementation available on github.

In essence, trips are broken down into connections between stops A and B and given a departure and arrival time. Footpaths are created as a pseudo-connection between A and B with a duration.

The list of connections is sorted by arrival time and then iterated a single time. Each connection is tested to see if it is reachable, and if it is it is tested again to see if it improves the arrival time at it’s destination. If it does improve the arrival time at it’s destination the earliest arrival time at that stop is updated and the connection is set as the “best” connection to get to that stop:

Without the application of interchange time the implementation of isReachable and isBetter is:

Original implementation

The original paper updates the isReachable function to detect whether interchange time is required and check for it if necessary:

While this change appears to be simple, it introduces an edge case that results in sub-optimal journeys being created.

Given two trips, Trip 1 and Trip 2, running in parallel along stops A, B, C and D. Trip 1 arrives earliest at A, B and C and Trip 2 arrives earliest at D:

Trip 1:
  A -> B, 1000, 1010
  B -> C, 1010, 1020
  C -> D, 1020, 1040

Trip 2:
  A -> B, 1005, 1015
  B -> C, 1015, 1025
  C -> D, 1025, 1035

If the interchange time at stop C is greater than 5 it makes it impossible to change from Trip 1 to Trip 2 to get the earliest arrival time at stop D.

The crux of the issue is that the algorithm assumes that using the best arrival time at A, B and C will result in the best arrival time at D. Applying interchange time breaks this assumption. In this case, how you arrive at stop C is important.

Updated implementation

In the 2017 paper, interchange time is applied using footpaths. A footpath is added between each platform within the station and the interchange time between those platforms is used as the duration of the connection. If platform information is available

This approach gives a much more granular level of interchange time, and removes to modify the isReachable check, but it does require platform information to be available ahead of time. In the UK it is quite common for platforms to change at short notice which could result in unrealistic journeys being suggested.

The edge case from the original application of interchange still applies. Storing the best result for stops A, B and C means that the earliest arrival at D is unobtainable.

An alternative

The connection algorithm stores its results in two indexes, one with the earliest arrival time at each stop and the other with the connection that got there. To avoid the issue with interchange time its possible to modify the isReachable method so that it first checks whether the connection is reachable with interchange time, and if not, it checks whether it was possible to board to the connections trip at an earlier point. For this an index of boardable trips needs to be maintained:

While this does fix the issue highlighted above, it comes with a drawback. The best connection to get to D is now set to the Trip 2 C->D connection, but the best connection to get to C is still Trip 1 B->C and it is not possible to change between them at C. Luckily some fixes I have previously outlined will correct this as it eliminates redundant legs.

Takeaways

Interchange time is a messy real-world problem that isn’t always accounted for in algorithm papers, and even when it is, it is difficult to do comprehensively. The Connection Scan Algorithm is admirable for being simple and fast, but there are so many caveats and workarounds that need to be applied that it is usually best to stick to other algorithms such as Raptor or Transfer Patterns for anything mission critical.

Setup

First of all, we need to create a table for job queue and add some jobs:

CREATE TABLE job (
  id INT(11) unsigned auto_increment PRIMARY KEY,
  state ENUM("NEW", "DONE")
);
INSERT INTO job VALUES
  (null, "NEW"),
  (null, "NEW"),
  (null, "NEW"),
  (null, "NEW"),
  (null, "NEW");

Locking

In a distributed system with multiple instances of your application there is always the risk that two instances will attempt to take the same jobs from the queue. Luckily, almost all relational databases come with some form of locking built in. In MySQL’s case we can use SELECT ... FOR UPDATE

Let’s simulate multiple instances of our application trying to pick up the jobs at the same time using multiple terminals.

Terminal 1:

SELECT * FROM job WHERE state = "NEW" LIMIT 1;
+----+-------+
| id | state |
+----+-------+
|  1 | NEW   |
+----+-------+

Terminal 2:

SELECT * FROM job WHERE state = "NEW" LIMIT 1;
+----+-------+
| id | state |
+----+-------+
|  1 | NEW   |
+----+-------+

As you would expect both instances have now started processing the same job. Not what we want.

However, using SELECT ... FOR UPDATE inside a transaction will lock the table until it’s been updated by the instance that locked it.

Terminal 1:

START TRANSACTION;
SELECT * FROM job WHERE state = "NEW" LIMIT 1 FOR UPDATE;
+----+-------+
| id | state |
+----+-------+
|  1 | NEW   |
+----+-------+

How switch over to Terminal 2 and try the same thing:

START TRANSACTION;
SELECT * FROM job WHERE state = "NEW" LIMIT 1 FOR UPDATE;

At this moment Terminal 2 is locked until Terminal 1 commits the transaction and the query will not return a result.

Terminal 1:

UPDATE job SET state = "DONE" WHERE id = 1;
COMMIT;

Now Terminal 2 has been unlocked and returns:

+----+-------+
| id | state |
+----+-------+
|  2 | NEW   |
+----+-------+
1 row in set (37.304 sec)

Not only did it wait for the lock to be free but it also waited for the update to be applied and selected the next job in the list.

Scaling

You may have noticed that this approach locked the whole table, meaning that only one instance can process jobs at a time. This is also not what we want.

Luckily, the second instance can ask MySQL to skip over any locked rows when it tries to select a job.

Lock a row in Terminal 1:

START TRANSACTION;
SELECT * FROM job WHERE state = "NEW" LIMIT 1 FOR UPDATE SKIP LOCKED;
+----+-------+
| id | state |
+----+-------+
|  2 | NEW   |
+----+-------+

And switch over to Terminal 2:

START TRANSACTION;
SELECT * FROM job WHERE state = "NEW" LIMIT 1 FOR UPDATE SKIP LOCKED;
+----+-------+
| id | state |
+----+-------+
|  3 | NEW   |
+----+-------+

And now our second instance selects the row that is not locked by the first instance. It’s now safe for each instance to start polling for jobs without fear of stepping on each others toes. You can add as many instances as you want to start processing jobs to horizontally scale your service.

Postgres 9.5 and MySQL 8.0 both have support for the SKIP LOCKED feature, unfortunately MariaDB does not currently support it.

There’s no doubt that queues can grow more complex with features like TTL and dead letter queues, but if you want something simple and don’t need all the bells and whistles of something like RabbitMQ then this approach should prove simple and scalable.

In this guide I’ll walk through setting up a database backup where hourly dumps are stored for a day, daily for a week, weekly for a month and monthly for a year.

Setup

If you have an existing node project you can just add this script and some dependencies to it. If you don’t then set up a new project with npm init. Then add the AWS SDK and mysqldump libraries:

npm install --save mysqldump aws-sdk

Generating the MySQL dump

With the project set up we need a script to trigger the backup every hour. Create a file called backup.js

async function backup() {
  const now = new Date();
  const dateString = now.toJSON().substring(0, 16).replace(":", "");
  const filename = `db-${dateString}.sql.gz`;
  const path = "/tmp/" + filename;

  await mysqldump({
    connection: {
      user: "root",
      host: "localhost",
      password: "",
      database: "mydb"
    },
    dumpToFile: path,
    compressFile: true,
  });
}

// run the backup function every hour
setInterval(backup, 3600 * 1000);

Be sure to plug in the correct database credentials to the mysqldump call. One thing to note is that the library does not accept null passwords, but it does treat an empty string as null.

If you run the script with node backup.js a backup file will be created in the /tmp folder every hour.

Setting up an S3 bucket

Log in to your AWS console and navigate to S3. Select create a bucket:

Make note of the name as it will be needed later.

Now we need to create the S3 lifecycle policy to remove the old backup files. Click on the bucket and go to the Management tab:

Select add lifecycle and then enter the name RemoveDaily and apply it to the hourly/ folder

Next, set the policy to expire files older than 1 day:

Now repeat this process so that items in the daily folder expire after 7 days, weekly expire after 31 days and monthly expire after 365 days.

Uploading to S3

Now the dumps are generated and the S3 bucket has been set up we can upload the dumps. You’ll also need to generate an API key to use with the AWS SDK. Typically these are stored as the environment variables: AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY.

async function upload(filename, path, date) {
  const s3 = new S3({
    accessKeyId: process.env.AWS_ACCESS_KEY,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
  });

  const s3directory = getS3Directory(date);

  return new Promise((resolve, reject) => {
    s3.upload({
      Bucket: "yourbucketnamehere",
      Key: s3directory + filename,
      Body: fs.createReadStream(path),
    }, (err, res) => err ? reject(err) : resolve());
  });
}

function getS3Directory(date) {
  if (date.getHours() === 0) {
    if (date.getDate() === 1) {
      return "monthly/";
    }

    if (date.getDay() === 1) {
      return "weekly/";
    }

    return "daily/";
  }

  return "hourly/";
}

The getS3Directory function will categorize the current backup by classing it as “monthly”, “weekly”, “daily” or “hourly” depending on the date given.

Now we just need to update our backup function so that it uploads the file and then deletes it afterwards:

async function backup() {
  const now = new Date();
  const dateString = now.toJSON().substring(0, 16).replace(":", "");
  const filename = `db-${dateString}.sql.gz`;
  const path = "/tmp/" + filename;

  await mysqldump({
    connection: {
      user: "root",
      host: "localhost",
      password: "",
      database: "mydb"
    },
    dumpToFile: path,
    compressFile: true,
  });    

  await upload(filename, path, now);

  fs.unlink(path, () => {});
}

Now when you run node backup.js it will take a dump of the database every hour and upload it to your S3 bucket.

One thing to note about the S3 policy is that it is only executed once a day, which means your hourly backups won’t expire exactly when they a supposed to but they won’t be more than 24 hours out, and it never hurts to have an extra hourly backups lying around.

Generating the reports

Before getting started, I would highly recommend using Damian Szczepanik’s cucumber reports as they are much more legible than the default HTML reports:

There is a handy guide on how to them up, but the short version is to add the dependency and configure your test runner to use the plugin:

dependencies {
    // ... other dependencies omitted
    testImplementation("io.cucumber:cucumber-java:5.1.3")
    testImplementation("io.cucumber:cucumber-junit:5.1.3")
    testImplementation("de.monochromata.cucumber:reporting-plugin:4.0.29")
}

configurations {
    cucumberRuntime {
        extendsFrom testImplementation
    }
}

task cucumber() {
    dependsOn assemble, compileJava
    doLast {
        javaexec {
            main = "io.cucumber.core.cli.Main"
            classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
            args = [
                '--plugin', 'de.monochromata.cucumber.report.PrettyReports:build/reports/cucumber',
                '--glue', 'io.ljn.jp.test.runner',
                'src/main/resources'
            ]
        }
    }
}

Now when the cucumber task is run the report will be in the build/reports/cucumber/cucumber-html-reports folder.

Publishing

Having generated the report we can now plug it in to our Travis CI pipeline so that the results get published to a website.

Before doing this you will need to generate a GitHub personal access token. Travis CI provide instructions on how to do this. Once you’ve generated your token, add it to the environment variables for your build as GITHUB_TOKEN.

With the GitHub access token is set up we need to add a deploy step to .travis.yml:

script:
  - ./gradlew check
  - ./gradlew cucumber || echo "done"
before_deploy:
  - cp build/reports/cucumber/cucumber-html-reports/overview-features.html build/reports/cucumber/cucumber-html-reports/index.html
deploy:
  local_dir: build/reports/cucumber/cucumber-html-reports
  provider: pages
  skip_cleanup: true
  github_token: $GITHUB_TOKEN
  keep_history: true
  on:
    branch: master

There are a couple of things to note in there.

First, ./gradlew cucumber || echo "done" will avoid breaking the build if the cucumber tests fail. Travis will not run the deploy job if the build fails, adding || echo "done" will force an exit code of 0. Depending on the context you may or may not want this.

Second, the before_deploy step is copying the overview-features.html file to index.html so the published website has a homepage.

Now if you go to your GitHub repository settings an enable GitHub pages you will see the report published the next time you run a build:

Automating

You can go a step further and set up a Travis Cron to automatically run the build every day to regenerate the report.

Then you’ll have fresh Cucumber reports every day.

This guide will show you how to set up an example Koa API complete with Swagger documentation and validation. No knowledge of Koa or Swagger is assumed.

Setting up an API

Getting started with Swagger and Koa is super easy thanks to couple of packages - swagger2 and swagger2-koa:

npm init
npm install --save koa koa-router koa-bodyparser swagger2 and swagger2-koa

If we add a skeleton Koa API in start.js:

const Koa = require("koa");
const app = new Koa();
const Router = require("koa-router");

const app = new Koa();
const router = new Router();

router.get('/health', (ctx, next) => {
  ctx.body = {
    "status": "UP"
  };
});

app
  .use(router.routes())
  .use(router.allowedMethods())
  .listen(3000);

console.log("API started");

Running node start.js and going to http://localhost:3000/health in a browser should reveal a JSON response saying that the service is up and running.

Adding the Swagger UI

Now that the API is up and running we can start defining it using a Swagger file api.yaml.

The first part of the yaml file is general information about the API:

swagger: "2.0"
info:
  title: "Swagger Example"
  description: >
    This is an example Koa API.
  version: "1.0.0"
host: "localhost:3000"
schemes:
  - "http"

Next we need to add our health endpoint to the same file:

paths:
  /health:
    get:
      summary: "Check the status of the service"
      description: "If the service is up and running correctly the response will be 'UP'"
      consumes:
        - "application/json"
      produces:
        - "application/json"
      responses:
        200:
          description: "Service is up and healthy"
          schema:
            $ref: "#/definitions/Health"
        500:
          $ref: '#/responses/500'

responses:
  500:
    description: Internal Server Error
    schema:
      $ref: '#/definitions/Error'

definitions:
  Health:
    type: "object"
    properties:
      status:
        type: "string"
        enum:
          - "UP"
          - "DOWN"

Structuring Swagger files is a bit of an art form but I’ve found it useful to have re-usable responses for errors such as 500 that is defined in the responses section of the yaml file.

Next we add the swagger UI middleware to our Koa server:

const Koa = require("koa");
const swagger = require("swagger2");
const Router = require("koa-router");
const { ui, validate } = require("swagger2-koa");

const swaggerDocument = swagger.loadDocumentSync("api.yaml");
const app = new Koa();
const router = new Router();

router.get('/health', (ctx, next) => {
  ctx.body = {
    "status": "UP"
  };
});

app
  .use(ui(swaggerDocument, "/swagger"))
  .use(router.routes())
  .use(router.allowedMethods())
  .listen(3000);

console.log("API started");

Now when we boot up the application we can go to http://localhost:3000/swagger and view the documentation. It should look something like:

Not only is this use clean and easy to use, it also has an inbuilt HTTP client to test each endpoint. Click Try it out and then Execute and it will display the response and response headers in the browser:

Adding validation

To set up the API validation we can add a new endpoint /login that will take a POST request containing a username and password, and return a token if the login is successful.

  /login:
    post:
      consumes:
        - "application/json"
      produces:
        - "application/json"
      parameters:
        - in: "body"
          name: "body"
          required: true
          schema:
            $ref: "#/definitions/LoginRequest"
      responses:
        201:
          description: "successful operation"
          schema:
            $ref: "#/definitions/LoginResponse"
        400:
          $ref: '#/responses/400'
        401:
          $ref: '#/responses/401'

  # ... other responses

  LoginRequest:
    type: "object"
    required:
      - username
      - password
    properties:
      username:
        type: "string"
        example: "username@email.com"
      password:
        type: "string"
        example: "secret"

  LoginResponse:
    type: "object"
    properties:
      data:
        type: "object"
        properties:
          token:
            type: "string"
            example: "dfjghjk34h5kjhdfkjdfgklj76l8k"
      links:
        type: "object"

The full api.yaml can be seen on GitHub.

Next, we need to add the endpoint and validation middleware. As it’s a POST request we’ll also need the bodyParser middleware:

 router.post('/login', (ctx, next) => {
   if (ctx.request.body.username === "admin" && ctx.request.body.password === "password") {
     ctx.body = { "data": { token: "atoken" } };
     ctx.status = 201;
   }
   else {
     ctx.body = { "data": { error: "invalid login" } };
     ctx.status = 401;
   }
 });

 app
   .use(bodyParser())
   .use(ui(swaggerDocument, "/swagger"))
   .use(validate(swaggerDocument))
   .use(router.routes())
   .use(router.allowedMethods())
   .listen(3000);

Now restarting the app and going to http://localhost:3000/swagger will show the new endpoint. Hitting Try it out will bring up a form and let us start experimenting with different requests.

By default the example values from the yaml file will be populated:

{
  "username": "username@email.com",
  "password": "secret"
}

Sending that request should return a 401:

{
  "data": {
    "error": "invalid login"
  }
}

Setting the username to admin and the password to password should return a 201 with a token in the response:

{
  "data": {
    "token": "atoken"
  }
}

And finally, sending an invalid request by removing one of the fields will return a 400 with an error from the Swagger validator:

{
  "code": "SWAGGER_REQUEST_VALIDATION_FAILED",
  "errors": [
    {
      "actual": {
        "username": "admin"
      },
      "expected": {
        "schema": {
          "type": "object",
          "required": [
            "username",
            "password"
          ],
          "properties": {
            "username": {
              "type": "string",
              "example": "username@email.com"
            },
            "password": {
              "type": "string",
              "example": "secret"
            }
          }
        }
      },
      "error": "data.password is required",
      "where": "body"
    }
  ]
}

It’s quite a helpful, if verbose, response. It tells the client that they have omitted the password field and gives an extract from the documentation.

That’s it! It doesn’t necessarily mean you don’t need to provide your own layer of input validation but it certainly covers the most basic cases: missing fields, invalid types etc.

Further reading

A full code example of this project can be found on GitHub.

Swagger files can become very verbose and there is a lot you can do with them. I recommend using the Swagger Editor if only for the very complete example file it contains.

The code for the demo is available on GitHub. Each step of the process in a branch step1 to step7 and the final result is in the master branch. The build is available on Travis but the deployment key has been disabled as it is visible during the demo.

In this post we’re going to set up a continuous integration and continuous deployment pipeline for a node.js app using Travis and PM2.

Continuous integration

Travis is a widely used continuous integration service that is simple to use, and most importantly, free for open source projects.

Configuration

The first step is to add a .travis.yml configuration file to your repository. A basic example for a node.js app would be:

language: node_js
node_js:
  - 12
install: npm install
scripts:
  - npm test

This configuration pretty straight forward, Travis will check out the code from GitHub, run npm install and then npm test. If both these commands exit normally then the build will be a success.

It’s also possible to test against multiple versions of node.js by adding them to the node_js field:

node_js:
  - 12
  - 10
  - 8

Add the .travis.yml to the root folder of your project and push it to GitHub.

Account

Now that the configuration has been added to your repository it’s time to set up the project in Travis. Go to travis-ci.org and create an account if you don’t already have one. When you sign up it will ask you to authorize the travis app so that it can access your repositories.

Press the plus sign to add a repository:

If your repository is not listed then trigger a scan of your account and it should appear.

Then you can toggle the build to on:

Now when you push changes to master or create PRs Travis will automatically run tests for you. You also get to add a trendy shield to your projects README.md:

[![Travis](https://img.shields.io/travis/<your-org>/<your-repo>.svg?style=flat-square)](https://travis-ci.org/<your-org>/<your-repo>)

Continuous deployment

Now that the continuous integration is set up we’re going to modify the .travis.yml configuration to deploy the changes to a server after a successful build, but first we need to create a deployment key so Travis can access the server.

Create a deployment key

You can create a key with this command:

ssh-keygen -t rsa -b 4096 -C "DeployKey"

It will prompt you for a passphrase but you can leave it blank. For the purpose of this example I will assume the key is called deploy but it can be whatever you choose.

Add the key to your server(s):

scp deploy.pub <your-user>@<your-server-ip>:/home/<your-user>/.ssh
ssh <your-user>@<your-server-ip>
cd .ssh
cat deploy.pub >> authorized_keys

You can test the key by logging out of your server and then trying to connect again with:

ssh -i deploy <your-user>@<your-server-ip>

Now we need to give that key read access to the repository. Go to GitHub -> your repository -> settings -> deploy keys and add the contents of the deploy.pub file:

There is no need to add write access.

Add the deploy key to Travis

It’s important not to commit the deploy key to your repository as it will give anyone with that key access to your servers. Luckily, Travis has a mechanism to encrypt the file so that it can use it as part of the continuous deployment pipeline.

To encrypt the key we’ll need to install the travis CLI tool. Note that you will also need to have Ruby installed on your system.

sudo gem install travis -v 1.8.10 --no-rdoc --no-ri
travis encrypt-file ./deploy --add

If you inspect your .travis.yml file you’ll see that some new lines that look similar to:

before_install:
- openssl aes-256-cbc -K $encrypted_ad8ca1f2b447_key -iv $encrypted_ad8ca1f2b447_iv
  -in deploy.enc -out deploy -d

We can test that key by adding an after_success block that will log in to the server and run a command:

after_success:
- eval "$(ssh-agent -s)"
- chmod 600 ./deploy
- echo -e "Host <your-server-ip>\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
- ssh-add ./deploy
- test $TRAVIS_BRANCH = "master" && ssh -i ./deploy <your-username>@<your-server-ip> pwd

Setting up PM2

Now that Travis has access to the server it’s possible to write a deployment script that will pull the latest code, but restarting the service is a bit more difficult. In order to manage this we can use PM2, a process manager for node.js.

To get started with PM2 we create an ecosystem.config.js file in the root of the project:

module.exports = {
    apps : [{
        name: "Your app",
        script: "npm start",
        env: {
            NODE_ENV: "development",
        },
        env_production: {
            NODE_ENV: "production",
        }
    }],
    deploy : {
        production : {
            "ssh_options": "StrictHostKeyChecking=no",
            "key": "deploy",
            "user": "<your-username>",
            "host": ["<your-server-ip>"],
            "ref": "origin/master",
            "repo": "git@github.com:<your-org/your-repo>.git",
            "path": "/home/<your-user>/<project-name>",
            "post-deploy": "npm install && pm2 startOrRestart ecosystem.config.js --env production",
            "env"  : {
                "NODE_ENV": "production"
            }
        }
    }
};

For the purpose of this example I will assume there is already a npm start script that will start the node.js application but you can put any command in the script field.

We can test the configuration by installing PM2 locally and starting the service:

sudo npm install -g pm2
pm2 startOrRestart ecosystem.config.js

Node.js, git and PM2 need to be installed on the server:

ssh <your-user>@<your-server-ip>

# assuming a Debian/Ubuntu server
sudo apt install git nodejs
sudo npm install -g pm2

The final step is to modify the deployment pipeline so that it pulls the latest code and restarts the app:

after_success:
- eval "$(ssh-agent -s)"
- chmod 600 ./deploy
- echo -e "Host <your-ip>\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
- ssh-add ./deploy
- npm install pm2 -g
- pm2 deploy ecosystem.config.js production update

Now when you trigger a build the latest code will be deployed to your server and the application will be restarted.

That’s it

Your app is now set up for continuous integration and continuous deployment without using a plethora of tools, and best of all, it doesn’t cost a thing.

There is a lot more you can do with Travis, such as reporting code coverage and running functional tests. If you’re are interested you can see a more complete project that has codecov integration and cucumber tests:

https://www.github.com/ecorewards/eco-rewards-hub

The original paper suggests using Dijkstra Algorithm to reconstitute journeys from transfer patterns but does not go in to much detail, this post explores a couple of implementations of a journey planner based on transfer patterns.

A simple approach

There are a number of ways to store transfer patterns, but for the initial approach it’s best to use a simple list of stops where the passenger changes to another service. For example, the journey A→E might contain the following patterns:

A,B,C,E
A,B,D,E
A,C,E
A,C,D,E

In order to turn these transfer patterns into journeys, process them individually and turn them into a list of origin and destination pairs representing one leg of the journey. For the transfer pattern A,B,C,E this would be:

[A,B],[B,C],[C,E]

Then look up all the trips operating between each pair of stops in the pattern. The relevant part of these trips can be cut into a journey leg that runs between the origin and destination:

A→B
d10:00,a10:30
d11:00,a11:30
d12:00,a12:30

Creating an index of trips

Scanning all the trips in a dataset to extract legs is expensive so it’s best build up an index of trips that pick up and drop off at particular stations when the data set is loaded.

This example code iterates every stop of every journey to create the index:

const legIndex = {};

for (const trip of trips) {
  for (let i = 0; i < trip.stopTimes.length - 1; i++) {
    if (trip.stopTimes[i].pickUp) {
      const origin = trip.stopTimes[i].stop;

      for (let j = i + 1; j < trip.stopTimes.length; j++) {
        if (trip.stopTimes[j].dropOff) {
          const destination = trip.stopTimes[j].stop;

          legIndex[origin] = legIndex[origin] || {};
          legIndex[origin][destination] = legIndex[origin][destination] || [];
          legIndex[origin][destination].push(trip.toLeg(origin, destination));
        }
      }
    }
  }
}

Completing the journey

Given a list of legs between every pair of stops, it’s possible to progressively scan through those pairs to find a leg that departs the origin station on or after the target departure time.

After each transfer to the next leg the target departure time is updated to reflect the arrival time of the previous leg.

If it is possible to progress to the final leg and find a trip then a complete journey can be made, otherwise the journey is not possible.

const patternStops = [[A,B],[B,C],[C,E]];

function getJourney(legIndex, patternStops, departureTime) {
  const legs = [];

  for (const [origin, destination] of patternStops) {
    const leg = legIndex[origin][destination].find(t => l.departureTime >= departureTime);

    if (!leg) {
      return null; // journey not possible
    }

    legs.push(leg);
  }

  return legs;
}

Cleaning up results

Doing this for a large number of transfer patterns will result in a long list of journeys, many of which will be redundant as they are slower than other journeys. These can be removed by using a filter that removes slower journeys unless they have fewer legs.

Range queries

One of the benefits of this approach is that it is well suited to range queries, where a number of results are required. By extracting all the trips available between the first leg it’s possible to construct a journey for each departure.

As each subsequent journey will only ever arrive at the same time or later, it’s possible to optimize the scanning by maintaining an index of the last leg found for each origin / destination and searching from that point on. This index only applies within the context of a single transfer pattern, other transfer patterns need to maintain their own index.

Tree compaction

It’s quite common for transfer patterns to contain a lot of duplication. Looking at the original transfer patterns:

A,B,C,E
A,B,D,E
A,C,E
A,C,D,E

The first leg of transfer patterns 1 and 2, and 3 and 4 are the same. By converting the transfer patterns into a directed acyclic graph (a tree) it’s possible to eliminate some of that duplication.

     A
   /   \
  B     C
/   \   | \
C   D   E  D
|   |      |
E   E      E   

The overall process is the same but at each node with multiple children the journey so far is cloned and continued down each path independently.

function getJourneys(patternNode, legs, departureTime) {
  const leg = patternNode.findLeg(departureTime);

  // journey can't be completed
  if (!leg) {
    return [];
  }

  // reached the end of the pattern
  if (patternNode.children.length === 0) {
    return [[...legs, leg]];
  }

  return patternNode.flatMap(n => getJourneys(n, [...legs, leg], leg.arrivalTime));
}

Dijkstra

There is still some duplication in the graph, between C->E and D->E. The natural continuation of the previous approach is to compact all patterns into a single graph and run a modified version of Dijkstra’s shortest path algorithm. In doing this it is no longer possible to track the last returned journey along each edge of the graph as there is no guarantee that edges are scanned in time order. In my testing this meant that Dijkstra’s algorithm was consistently slower than the previous approach.

Another disadvantage of this approach is it becomes harder to return results from multiple transfer patterns. It will always return the journey with the earliest arrival time, regardless of whether it has more changes.

Conclusion

Transfer patterns are still the fastest way to perform real-time queries, even if they do require a large amount of preprocessing. We’ve seen two novel approaches to journey planning that perform well. The first approach is simpler to implement but does not perform as well on routes with many transfer patterns of a similar nature. The second approach is more complex but performs very well in situations where there is a lot of duplication between transfer patterns. This is especially relevant for queries where the destination is a list of stations rather than a single station.

The original paper recommends using Dijkstra’s algorithm and it is possible that certain implementations of it perform better than these approaches but the simple optimizations make these approaches after than a stock Dijkstra’s algorithm.

Open source implementations of the tree compacted approach is available GitHub:

https://github.com/planarnetwork/transfer-pattern-planner