Serving text/plain for curl with Next

I have recently been trying out the Next.js for some projects again. I have had earlier projects where I have been really happy with Next, but for some time now Gatsby has been my go to. One of the big things separating these two are that Next has runtime server side rendering and thought it would be a neat to utilize this to have separate content-types based on the client. The idea is that if the client is a CLI based client like curl the server should respond with a plain text response. This should not be the case if the accept header is set to something specific. The default accept header in curl is */*.

Next has a feature where you can drop in your own http server and still let the framework handle the rendering. This is helpful in many situations like protecting things with login or using Next as a part of an already existing application. Below is a minimal custom server. The highlighted line is where the rendering of the response is handed of to Next.

const { createServer } = require("http");
const { parse } = require("url");
const next = require("next");

const dev = process.env.NODE_ENV !== "production";
const app = next({ dev });
const handle = app.getRequestHandler();
const port = parseInt(process.env.PORT || "3000");

app.prepare().then(() => {
  createServer((req, res) => {
    const parsedUrl = parse(req.url, true);
    handle(req, res, parsedUrl);
  }).listen(port, err => {
    if (err) throw err;
    console.log(`> Ready on http://localhost:${port}`);
  });
});

This custom server does nothing that the default Next server does not, so putting this a project does not make very much sense unless one adds something more. That said, if this is put in server.js in a Next project running node server.js in dev and NODE_ENV=production node server.js in production should work nicely.

To achieve the custom content type responses without duplicating the content we can use Next to render the content. Then we will pass it through a npm module that returns text from HTML. It could also be interesting to do the same with something that returns markdown if you want to keep more of semantic elements. In order to get the html from Next we need to give it a fake response that we can read from afterwords. I will admit, this is a hack, but hey it works. 🤷‍♂️

class FakeResponse extends EventEmitter {
  constructor() {
    super();
    this.headers = {};
    this.data = "";
  }

  getHeader(key) {
    return this.headers[key];
  }

  setHeader(key, value) {
    return (this.headers[key] = value);
  }

  write(data) {
    this.data += data;
  }

  end(data) {
    this.data += data;
    this.emit("end", data);
  }
}

The class above emulates the part of Node's http.ServerResponse that is needed and emits end when the response is done. This makes it possible to get the HTML generated by Next and the headers set by Next. Below is a function that takes the same arguments as Next's handle function, but it uses the fake response to get the generated HTML and pass that to html-to-text. It also copies all the headers from the fake response to the real response. Content-Length and Content-Type is excluded from this copying since those will be different from what Next has set.

function handleWithText(req, res, parsedUrl) {
  const fakeRes = new FakeResponse();
  fakeRes.on("end", data => {
    Object.keys(fakeRes.headers).forEach(key => {
      if (key !== "Content-Length" && key !== "Content-Type") {
        res.setHeader(key, fakeRes.getHeader(key));
      }
    });
    res.setHeader("Content-Type", "text/plain");
    res.statusCode = fakeRes.statusCode || 200
    res.end(htmlToText.fromString(data));
  });
  handle(req, fakeRes, parsedUrl);
}

With that set up, the custom server code can be changed to the following. The code below checks the accept header and uses the new handleWithText function if the accept header is set to text/plain or the user agent is curl or httpie, the two CLI based http clients I know of, with an accept header set to */*.

app.prepare().then(() => {
  createServer((req, res) => {
    const parsedUrl = parse(req.url, true);
    const [accept] = (req.headers["accept"] || "").split(";");

    if (
      accept === "text/plain" ||
      (accept === "*/*" && /curl|HTTPie/.test(req.headers["user-agent"] || ""))
    ) {
      handleWithText(req, res, parsedUrl);
    } else {
      handle(req, res, parsedUrl);
    }
  }).listen(port, err => {
    if (err) throw err;
    console.log(`> Ready on http://localhost:${port}`);
  });
});

This would result in this output if it would be used on the front page of this website (which would require me to rewrite it from Gatsby first, but that is another thing).

$ curl -L rolflekang.com
RL [/]

Hi, I am Rolf

I am an independent software developer who enjoys working
with the whole stack. I occationally write posts [/writing], take photographs
[/photography], ride mountain bikes and snowboards. I do consulting work, if
that is something you are looking I might be able to help out [/hire].

This might not be the most useful feature, but it might be neat if your website contains information that people often would want to get from the terminal. Anyhow, it is a fun little easter egg. 🐣