Adding a Postmark Email Batcher to Ghost CMS

A person from behind typing on a keyboard in front of a brightly lit screen with sunlight beaming in through a window book shelves in the background, pixel art

💡 This post walks through the logic and decision making portion of establishing email batching for Postmark in Ghost. I had originally intended for this project to be contained in a single post, but as I began drafting this piece I found that the most interesting parts of the project were contained in the decisions made as opposed to the code itself.

If you’re eager to take a look at the code, you can find that here. I tried to produce a clear enough README to get you up and running with the code if you’re interested.

Background

As I mentioned in a previous post, this blog (and one that I run for my family) is self-hosted using the Ghost CMS platform. I recognize the full suite of tools and problems this software solves may seem extravagant for someone running a blog that will retain, at most, 30 users but hear me out.

I chose Ghost because it was the only full-featured, nice looking open-source and self-hostable¹ solution in my eyes. I am sure there are a few people reading this that might be thinking “What about [X]?”. Don’t worry, though. I checked that one out and it sucked. This state of affairs isn’t totally surprising. Unlike Ghost, these services often do not have a business operating in the background in addition to offering a complete and stable piece of software. Call me old fashioned, but I expect software I use to work somewhat well even if it’s free. The software also needs to, vain as this may sound, look decent to boot.

The Problem at Hand

Despite integrating with basically any mail provider under the sun for transactional emails, in order to utilize the newsletter sending functionality in Ghost one must have a working Mailgun account. The setup is very simple and this works just fine, if you have or can get a Mailgun account in good standing.

It works less well when you attempt to (twice) start a new Mailgun account, making sure to indicate in clear detail the purpose of said account, and then have your account terminated without cause or explanation (twice). Yes, really².

My first account started out initially as “temporarily disabled” which I can understand. They have a reputation to protect. I was encouraged to reach out to support to have it re-enabled once I clarified my use case. I reached out to support with such and received this notice:

After reviewing your account details and activity, we’ve decided to permanently disable your Mailgun account. Unfortunately, we are unable to fully disclose specific reasons behind this action in order to protect our customers, internal processes, and compliance systems.

Mind you, there had been exactly zero activity on my account and I hadn’t even finished setting it up. I was offered the chance to appeal if I believed the action was taken in error. I did so and I haven’t heard anything back as of two weeks later. Given the opaque support process exhibited above, I wasn’t going to hold my breath.

I again attempted to create a new account without any obfuscated personal details this time and again was met with the same chain of events.

Since I apparently cannot make it past Mailgun’s inscrutable filters in order to pay to use their service, I had used up my one option for newsletter sign-ups on my blogs. Bummer. I also really did not want to request that people interested in reading either of my blogs simply “check in” every once in a while. Again, mine will not be a large audience, so the added friction of checking a website for updates is a non-starter for some people (including some of my family and friends).

What to do?

A Fork in the Road

Well, I’m a software developer, I can make this work! Ghost is open to code contributions, so I’ll add other service providers and then the problem will be solved. That would probably be the Correct Way™ to do it from a design perspective. There’s also a Quick Way™ which is a bit janky from a design perspective, but achieves the same goal much sooner.

When considering approaches to solving a software problem, it’s always useful to think about the trade-offs and balance those against the ultimate goal of the solution.

The Ultimate Goal

Allow users to sign-up for and receive newsletter emails to their inbox whenever I publish a new post without using Mailgun.

The Correct Way

The “Correct Way” entails a lot more work, but the work will represent a much more robust solution in the long run. As opposed to some of the hairiness of the Quick Way, this approach is most appealing to my “pure” software engineering and anal retentive^p brain.

The Ghost code is organized in accordance with solid design practices from my perspective. This is good for maintainability and overall functionality, but, as I’m sure you all know, this represents a larger upfront time expense to complete.

The extra time allottment here is not just in writing better designed code. It also involves researching the design patterns that Ghost follows, understanding things as simple as code organization and styling conventions, and just generally ensuring the code adheres to the same standards of quality.

After all, an outstanding goal for this approach would be to have Ghost actually merge my changes into their codebase going forward. One of the risks of the “Correct Way” is that I would do all of this work and the Ghost team does not or cannot merge the changes into the codebase, leaving me to maintain my own bespoke version which would need to be kept up-to-date with their releases separately.

As I mentioned, Ghost is an actual business with employees and a product roadmap, etc. They already provide their software as FOSS and welcome contributions to the code base³. I do not begrudge them their focus on other feature sets or product enhancements.

This approach just screams “slow, methodical, and deliberate”. If I had to estimate my time investment here, I’d say this approach would take me somewhere in the 40-60 hours range of time investment⁴.

The Quick Way

The “Quick Way” entails less work with a less robust solution and a decent level of jank to get there.

However, as understood by its name, it will be completed much sooner! This is especially pertinent to me as I have a full-time job, a 3-month old baby, and an entire life outside of writing software.

In addition, since I am also writing a separate blog for family and friends to read, the ability to get this functionality up and running sooner rather than later weighs heavy in my deliberation between these two approaches.

The other benefit of this approach is that it is more or less divorced from the Ghost codebase at large. The mechanism by which this approach will be sending emails, though dependent on Ghosts continued implementation of its current feature set, is slightly adaptable outside the context of Ghost.

There is, of course, risk to this approach. The approach is somewhat brittle to its core dependencies within Ghost itself. If the code for this feature set changes even slightly in one of myriad ways, the could (and likely will) break.

This approach is very much the “I’m low on time and I need an MVP now”. The time investment here would be somewhere in the ballpark of 4-7 hours.

Door #2, Mr. Hall

Fortunately, from a User perspective (i.e. people that will sign up for a newsletter) regardless of which approach I take, if I do my job right they’ll be none the wiser. That is, the outcome for them will be the same: They will receive a newsletter in their inbox whenever I publish a new post.

With that outcome in mind and weighing all of the other various trade-offs, I decided to work on the Quick Way first. Though it’s not without risk, it will allow me to get functionality up and running ~10x faster than the Correct Way. I also sincerely intend on attempting the Correct Way at a later time⁵

So without further ado, here’s the plan.

The Plan

The Quick Way entails taking advantage of the Ghost Webhooks functionality, a custom webhooks server, and a third party mail provider (in this case, Postmark).

I’ve had a Postmark account in good standing for the last 18 months, so it was a little disappointing when I was going to be required to create a new mail service account. Fortunately, Mailgun provided some roadblocks, so here we are.

The basic idea is as follows:

Ghost allows users to configure webhooks, including Custom webhooks which can point to an arbitrary endpoint These webhooks are configurable for a variety of events that may occur on the Ghost platform, but for my interest I’m going to use the “Post Published” event When the event fires, Ghost will send a JSON payload to the endpoint I specify My custom webhook server will be waiting to receive the payload, process the payload for relevant data, and also capture some other data from the Ghost database Once the server has packaged the data up nicely, it will use a very basic “Mail Provider API” (via the adapter pattern) to send the email through a third party provider

Dat Jank

There are definitely a couple elephants in the corner that need to be addressed vis-a-vis the design downsides of this approach. I’ll provide an overview of each challenge with a brief description of its utility.

One of the ways this is able to function is by spoofing Mailgun credentials in the Ghost Admin panel.⁶ Suffice it to say this was necessary in order for all of the data to be available for the webhook server.

All posts which are published via this workflow will report as “failed to send” in the Ghost Admin panel I’m fairly certain this can be mitigated via modification of database records, but fixing this behavior wasn’t in scope for this project as it’s not indicative of a true failed event, but an externality of the approach

There is no associated email tracking or analytics of any kind built-in Ghost has extensive analytics tracking since the platform is designed for creators who want the option to build a business on their content. All of this is provided via Mailgun. Since we’ve cut Mailgun out, no tracking for you!⁷

The webhook server needs to have access to the database This is done via the selfsame user that Ghost uses to access the database; that means another access point to the database (not great) and it’s a read/write user with pretty broad permissions (really not great).⁸

Decisions, decisions

Before I started my work, I needed to determine what technologies to use for this project.

I had originally thought of using Python/Flask because setting up the server is dead simple and I love writing Python. Since the rest of Ghost is written in Javascript, I thought it might be interesting to have this project be written in something else before moving back to the “main” language for the Correct implementation.

On the other side of the same coin, should someone who uses Ghost want to use and/or extend my code, I would be making that process a little more difficult by requiring any such developer to know two different languages. I know Python and Javascript are probably neck-and-neck for the two most well known (and used) languages, but still the barrier to entry was real, however small.

With all of that in mind, I decided to use Javascript/Express for this project. Technically, I am using Typescript (because I’m not a masochist). I chose Express because it’s about as simple as one can get when it comes to rolling a Node server. I did consider Fastify (which I’ve used before), but that felt like overkill. The actual implementation of the server endpoint is very straightforward and modularized, so if this project grows some legs and needs to be expanded it will be trivial to rewrite the actual server portion in Fastify to take advantage of all of its nifty goodies.

Onward

This project was really, really enjoyable to work on. I was able to move from idea to conception to functioning solution relatively quickly.

If you made it all the way to the end of the post maybe you should subscribe to my newsletter so you can be notified whenever my next post goes up? It uses the nifty webhook server I’ve been blathering on about, so why not test it out?

Thanks for taking the time to read this. Cheers!

---

Footnotes:

1. The reason I am self-hosting these services is for privacy reasons. Though I cannot afford to right now, I'd love to support Ghost financially in some way in the future and I genuinely encourage you to do so as well! But I'd prefer to not store my data on their servers. ↩️

2. For the sake of full transparency, I used as much deliberately obfuscated profile information as Mail Gun would allow (i.e. My name was John Doe). Remember, I'm a privacy weirdo. However, I did use my real credit card as part of the sign-up process. ↩️

P. Stop referring to yourself or your behavior as "being OCD". You are not. You are anal retentive and we will still love you forever even if you don't tie your proclivities to an officially diagnosible mental/behavioral disorder. Pinky promise. P.S. the 'P' superscript stands for Pet Peeve. Heh. ↩️

3. Seriously, I cannot underscore this enough: the Ghost team owes almost nothing to their self-hosting users. I just want my stance here to be ultra-clear. ↩️

4. I find that people often say a project took them "a weekend to complete" or something similar. As a new developer, I personally always found this a frustrating type of metric because it doesn't tell how much time you actually spent on it and (most importantly) whether I was on-track, ahead, or behind schedule at any given time. Did you spend 6 hours per day, Friday thru Sunday working on this? Was it 6 hours one day and then just some dabbling hours each day after that? Hence, I find the hours metric more useful. ↩️

5. No one has ever forgotten to go back and write their code the proper way, right? RIGHT?!?! ↩️

6. Ghost apparently does not currently have any logic to just disable newsletter sending should the Mailgun credentials fail to work after a certain number of attempts? In my testing of dozens of email sends, the functionality hasn't been disabled, so 🤞. To any Ghost employees or maintainers that might be reading this: please don't add this functionality in yet.🙏 ↩️

7. Each of the various third party mail providers does offer mail tracking and analytics which are (most likely) accessible via their related APIs. However, if I went down the road of replicating tracking and analytics for each provider, I would basically be recreating what Ghost already does via Mailgun and I'd rather save that for a larger project which could be merged into the Ghost codebase instead. ↩️

8. The situation in which I designed and implemented this project utilizes docker-compose. In this setup, Ghost is a separate service from the database service, so the database credentials are already being shared across the internal Docker network. Adding the webhooks server to that network (and thus sharing said credentials) probably isn't that big of a deal. Giving more services pretty broad access to the database here is probably fine, but it makes my teeth itchy. ↩️