Pegasus Mail and Mercury Progress Updates

For much of this year, I have been working on a message validation protocol calld DKIM (DomainKeys Identified Mail) in Mercury. Last weekend, I finally sorted out the last glitch and went live with it on Hera, my primary mail exchanger. Since that time it seems to have worked as expected, although I still have a couple of areas I want to double-check to ensure there are no lingering issues (these are mostly to do with mail sent to mailing lists, which are a long-standing issue with technologies like DMARC, SPF and DKIM).

DKIM has an awful lot of moving parts - RSA key generation, hash generation, digital signing, DNS integration, quite complex message parsing and normalization... But I believe it's worth the effort, if only because it significantly decreases the likelihood of your mail being quarantined as spam by the big mail service providers (assuming you have valid SPF and DMARC records to go with it).

As soon as I'm sure there are no residual issues with mailing lists, I'll be making it available to my test team for a short beta period, then I'll bring it out as a new release, aiming for late in December or early in January.

In another significant development this week, I have finally overcome my distrust and reluctance, and have begun using ChatGPT as a search and code generation tool. I have to say, the results are really pretty astonishing. As an example, I asked it to write a routine that would open specified ports in the Windows Firewall, so I can add that to Mercury, and it did it effortlessly with only a couple of small tweaks required. This is something I've wanted for years, but couldn't work out how to do myself.

Anyone who has known me for a long time knows that I'm not very good at using Google (or DuckDuckGo, which I prefer for privacy reasons) to find information - I tend to get a bit lost and eyes-glazed-over in the result lists and often miss things that would be helpful. For this reason if for no other, ChatGPT is going to have a significant impact on the way I work, all for the better, leading to more efficient and quicker development.

As for WinPMail... Well, if I'm honest, it's been taking a bit of a back seat to Mercury recently as I trudged my way through the complexity of doing DKIM, but that changes from here on in. I've identified an issue to do with the way the program attempts to use the Windows Registry when running on Windows 11, and will be rushing out a patch for that as soon as I can; after that, a number of long overdue UI overhauls will finally get their moment in the spotlight, in particular major reworks of the message reader and composer windows, and the Contact Manager (the replacement for the now utterly ancient Pegasus Mail addressbook). I look forward to offering previews of these new features early in the New Year.

Well, that about covers it for now - watch for another instalment in a month's time.

Cheers!

-- David --

Introduction

This is the first instalment of what is intended to be a regular monthly progress report on the development of Pegasus Mail and Mercury. Each month, on the first day of the month, I will provide a brief summary of what I have been working on in the programs in the preceding period, posting it both on the main https://www.pmail.com web site, and on the community server at https://community.pmail.com.

If there are other pieces of software that have been in continuous existence for nearly thirty-five years, I can't quickly think of one, but Pegasus Mail sent its first ever message at the end of 1989, and Mercury had its first release in 1993. During that time I have tried to be as available to my users as possible, but I won't deny that after thirty-five years I am quite burned-out when it comes to communication, and I'm simply not very good at it any more. It's my hope that these regular progress reports will make up for some of the relative absence you may have sensed from me in recent years, and for which I cannot offer enough in the way of apology.

This progress update will be a little longer than the ones I expect to post in future, because I want to give a slightly broader overview of what's happening behind the scenes, in the hope that it might explain why things seem to be taking so long. So, without further waffle then, let's get into telling you what's been going on.

Background

One of the problems with a big program that is developed over a long period of time is that parts of it can suffer "update starvation" - they work well enough that even though they're not ideal, there are always other things that seem more important to spend time on. As the years pass, the less-than-ideal parts get more code bolted on to them, so they get even less ideal, until eventually the program becomes so difficult to maintain that you have to make a crucial decision: do I spend a lot of time fixing what I have, or do I start from scratch and write a completely new program? A number of years ago I was faced with this decision and decided to overhaul what I already had.

With the benefit of hindsight, I can now see that this may have been the wrong choice.

A lot of the code in WinPMail v4.8 was written in the 1990s, and some of it even dates back to the DOS era. Replacing it with modern code that works better and offers much easier future maintainability has ended up being a quite monumental task. To give you an example, I have torn out vast amounts of general purpose core code from WinPMail and produced modern versions of it, often from scratch, in a library I call BaseLib: BaseLib consists of some 58 separate files totalling well over 100,000 lines of code. To put that in perspective, Leo Tolstoy's "War and Peace" has 71,500 lines - and BaseLib is just one component of a larger ecosystem: there is also OIF (my object library, 24,000 lines), DHUI (my user interface library, 64,000 lines), FS3 (my multi-purpose I/O library, 5,500 lines) and TCPIP (my network and cryptographic library, 23,000 lines).

Of course, all of this is before I even start talking about changing the code that was left behind after I moved to the new libraries. Pegasus Mail is now about 250,000 lines of code, and Mercury is about 125,000 lines, so that's still a lot to bring up to date.

Obviously, there are real benefits to having done all this work - all of the libraries work identically in both Mercury and Pegasus Mail, so I'm now having to maintain much less code than was previously the case. The same is true of one of the most important pieces of code I have ever written, which I call M3 (MailStore version 3), and which is near completion: it replaces the entire mailbox management layer in both programs with one that is identical for both, truly modern and enormously robust. You'll notice it's called "M3": that's because there was also "MailStore" and "MailStore v2" preceding it, both of which I ended up scrapping before they were entirely complete because they did not meet the standards or specifications I expect of myself; I don't think of those abandoned versions of MailStore as *wasted* time, because I learned a great deal from them, but they certainly represent *lost* time, and when you're a solo developer, lost time is time you can never make up.

Had I started the programs from scratch, there is the possibility that I might have been able to use a more modern language than the one I use, or that I could have taken advantage of third-party libraries instead of having to write my own... But I would have had to learn these things, and it's my experience that the things you write while you're learning a new language or environment inevitably get scrapped later once you gain enough familiarity to understand the way you *should* have done them... More lost time.

Things might be even more difficult if I weren't helped by some talented and committed people, like Michael in der Wiesche and Martin Ireland, who handle the HTML rendering in Pegasus Mail; or Sven Henze, who tirelessly maintains the German version of Pegasus Mail; or Han van den Bogaerde, who puts countless hours into supporting people who have problems in the program; or my test team, who help iron out the bugs before they can reach you: without people like this backing me up, I don't believe it would be possible to keep the programs going.

But that's enough background and history though - what are we doing at the moment?

Current focus

The overwhelming focus this year has been on Mercury, in particular adding support for DKIM (a standard for verifying message origins) and ACME (automated installation and renewal of SSL certificates). Mercury's DKIM support is now finished, apart from a little work in one of the backing libraries I mentioned above, FS3; a bit of testing and polishing on it and I expect to have a release with DKIM support before the end of this year. ACME support is a little more difficult, with the "Certbot" program I expected to use to implement it ceasing support at the start of the year. I've found a replacement though - a system called WinACME, which I believe I can integrate quite easily into Mercury: expect to see a release with ACME support early in 2025. Other things I'm working on in Mercury are a completely new domain system which will allow separate user lists for each domain the program supports, IPv6 support, support for OpenSSL v3.x, and support for a wider range of IMAP extensions in MercuryI.

As far as Pegasus Mail goes, it's been a year of refactoring - pulling out more of that old code and replacing it with newer stuff; not very satisfying either for me (because it's hard work) or for you (because it doesn't result in anything that looks very different) - but essential nonetheless. Sven and I have been putting a lot of work into rearranging the program's resources (things like dialogs, graphics and menus) to make them easier to maintain moving forward. I hope to have a full v4.91 release of WinPMail out in December or January.

That's about it for now - watch out for the next progress update on December 1st.

All my best to you,

Cheers!

-- David --

[ Page modified 1 November 2024 | Content © David Harris