The Ticketiser 2: Now in Haskell

This is a project recap on writing some non-super-trivial Haskell for the first time.

After reading Finding Success (and Failure) in Haskell, I decided I needed a non-trivial project to write on my own. I have heard from smart people that Haskell is pretty quick to write once you get your groove, and I was curious to see if I could at least glimpse that.

The project I chose is a rewriting of a CLI tool I wrote in Python close to two years ago, which we jokingly call at work el tiquetizador, the ticketiser. It takes (from stdin) a flat YAML list of ticket details (summary, user story, acceptance criteria and optionally point estimate or description) and calls the Jira API to create them.

It can create epics and any ticket following an epic is added to it. If you use Jira cloud, you can imagine how much time this has saved me during these years.

If you don’t feel like reading this, you can find the code for the Haskell version here.

When I have mentioned this project to people, they have been curious to see it. Jira pain is real. Sadly, it has several hard-coded pieces, and can’t be configured without tinkering with the code. It is tested and typed, of course but it would be more hassle sharing it as it is than… rewriting a better version in Haskell?

It ticks several good properties for a learning project:

1. Scratches an own itch: rewriting this to make it usable to others if possible
2. It is an understood problem with clear boundaries
3. It covers a range of common patterns:
   • Read from files
   • Process command line options
   • Load a configuration file
   • Interacting with an API
   • Encoding and decoding (JSON, YAML)
   • Logging/printing to the terminal

It was then an easy choice, and I want to explain the process I followed.

My development process is pretty straightforward in cases like this. First, I consider the questions this program has:

• How can I test what I write?
• How do I parse YAML?
• How do I serialise and deserialise?
• How do I read from a file?
• How do I process CLI options?
• How do I call an API?
• How do I log to stdout?

Note that this is regardless of language. In a language I’m familiar with, like Python, Go or Scala, I know the answers to these questions, but the questions need to be answered for the program to work.

My initial goal once I have these questions is to

1. Find answers,
2. Build a tracer bullet implementation

You don’t need to exercise all questions to have a viable tracer, you need to have answers to them. In this case, my initial tracer bullet would:

1. From a YAML-like object,
2. Call the API and create a ticket

The catch here is that to get to point 1 here, the best way is to start from YAML and convert it to an object, answering 2 questions. And to get to 2, you need to call an API and serialise/deserialise.

Before writing the tracer, though, I wrote small methods to answer the other questions. Get a file name from the CLI, read it, write to to the screen (which is a form of logging in a way). Once I had YAML parsed, I wrote a small test to confirm it was the expected object.

Days 1 and 2: First steps and PoC

My first Haskell weekend was spent writing this tracer. I started by using the yaml library by Michael Snoyman. In particular, I copied from the config example here. This example is interesting because it serves almost as a test harness about how to go from inlined string to object.

Thus, the first half of the day was spent writing data types for the YAML objects, and tweaking how they looked.

Once I had this down, the other half went for CLI parsing. I went for command-line option parsing, and chose optparse-applicative. This was trickier to write and understand, but I got a CLI argument parser. For now I didn’t have any flags, only a filename.

For the second day, I went for the API calls. This is always tricky, regardless of the language. In the case of Haskell, doubly so: add IO, monads, do notation and the rest of Haskell arcana.

To solve it, I went with whatever had an example of a POST request with JSON payload: http-conduit (also by Snoyman). In particular, the tutorial here is clear enough even I could handle it.

I ended the day with a win, by sending a (static) YAML object to the Jira API. And a ticket was created. Always try to finish work with a win.

By this point I had:

• FromJSON instances for the YAML serialisation (the YAML library uses JSON instances under the hood) of Ticket
• filenameToTickets :: FilePath -> IO [Ticket] From a path to a list of tickets
• Configuration data type to have the custom field mappings for Jira (what is an epic, what is a story, what field is the acceptance criteria, etc)
• ToJSON instances for Ticket to convert into Jira POST payloads
• A main function that could do a POST request, all inside a do
• Scattered “nomain” functions with several examples of parsing CLI arguments I had tried.
• A test for the serialisation FromJSON
  

Day 3: It works!

I started by plugging the pieces: read YAML, process it and send it to the Jira API. This was pretty straightforward:

• Extract the part that was generating the request, creating sendOneTicket :: Ticket -> IO ()
• Plug the CLI parsing to get the filename
• Use filenameToTickets to get `IO [Tickets]
• mapM_ over IO [Tickets] with [sendOneTicket]

You can think of mapM_, from Control.Monad as an equivalent to map, but for lists wrapped inside a monad. In this case, I had a IO [Tickets], the list is [Tickets], the monad IO. Since I have a function from ticket to IO, I can use mapM to go through the list, preserving the IO. The underscore is because I don’t care about the returning results (which were empty at the moment anyway).

The implementation at this point was usable. You could create a file with:

- epic: Foo1
  us: As a foo1, I want to  so that 
  ac: Bar
- story: Ticket1
  us: As a ticket, I want to ticket so that I ticket
  ac: ticket

and it would be processed by sending this as tickets created in Jira.

It had an issue, though. The goal of this tool is to prevent you from having to click anything in Jira as much as possible. Imagine that you create an epic, and this epic contains 15 tickets. As the program was right now, you’d need to open Jira, find all these tickets, bulk update (if you have permissions) and assign them to the epic. If you have ever used bulk update I see you cringing as you read this.

The Python implementation had this as a feature: if a ticket was an epic, it would keep the created epic id and add it as the related epic to all following stories. In the example above, Ticket1 would then belong to epic Foo1. If there were subsequent epics, the epic for tickets after would change to the most recent one.

Solving this in Haskell was stumping me. I was thinking isn’t this just a kind of State I can handle with a State monad? I can barely use a state monad in Scala, the prospect of using it in Haskell was not exciting. If you squint, this looks like folding. If we modify sendOneTicket to be:

sendOneTicket :: Maybe String -> Ticket -> IO (Maybe String)

Then:

• We expect maybe an epic identifier for the ticket
• If the ticket is an epic, we return its identifier
• If the ticket is a normal story, we return the possibly Nothing epic identifier passed to us

And this can be folded over, by using foldM_ from Control.Monad again, the equivalent to fold when inside a monad.

At this point there is more or less feature parity with the Python implementation. Things missing:

• I wanted to make this implementation as configurable as possible. All indicates this will be impossible in complete generality, I will do my best. This is an improvement on the Python implementation
• I broke the tests by modifying several things. I want tests, even if just a couple.
• There are additional flags to pass to add tickets to other boards (this is a feature that prevents it from being as general as I’d like in the first bullet)
• It needs more logging, at the moment it is just printing the URLs to the tickets. This is good enough but a verbose flag and getting the responses in that case can go a long way to troubleshoot setup. I may skip this.

This day was also when I added compiler flags to the build:

• fwarn-unused-imports: I don’t like having more imports than I need
• Werror=incomplete-patterns: I like cases to be exhaustive. How come this is not the default?

Day 4: Refactor, cleanup

This was mostly cleanup day. It involved:

• Add a linter to the build (hlint)
• Auto-format the code (ormolu)
• Change some Strings for newtypes to get more help from the compiler
• Add configuration for credentials and URL instead of hardcoding them
• Try to extract custom Jira settings (fail)

The last point deserves an aside.

All Jira cloud setups share a small subset of properties, or some are at least common enough to be standard:

• There are story tickets and epic tickets
• Tickets have titles
• Tickets have acceptance criteria
• Tickets can have description
• Stories can have point estimates

Even some of these can be contested: there are probably Jira setups without epics. But a system as described above should be good enough for most.

The first problem is that these easy-to-understand fields need to be mapped to the corresponding internal identifiers in Jira. I.e. acceptance criteria might be the JSON/API field customfield_14300.

This is actually tricky to solve as it stands. My design is:

• Tickets are a data type Story|Epic
• They have a FromJSON instance used by the YAML library to decode (hand-rolled)
• They have a ToJSON instance used by Aeson for the API POST (hand-rolled)

The information about the field naming is needed as part of the ToJSON instance. I hardcoded these, and this is good enough for one user locally. There are some ways to solve it in general, the easiest one having a composite TicketWithConfig data type that is created from the deserialised tickets (and more esoteric ideas like constructing a Reader).

The problem comes when there are fields that are required and have special constructs (like team, which is a nested field with id). One possible solution would be to deserialise it as a list of required fields from the config and then iterate, adding them to the tickets.

In the end, I decided that this was a possible area of improvement if anyone outside the company I work for starts using hticketiser. It’s not worth overcomplicating the code for zero benefit. At the moment, if you want to use it, you would need to edit a bit what is in Api.hs to conform to the rules of your Jira setup. The most straightforward about this is (you can find this also in the README of the project):

• Open your Jira path (while logged-in) at /rest/api/2/issue/ISSUE IDENTIFIER
• Check the most important fields
• Stack build and install
• Try to send a new ticket
• Check what the error message says and modify accordingly (in general the errors will be field so and so required or field so and so is not available in this view)

Day 5: Wrap it up and push to Github

This was finish-up. Add a README.md, an example of ticket file and configuration. Send to Github and wrap up the post.

I added additional configuration to select the board (since I create epics/stories in two boards) and the required changes, which imply modifications in the serialisers as detailed above. It was pretty painless thanks to the compiler telling me “nope, nope, nope”.

Git push and done.

Summary

• It took less than 3 days to go from huh to it works.
• The code is not the nicest-looking code I have written, but for a first project I’m reasonably happy with it.
• Somewhat unexpectedly, there is more code in the Haskell version than the Python version. This is due to writing the instances for ToJSON and FromJSON by hand, but is kind of required to make parsing as strict/flexible as I need.
• Haskell code you find in the wild may look unreadable, but once you write something yourself you realise what all those arrows are. And the compiler will be of great help.

If you are wondering: yes, I will write more Haskell code. Not sure for what now though!

Addendum

There were some interesting suggestions for improvement (some I have tried, and some I leave for future projects) in this Reddit thread.

• Python
• Haskell
• Scala