D3 sitemap using a force layout and web workers

June 8, 2020 3 minutes read | 506 words by Ruben Berenguel

A D3 sitemap using force layout and a web worker. This will come to bear-note-graph at some point (and other projects)

I’ll keep this post short. You may have seen my project, bear-note-graph mentioned around here. Its goal is to convert your Bear notes into a graph you can explore (with links across notes and links from tags to notes).

Its core processes the markdown documents from the Bear SQLite database, parsing it with a custom markdown parser I wrote. Then, it uses Graphviz as the graph engine: it’s convenient, but not the prettiest. Also it has some drawbacks in output formats, and the interactivity could be better. As a result, I’ve had a card to improve bear-note-graph to optionally use a D3 force directed layout for some time.

A few weeks ago I started work on this feature, by rewriting a D3 force layout I had written for a half personal, half work project (unpublished): using YAML descriptions to create system diagrams for your (data) architecture. Those diagrams have 30-50 nodes, and as such are pretty fast to render on the browser, live, with draggable nodes. My note database though, has around 1000 notes, with many links. This made waiting for the force layout to converge painful. It took a significant time to render, and dragging nodes was super slow.

I found a fix (as always with D3) in a sketch by Mike Bostock: using a web worker. With this, I could calculate the force layout on the background.

Now, the remaining problem is that when I publish some code (especially code others are going to use), I want it to be as clear and understandable as possible, and I don’t feel like cleaning up this force layout code at the moment. It also requires a significant refactor in bear-note-graph, which I don’t feel like doing now either. On the other hand, I was itching to use it for something… So I whipped a quick Hugo post parser in Python (unpublished, it’s just a bunch of regexes after all) that creates a JSON document that describes the graph.

Filtered and zoomed with search term spark — Filtered and zoomed with search term `spark`

And just like this, I now have a sitemap on this site. It is still in beta: I cleaned up several tags that were repeated, or useless but there is still work to do in that regard. Also some links may not work, if it happens please reach to me on Twitter.

It has a neat autocompleting search (although it has a bug I need to fix) that reduces the amount of nodes to nodes matching the search term. Clicking on any node will take you to that tag or post. It doesn’t have draggable nodes because I haven’t figured out how to share the update ticker code between the worker and the graph… oh well. The code is available for you to see, but don’t expect it to be pretty. If you want something neat, wait until I publish the final version in bear-note-graph.