5 minutes read | 915 words by Ruben BerenguelSome links are affiliate links
The video edition
This is a video heavy edition. Over the past “Readings of the week” years I have managed to reduce my pending readings from 392 posts to 4 (right now I have 12). I more or less have been tracking the length of this list weekly:
This was well and good… But I hadn’t tracked my “videos pending to watch”. The pandemic made this grow enormously, even if I kept deleting videos from past conferences which were superseded by new conferences:
It hit more than 100 after the last Spark Summit, and even though I tried to watch more of them, a conference video takes 30 something minutes, even at 1.5x. And a lot of times, I realise it’s not that interesting halfway through. And very often, with the slides and a bit of the spoken context I have enough to see the point and run with it.
Haskell and automation to the rescue: I wrote a small helper utility that converts videos to static webpages with images and captions on the side (first link today). It took me a couple afternoons to do it (around 7 hours-ish?). That big drop at the end of the graph is my glancing over 18 videos, which probably took me 40 minutes total. Watch time would have been around 8 hours, so it has already paid in time saved and will continue to save me time.
Never underestimate what you can automate and how it can give you more time for other stuff.
This is the cause of this being the video edition, and probably one of my most useful “small helper programs”. It takes a Youtube video URL, extracts frames every 30 seconds and creates a static webpage with the frames and Youtube’s automated captions. Makes breezing through technical videos super easy. You can see an example output here.
So, Ibis is a declarative query “frontend” that can use several backends (think Spark, Dask, others). Learn one API, run it in many places. This may sound familiar: it’s basically SQL over different query engines. The problem would then be the quality of the Ibis layer on top of a Spark executor, for instance.