Month: March 2020

As time goes by, I find myself interacting more often with giant numbers encoded in hexadecimal strings that are hard to tell apart. There are git commit hashes, API keys, magnet links, public private key pairs, transaction IDs, and with IPv6 even IP addresses are 64bit hexadecimal strings.

Two useful qualities of using random number generators for unique ids and hash functions:

• With a large enough bitspace, the resulting hash can be assumed to be universally unique (infinitesimal chance of collisions)
• the output is uniformly distributed — if the input changes even one bit, the output will be drastically different.

However, while its easy enough to look at a list of git hashes and remember the first few letters while you’re checkout out commits, you’ll forget it as soon as you have to remember the next one. Hexadecimal numbers are unique, but they all look the same.

In the interest of making the distinction between large numbers more visual, I started using geometric tessellations where I saw a similarly infinite range of possibilities. In this case with 5 x 24bit colors (between #000000 and #ffffff), and 8 bits to encode edge thickness, you get a large array of visually distinct patterns you could use as a header image, or background images – but this is just with the same 12-pointed star. There are a very large number of possibilities with 4, 5, and 6 fold symmetries and different choices of when to use space and when to alternate stars…

You can try out randomly generated mosaics here by clicking on the “flip 128 coins” link. That’s one coin for each bit of information encoded by the pattern! The URL is updated each time so you can share a pattern, and you can download the SVG ready to use as a background image.

On Twitter, Medium, and Instagram, everyone’s feed looks the same and I can never remember where I saw some piece of information. Maybe by allowing a geometric motif generator to customize a page, or just hashing the contents of a page into a mosaic, we can provide a more visual signal that something on a page changed since you last looked at it, and a way to jog your memory when you do see the same pattern.

This project came out of writing my own framework to create components that would be opening some file on disk, whether it was Markdown, source code, or media files. The priority was to make it easy to extend components by adding “actions” and “reactions” to a dropdown menu for each component.

In the video, I open a ‘library’ component that knows how to render icons and metadata from the filesystem. Then I open a file in a textarea component, which is the prototype for a component which renders markdown and another that becomes a code editor (with showdown and codemirror libraries).

You can see the menu options added to the codemirror component in the source code below, but in the video note the menu contains all the options that belong to classes it inherited from. If you watch closely you’ll see that each menu selection is two steps, and when an action is selected, it shows the javascript call it’s going to make. e.g. remove from window becomes this.remove() and become turns into this.become(codemirror) and offers a drop down menu. This design feature is inspired by my favorite part of the 3D modeling program Blender, which allows you to mouseover every button and menu option in the entire interface, showing you the python call you could make instead. For instance, File > Save becomes bpy.ops.wm.save_as_mainfile

Exposing the underlying API this way allowed me to learn how to automate my Blender workflows as seen in Producing data-driven holograms with Python/Blender/FBX and I hope any application I write would allow others to automate it just as easily.

Source code for the proto component at the top of the inheritance tree can be found here and implements all the lifecycle callbacks provided by custom elements. For CodeMirror, the actions and reactions just allow updating the key binding to vim and changing the syntax highlighting, but options to download, overwrite, delete, and toggle word wrap are provided by the TextArea component.

AKA the longest shell script I ever wrote

Conda is an environment manager similar to virtualenv, but it features automatic dependency resolution that has made it a joy for me to use. To get a python environment set up with conda, its only a matter of running something like:

conda --name whateveryouwant python=2.7 a list of modules you need

You can then enter this environment at any time by calling source activate whateveryouwant and then run scripts from there. You can also print a “requirements.txt” file that locks in the dependencies and versions so that the next person running your script can reproduce the exact environment on their system — which would ease a lot of pain in trying to run random python examples off the internet.

So what do I need CondaVision for?

But what if I don’t want to run scripts interactively? I have python scripts that I want to execute as a web API, but they all require different versions of pythons and python packages, so I needed a way to automate creating environments at runtime. Conda Execute aims to solve the same problem, but it introduces a new syntax to declare dependencies as a comment inside python files, and I couldn’t get it to recognize that I needed a python less than 3, so I went ahead and wrote a bash script that does the following:

1. Scan through files in PYTHONPATH to get the names of modules that might be defined locally
2. Combine that with a list of modules python includes to get a list of modules I don’t want to ask conda to install (conda create will throw an error if you say you need the sys built in module, for example)
3. perform regex on the python file I want to execute (and all python scripts in PYTHONPATH) to extract all the modules required
4. compare the results of regex with the list in step 2 to get a new list of modules I need to ask conda to take care of for me
5. create a hash representing the combination of modules so I can compare it with environments that were created earlier
6. if there is no existing conda environment that matches that hash, I create one
7. then I activate the necessary environment and in that environment, execute the script

The purpose of this script is to provide an interface for data scientists to interact with parameters submitted via an HTML form. By defining a schema at the top of a python script, the named parameters will either be the correct type and validated against regex, or the program will fail at the validation step. The schema can also be returned as JSON so a consumer of an API can understand what is required.

It also provides methods to read and write from mysql or psql, and write results to AWS buckets via Boto 3.

When writing this I was especially enthusiastic about list comprehensions and anonymous functions so this might not be an example of doing things the most “pythonic” way, I am a javascript programmer at heart.

A TODO for this project is to create an endpoint that builds the web form based on the schema of any requested script, since the programmer will have already defined the input type and name attributes as part of the schema.

This was written back when Python 2.7 was cool, so I’ll have to update it to replace ConfigParser and StringIO with the modern equivalents.

And for today’s episode of “I don’t know why it’s not working for you, it works on my system!” I was just testing this for the first time in 2 years and the files it created were empty. Looks like when running python on Windows using the io module, I have to explicitly call valid.file.close() for the file to finish writing. I’ll have to test and update the rest of my example files.

Moving from Medium to self hosting wordpress, I was missing one great feature I learned to love: pasting a github url into my blog and having syntax highlighting, line numbers, and automatic updating ‘just work’ — as opposed to using a javascript library that can perform highlighting from markdown, or just pasting source code in <code> blocks without any syntax highlighting.

Worse than that is having to log into wordpress and edit a blog post to make adjustments to the code — once code has been pasted into a blog, when will you notice that it has a typo?

Keeping code snippets as github gists gives me shareable revision history and an easy interface to grab plain text source code, so I can actually use the code that I’m embedding in my blog, and update it all in one place. (One caveat: neither Medium nor this Plugin render anything without javascript enabled — GitHub.com works like a charm without javascript so it would be nice if they could at least fallback on an iframe inside a noscript, but GitHub doesn’t allow gists to appear in iframes, sending a X-Frame-Options: deny header, so it would have to be server-side.)

I was happy to find the WordPress 5 compatible plug in called Gist Github Shortcode. I’m surprised it only has a few hundred active installations because it works great except for one thing, the styling on line numbers was broken! The line number is a :before pseudo element, and the gutter that is supposed to contain it is table data (<td>) , but the psuedo element got painted outside of the layout “flow” and gets clobbered by the source code which is painted right on top.

After fiddling with the display rules for a while I found that applying display: flex to the table row expanded the gutter to almost fit the number, I then had to apply position: relative; right: 0.5em; to shift the number back to the center.

I added this CSS snippet to the “Additional CSS” form on Dashboard -> Customize Your Site and was good to go, here’s a live embedding of the gist shortcode with the style applied: