How not to architect a social network on its way to the proverbial moon

Whether BitClout is a nightmare, a scam, the future of social media, or all three is up for fierce debate on newsfeeds everywhere — but one place there’s little socializing happening is BitClout itself, because the first 24 hours of public availability has brought the site to its knees. Frustrated with the sluggishness, I ctrl-shift-I‘d into my browser’s developer tools and spied a couple of disastrous design choices that is hammering the centralized gateway with unnecessary data lookups.

I have no idea who Davidsun is, but since I didn’t use the $5 they start you with to drive the price of my own coin up, I looked like a good deal.

Now, BitClout’s core functionality is essentially Twitter (and indeed their primary attraction is the 15,000 most popular twitter identities that they have copy-pasted and started selling shares of whether the account owner knows about it or not). You follow people, people follow you, you post, you comment on others’ posts. So to read any timeline, your browser must make a request to pull these posts and comments, plus some metadata and a link to the profile pic. So, peeking into my network traffic, I thought it was bad enough that the user’s profile pic was base64 encoded string 3kB in length — embedding binary data into a JSON response prevents any cache mechanism from taking it off your hands, not to mention that gzipping these bytes has little effect.

ProfilesFound[“0”][“ProfilePic”] was the first large data blob that caught my eye

But that’s not the half of it even one percent of it. Expanding the Posts array of length 8 shows that the exact same 3kB base64 profile pic is a property of each comment, needlessly exploding the size of the response. What’s more, running ctrl-f to find all occurrences of ProfilePic returned 244 results – where are all these hiding? Ah, let’s peek in this 267 Length Array called UsersThatHODL: each of these elements also includes a ProfileEntryResponse object — these are the profile pics of every user who has bought a share of this identity.

Hopefully this would be a quick fix, as user uploaded photos are already pushed to imgur.com and cached — so it was perhaps a simple oversight that profile photos got base64ified and stuck in a database. By performing a search and replace on the base64 strings and replacing the 3,000+ character binary data with 56 bit unique id (14 hex characters) to use as a URI, the zipped size of this payload that occurs on every timeline is minimized by over 90%. At this moment not everyone has 240+ “shareholders” for lack of a better term (cloutchasers?), but I expect they will quickly outgrow their current strategy.

Response Text1,085 KB
Response Zipped534 KB
Response without ProfilePic228 KB
Response w/o ProfilePic Zipped39 KB

I’ll clarify here that I don’t think serving 1MB of text every page load is the bottleneck — but the fact that 1MB of text must be serialized out of the database with each request. Depending on their choice of server (which, by the way, their tech stack is not yet disclosed, they don’t even say what kind of blockchain they use besides Proof of Work), only so many requests may be handled concurrently. In a worst-case-scenario, their server may have to finish up one connection before moving on to the next (as opposed to answering requests in parallel), as is the case with a basic Apache server, but I’m sure they have something more advanced than that, right? 🦗🦗🦗

UPDATE: apparently a ‘minor bug’ brought down the whole network in a recursive reboot of death, as you can see in the rather embarrassing error message below. Reverted transactions? Full refunds? What kind of blockchain are these folks running here?

As for how BitClout got to this point, from what I can tell, this utopia/dystopia of a reputation marketplace was secretly booted into existence in mid Jan (see the blockchain explorer for the null hash), where coins have been pre-mined to distribute to the various Venture Capitalists whose involvement was only announced yesterday. Over that time span, it seems that instead of an invite-only system like Facebook and Clubhouse had utilized, a simple password was passed around allowing friends of friends to register (my only source here is Clubhouse chatter). Whether yesterday’s public launch was scheduled is unclear — but the password was leaked to reddit a few days ago so perhaps the administrators decided the crypto-cat was out of the bag and opened the site for all.

I’ll withhold my ethical and economic concerns until I have some more experience on the platform, but so far the community is welcoming, full of positivity and futurist (if accelerationist) optimism with a bias for boosting small creators: there’s not much to be gained in buying shares of Kanye West; maybe you double your money, maybe it cuts in half. If you’re here for gains, you want to identify the creators that are going to climb their way through exponential growth, from $1 to $10,000. In this way, the platform encourages money-flow from the rich (seeking an inflation-busting asset) to the masses of content-creators currently competing for ad-industry crumbs.

One last point: the screenshots above are errors I received today browsing BitClout. I uploaded them to OpenSea.io intending to mint them as NFT, but the transaction fee to do so is currently 160 USD, not including fees paid by the buyer. Besides, I don’t feel great about burning a month’s worth of electricity to list these screenshots on a marketplace. I think BitClout is going to have a much lower barrier to entry for getting money in your pocket as an artist, as soon as they provide some means of withdrawing funds and trading tokens on the crypto-marketplaces at large — check out their “one-pager” overview for more info on what future the BitCloutians have in mind.

animated mosaic experiments

The reflecting pool honeycomb and frog emoji / kiwi grid are a couple of options I was exploring as I built the web design software I want to see in the world — one that makes it easy to arrange any kind of content on a geometric grid and play around with visual motif in order to make every piece of information that gets published on the web a little more unique and memorable.

As it is, when I’m trying to remember where I saw some piece of information I’ve got nothing to go on. Every tweet looks the same, every facebook post fits same rectangle, the only way to navigate is to scroll up and down, as if computers were limited to moving back and forth on a paper tape, so where was I just looking? was it up? no it was down… maybe it was this other tab…

YouTube seems to be a more visual medium, and you can scroll sideways in places, but the home page is still algorithmically arranged so nothing is in the same place as the last time I saw it, and the only way to search is by text string, so you’re supposed to know the name of the video you want or just channel surf. Recently I had this song stuck in my head and I could not remember the name for my life, so I type, “clowns car crash cross-dressing singer ” because that’s all I remember about it, but nothing familiar turns up and eventually I’m scrolling through my watch history from a year ago when I finally see the thumbnail (Yukon Blond’s Saturday Night is the video, it was worth it)

Anyway what I’m saying is it can be really hard to remember where you last saw something on the internet, even if you already downloaded some pdfs. Windows and Mac both have gone from demanding a well organized harddrive to a “type something into the box and hope the computer can guess what you want”, er, I mean “AI” approach

The alternative I’m suggesting is a web where the content publisher can choose to make big changes to how their page is structured — is your universe 4 sided or 6 sided? does everything face an origin at the center or is it all an infinite scroll with no beginning or end? Either way there’s a checkbox for element orientation, and a drop down menu that let’s you choose from different tiling patterns to build on. While most other design frameworks have gone through pains to get a toggle switch for ‘light mode’ and ‘dark mode’, my software allows for fine tuning light and color interaction on each page, hopefully with the effect of making each page a little more visually distinct, so it’s more recognizable when moving through history and file folders.

The effect of zooming into the pattern to reveal more white space, or zooming out to increase the density of the material, has a very noticeable outcome on the brightness of my laptop’s display. I later learned that controlling light is one of the primary use of many of the geometric patterns I’ve been using as source material — Persian and Indian architecture features these patterns carved in wood and stone and on top of cooling the air flowing through and keeping the sunlight in check, it provides a lot of meaning to a space — how familiar the patterns are, and yet how unique each one can be, such that seeing one you might immediately remember where you were the last time you saw it.

“While from within the jali one can view the happenings outside, those standing on the outside do not get a clear view of the inside. This allows the jali to be a perfect privacy provider, at the same time not cutting off those within with the world outside. 

The next step in developing the software is to get a function working that allows “painting” elements on the grid pixel by pixel, like you might in microsoft paint or minecraft. That is, allow selecting any set of tiles and editing the color (or ‘material’) of the group, or replace the selected tiles with text areas, photos and videos, iframes to other embedded content.

From there, the page is rendered as static HTML+SVG with no javascript so it’s cheap and relatively painless to serve the content from your own machine, or use free services like github-pages to quickly get your audio/visual compositions online with your choice of domain name.

So I’m intending for this to be a way to make visually distinct virtual spaces to keep important information that you don’t want to lose. That information could be family photos or a spreadsheet, either way, why not decorate the margins so that the thumbnail will stand out when you’re trying to find it again.

Creating the SuperHex textile

I’ve been computerizing different patterns I learned in my geometry class this spring, but this design is the first original work I can put my name on. I wanted to illustrate how the complex triangles in the lowest triangle is derived from the “flower of life” in the top triangle.

In the top left, I’ve highlighted the triangles that are inscribed by circles — In the top right I’ve highlighted the triangles between the circles’ midpoints. On the lower left and right, You can see the smaller triangles are rotated 30 degrees from the larger. The bottom triangle is the intersection of both of these patterns. In the video below you can see the Inkscape operations to intersection and arrange these divisions.

The SuperHex tessellates into a visually dense wallpaper. I’m excited to see what it looks like as a fabric so I’ve uploaded it to spoonflower.com. Once I receive the sample I’ll be able to put it up fro sale, first of many designs I hope!

Drawing a Globe with Compass and Straight Edge

I always wanted to make a globe. This is the “AirOcean World Map”, a design by Buckminster Fuller and cartographer Shoji Sadao, which aims to depict earth as “one island in one ocean”, and distributes geographic distortion so no country appears to be much larger than it really is.

I decided I could make my own after I noticed a familiar shape while practicing a simple sacred geometry pattern of tessellated equilateral triangles. Once the first grid of triangles is made, 3 further cuts are made across the grid, from each point of a triangle to the opposite midpoint, dividing each triangle into sixths. That’s when I was reminded of the Dymaxion map, with a unique subdivided triangle on a one of its edges. (You can see a better version of this grid in my blog on SuperHex)

Here’s the time lapse of sketching the coastlines… I felt like a real Slartibartfast if you know what I mean.

Using Color and Geometry to Represent Entropy and Other Big Numbers

A 12 bit cube representing 4096 colors from #000 to #FFF
https://jazzyjackson.github.io/colorspace/12bitcolor.html

As time goes by, I find myself interacting more often with giant numbers encoded in hexadecimal strings that are hard to tell apart. There are git commit hashes, API keys, magnet links, public private key pairs, transaction IDs, and with IPv6 even IP addresses are 64bit hexadecimal strings.

Two useful qualities of using random number generators for unique ids and hash functions:

  • With a large enough bitspace, the resulting hash can be assumed to be universally unique (infinitesimal chance of collisions)
  • the output is uniformly distributed — if the input changes even one bit, the output will be drastically different.

However, while its easy enough to look at a list of git hashes and remember the first few letters while you’re checkout out commits, you’ll forget it as soon as you have to remember the next one. Hexadecimal numbers are unique, but they all look the same.

In the interest of making the distinction between large numbers more visual, I started using geometric tessellations where I saw a similarly infinite range of possibilities. In this case with 5 x 24bit colors (between #000000 and #ffffff), and 8 bits to encode edge thickness, you get a large array of visually distinct patterns you could use as a header image, or background images – but this is just with the same 12-pointed star. There are a very large number of possibilities with 4, 5, and 6 fold symmetries and different choices of when to use space and when to alternate stars…

You can try out randomly generated mosaics here by clicking on the “flip 128 coins” link. That’s one coin for each bit of information encoded by the pattern! The URL is updated each time so you can share a pattern, and you can download the SVG ready to use as a background image.

On Twitter, Medium, and Instagram, everyone’s feed looks the same and I can never remember where I saw some piece of information. Maybe by allowing a geometric motif generator to customize a page, or just hashing the contents of a page into a mosaic, we can provide a more visual signal that something on a page changed since you last looked at it, and a way to jog your memory when you do see the same pattern.

Custom Elements from Scratch

This project came out of writing my own framework to create components that would be opening some file on disk, whether it was Markdown, source code, or media files. The priority was to make it easy to extend components by adding “actions” and “reactions” to a dropdown menu for each component.

In the video, I open a ‘library’ component that knows how to render icons and metadata from the filesystem. Then I open a file in a textarea component, which is the prototype for a component which renders markdown and another that becomes a code editor (with showdown and codemirror libraries).

You can see the menu options added to the codemirror component in the source code below, but in the video note the menu contains all the options that belong to classes it inherited from. If you watch closely you’ll see that each menu selection is two steps, and when an action is selected, it shows the javascript call it’s going to make. e.g. remove from window becomes this.remove() and become turns into this.become(codemirror) and offers a drop down menu. This design feature is inspired by my favorite part of the 3D modeling program Blender, which allows you to mouseover every button and menu option in the entire interface, showing you the python call you could make instead. For instance, File > Save becomes bpy.ops.wm.save_as_mainfile

Exposing the underlying API this way allowed me to learn how to automate my Blender workflows as seen in Producing data-driven holograms with Python/Blender/FBX and I hope any application I write would allow others to automate it just as easily.

Source code for the proto component at the top of the inheritance tree can be found here and implements all the lifecycle callbacks provided by custom elements. For CodeMirror, the actions and reactions just allow updating the key binding to vim and changing the syntax highlighting, but options to download, overwrite, delete, and toggle word wrap are provided by the TextArea component.

Automatic Environment Provisioning with CondaVision

AKA the longest shell script I ever wrote

Conda is an environment manager similar to virtualenv, but it features automatic dependency resolution that has made it a joy for me to use. To get a python environment set up with conda, its only a matter of running something like:

conda --name whateveryouwant python=2.7 a list of modules you need

You can then enter this environment at any time by calling source activate whateveryouwant and then run scripts from there. You can also print a “requirements.txt” file that locks in the dependencies and versions so that the next person running your script can reproduce the exact environment on their system — which would ease a lot of pain in trying to run random python examples off the internet.

So what do I need CondaVision for?

But what if I don’t want to run scripts interactively? I have python scripts that I want to execute as a web API, but they all require different versions of pythons and python packages, so I needed a way to automate creating environments at runtime. Conda Execute aims to solve the same problem, but it introduces a new syntax to declare dependencies as a comment inside python files, and I couldn’t get it to recognize that I needed a python less than 3, so I went ahead and wrote a bash script that does the following:

  1. Scan through files in PYTHONPATH to get the names of modules that might be defined locally
  2. Combine that with a list of modules python includes to get a list of modules I don’t want to ask conda to install (conda create will throw an error if you say you need the sys built in module, for example)
  3. perform regex on the python file I want to execute (and all python scripts in PYTHONPATH) to extract all the modules required
  4. compare the results of regex with the list in step 2 to get a new list of modules I need to ask conda to take care of for me
  5. create a hash representing the combination of modules so I can compare it with environments that were created earlier
  6. if there is no existing conda environment that matches that hash, I create one
  7. then I activate the necessary environment and in that environment, execute the script

Key Value Fetch with Kvetch.js

I really like using the fetch API packaged in evergreen browsers, but I was getting annoyed with having to set the credentials, redirect, and method options all the time, plus it always takes a bit of code to format the querystring correctly, plus it’s annoying to set headers and stringify JSON objects that I want to PUT to my server. So I wrote Kvetch. First argument is URL, which gets passed directly to fetch. Second argument is an optional query object. This can have as many key value pairs as you want, it just gets URIComponent encoded, joined with &s and =s, and appended to the URL after a ‘?’ (so don’t put the ? in yourself). Third argument is the Body a.k.a. Request Payload. It can be an object, a string, an ArrayBuffer (ie Binary data) or FormData.

A lot of people use axios, request, or other libraries on npm, but I didn’t want to add extra features and a bunch of dependencies, I just wanted to prevent repeating myself using the native API.

If you need it to work on browsers without fetch, just bring in some fetch polyfill and define that first, kvetch will use window.fetch just fine.

kvetch.get/post/put/delete/options(URL::string[, QueryObject::Object, Body::*)

You can leave the QueryObject and the Body blank if you don’t need them.

You can pass a falsey argument (null, undefined, etc) as a QueryObject if you only need a Body.

If you give an Object as a Body, it will be JSON stringified and sent with an application/json ContentType. If you send FormData (including files), the body is handed directly to fetch and it figures out what to do. If you pass a string, it will be sent untouched with a ContentType of text/plain. ArrayBuffers get sent with application/octet-stream but I haven’t actually tested this and don’t know if it’s appropriate.

Here’s the source code, you can even copy paste this into your browser console to try it out, or fork the repo here: https://github.com/jazzyjackson/kvetch.js

Python Type Coercion for HTML Forms

The purpose of this script is to provide an interface for data scientists to interact with parameters submitted via an HTML form. By defining a schema at the top of a python script, the named parameters will either be the correct type and validated against regex, or the program will fail at the validation step. The schema can also be returned as JSON so a consumer of an API can understand what is required.

It also provides methods to read and write from mysql or psql, and write results to AWS buckets via Boto 3.

When writing this I was especially enthusiastic about list comprehensions and anonymous functions so this might not be an example of doing things the most “pythonic” way, I am a javascript programmer at heart.

A TODO for this project is to create an endpoint that builds the web form based on the schema of any requested script, since the programmer will have already defined the input type and name attributes as part of the schema.

This was written back when Python 2.7 was cool, so I’ll have to update it to replace ConfigParser and StringIO with the modern equivalents.

And for today’s episode of “I don’t know why it’s not working for you, it works on my system!” I was just testing this for the first time in 2 years and the files it created were empty. Looks like when running python on Windows using the io module, I have to explicitly call valid.file.close() for the file to finish writing. I’ll have to test and update the rest of my example files.

CSS hot fix for github gists on WordPress, OR, Weird flex, but OK

Moving from Medium to self hosting wordpress, I was missing one great feature I learned to love: pasting a github url into my blog and having syntax highlighting, line numbers, and automatic updating ‘just work’ — as opposed to using a javascript library that can perform highlighting from markdown, or just pasting source code in <code> blocks without any syntax highlighting.

Worse than that is having to log into wordpress and edit a blog post to make adjustments to the code — once code has been pasted into a blog, when will you notice that it has a typo?

Keeping code snippets as github gists gives me shareable revision history and an easy interface to grab plain text source code, so I can actually use the code that I’m embedding in my blog, and update it all in one place. (One caveat: neither Medium nor this Plugin render anything without javascript enabled — GitHub.com works like a charm without javascript so it would be nice if they could at least fallback on an iframe inside a noscript, but GitHub doesn’t allow gists to appear in iframes, sending a X-Frame-Options: deny header, so it would have to be server-side.)

I was happy to find the WordPress 5 compatible plug in called Gist Github Shortcode. I’m surprised it only has a few hundred active installations because it works great except for one thing, the styling on line numbers was broken! The line number is a :before pseudo element, and the gutter that is supposed to contain it is table data (<td>) , but the psuedo element got painted outside of the layout “flow” and gets clobbered by the source code which is painted right on top.

After fiddling with the display rules for a while I found that applying display: flex to the table row expanded the gutter to almost fit the number, I then had to apply position: relative; right: 0.5em; to shift the number back to the center.

I added this CSS snippet to the “Additional CSS” form on Dashboard -> Customize Your Site and was good to go, here’s a live embedding of the gist shortcode with the style applied: