JOD Update: Version 0.9.97*

JOD LogoIn the last year much has changed in the J world.

  1. There are new official J 8.0x builds for all supported platforms.
  2. The QT based IDE JDE has matured and is in widespread use.
  3. The column oriented J database JD is drawing new users to J and enticing J veterans to reconsider how we use databases.
  4. There is a small group of J system builders experimenting with additions, extensions and revisions of core J source code.

In short, there are have been enough changes to revisit and update JOD.

JOD version 0.9.97*1 is the first JOD update in many years that mocks the god of software compatibility. In particular:

  1. The syntax of the jodhelp verb has changed.
  2. The jodsource addon no longer uses a zip file to distribute JOD dump files.
  3. JOD online help will no longer be supported or updated.
  4. Volume size is no longer checked before creating new JOD dictionaries.
  5. There is a new version of jod.pdf.

jodhelp changes (#1, #3 and #5)

jodhelp has always been a kludge. In programmer speak a kludge is some half-baked facility added to a system after more essential features have stabilized. The original versions of jodhelp pointed at my rough notes. It was all the “documentation” I needed! Then others stared using JOD which resulted in an “evolved” online version of my notes. I originally thought that hosting my notes online would simultaneously serve user needs and cut the amount of time I spent maintaining documentation. In retrospect this wasn’t even wrong!

I used Google Documents to host my notes. If you’ve ever wondered why completely free Google Documents hasn’t obliterated expensive Microsoft Word or hoary old excellent \LaTeX I invite you to maintain a set of long-duration-documents with Google Documents. During jodhelp‘s online lifetime the basic internal format of Google Documents changed in a screw-your-old-documents upgrade which forced me to spend days repairing broken hyper-links and reformatting. I was not amused; you still get what you pay for!

I originally choose Google Documents because of its alleged global accessibility. Sadly, Google Documents is now often blocked by corporate and national firewalls. Even when it isn’t blocked it renders like a dog peeing on a fire hydrant. All these problems forced me to rewrite JOD documentation with a completely reliable tool: good old-fashioned \LaTeX. The result of my labors, jod.pdf, is now distributed by the joddocument addon and is easily browsed with jodhelp.

After jod.pdf‘s appearance another irritant surfaced: synchronizing jod.pdf and the online version. I tried using pandoc and markdown to generate both the online and PDF versions from the same source files but jod.pdf is too complex for not-to-fancy portable approaches. I was faced with a choice, lower my jod.pdf standards, or get rid of something I never really liked. I opted to drown a child and abandon online help. I don’t expect a lot of mourners at the funeral.

Using the new version of jodhelp requires installing the addon joddocument and configuring a J PDF reader. It’s also good idea to define a JQT PF key to pop up JOD help with a keystroke. To configure a J PDF reader edit the configuration file:


this file is directly available from the JQT Edit\Configure menu. base.cfg defines a number of operating system dependent utilities. Make changes to the systems you use, save your changes, and restart J. The following example shows my Win64 system settings.

 case. 'Win' do.
   BoxForm=: 1
   Browser=: 'c:/Program Files (x86)/Google/Chrome/Application/chrome.exe'
   Browser_nox=: ''
   EPSReader=: 'c:/program files/ghostgum/gsview/gsview64.exe'
   PDFReader=: 'c:/uap/sumatra/SumatraPDF.exe'
   XDiff=: 'c:/uap/WinMerge-2.14.0-exe/winmergeu.exe'
   Editor=: 'c:/uap/notepad++/notepad++.exe %f'
   Editor_nox=: '' 

I use SumatraPDF to read PDF files on Windows. It’s a fast, lightweight, program that efficiently renders jod.pdf. Good PDF readers are available for all commonly used platforms.

To define JQT PK keys edit the configuration file:2


This file is also directly available from Edit\Configure menu. My JOD specific PF keys are:

 F3;1;Require JOD;require 'general/jod'
 Shift+F3;1;JOD Help;jodhelp 0
 F6;1;Dev Dicts;od cut 'joddev jod utils' [ 3 od ''
 Shift+F6;1;Fit Dev Dicts;od cut 'jodfit joddev jod utils' [ 3 od ''
 Ctrl+Shift+F6;1;Test Dev Dicts;od cut 'jodtest joddev jod utils' [ 3 od ''

Pressing Shift+F3 executes jodhelp 0 which pops up JOD help.

jodsource changes (#2)

The jodsource addon is a collection of JOD dump scripts. Dump scripts are serialized versions of binary JOD dictionaries. When executed they merge objects into the current JOD put dictionary. I use them primarily to move dictionaries around but they have other uses as well. Prior to this version I distributed the three main JOD development dump scripts, joddev, jod, and utils in one compressed zip file to reduce the size of JAL downloads.

The distributed script jodsourcesetup.ijs used the zfiles addon to extract these scripts and rebuild JOD development dictionaries. This worked on 32 bit Windows systems but failed elsewhere. J now runs on 32/64 bit Windows, Mac, Linux, IOS and Android systems. To better support all these variants I eliminated the zfiles dependency and pruned the JOD development dictionaries. The result is a more portable and smaller jodsource addon.

Bye bye volume sizing (#4)

Early versions of JOD ran in the now bygone era of floppy disks. It was possible to create many JOD dictionaries on a single standard 800 kilobyte 3.5 inch floppy. Compared to modern porcine-ware JOD, which many J’ers consider a huge system, is lithe and lean. In floppy days it was important to check if there was enough space on a floppy before creating another huge 48K empty JOD dictionary. This is a bit ridiculous today! If you don’t have 48K free on whatever device you are running you have far more serious problems than not being able to create JOD dictionaries.

Volume sizing code remained in JOD for years until it started giving me problems. Returning the size of very large network volumes can be time-consuming and there are serious portability issues. Every operating system calls different facilities to return volume sizes. Even worse, security settings on corporate networks and cloud architectures sometimes refuse to divulge national secrets like free byte counts.

To eliminate all these headaches this version of JOD no longer checks volume size when the FREESPACE noun is zero. To restore the previous behavior you have to edit the file


and change the line FREESPACE=:0 to whatever byte count you want. Alternatively, you could NGAF3 and just assume you have 48K free on your terabyte size volumes.

Still to come

You may have surmised from JOD’s version number that the system is still not feature complete.  The JOD manual lists a few words that I am planning to implement. I only develop JOD when I need something or I am bored out of my mind at work and need a break. Such intermittent motivators seldom insure project completion but I have found a new reason to finish JOD. To list a book on Goodreads or Amazon you need an ISBN number.  The hardcopy version of the JOD manual is a sort-of-published book. To complete the publishing process I need an ISBN. If I am going to bother with such formalities I might as well complete the system the manual describes. So there you have it a new software development motivator: vanity.

  1. The version number is *‘ed because you are always a point release from done!
  2. userkeys.cfg is only available for J 8.03 systems.
  3. Not Give a F%&k!

Mahin and Carl

On the very memorable date of New Year’s Eve my wife’s mother Mahin died. From now on the dropping ball will remind us of her. Mahin had a long and honorable life. She was loved by children, grandchildren, and in-laws. She was the calm matriarch in the storm of her family. I meet her late in life after marrying her daughter Mali. Mahin was almost eighty then and still living on her own. She liked me and regretted having a son-in-law that she couldn’t talk to.  In the 1980s Mahin followed her children out of the chaos of Khomeini’s revolutionary Iran. She settled in Canada in her late sixties taking on a new country, a foreign language and a new way of life. It would have been a big change for a young person but for someone of Mahin’s age it was almost heroic. She got on, adjusted, learned enough English to function, but not enough to properly talk to son-in-laws, and enjoyed life. She never manifested a trace of bitterness or self-pity. She was a strong woman.


Mahin standing beside a reflecting pool in Iran. This picture was taken long before I met her. My wife was still a school girl in Tehran.


In her later years Mahin suffered from dementia. Her children took turns helping her out. We were the last to look after her. She lived with us for nearly two years. Mahin and I similar tastes for the absurd. We both enjoyed the idiotic television show Wipeout. Being knocked on your ass works equally well in English and Farsi. We’d laugh at people plunging into gigantic vats of goo.  Dementia slowly ate away at Mahin’s mind. Eventually she required twenty-four care and her daughters placed her in a nursing home in Toronto. They agonized over putting her in a home and did what they could to make her life comfortable. Mali stayed with her sister Sedi in Toronto five months last year to look after Mahin. She went to the nursing home every day to feed her, give her baths, do her laundry and talk to her. Dementia claimed Mahin’s English. Nursing home staff could not talk to her or understand her. Mali and her sisters were very fortunate to find a sitter that spoke Farsi. Hiring nursing home sitters is more common than you would think. Many of the residents in Mahin’s home had sitters.

Mahin died surrounded by children, grandchildren, her sitter, and nursing home staff.  Home staff told my wife and her sisters that everyone loved Mahin. I am sure they tell many families something similar but having known Mahin I don’t doubt their sincerity.  Mahin was a sweet dignified lady to her last breath.

A week before Mahin died I learned from my ex-wife that Carl, an old mutual friend, was in a Calgary hospice and not expected to live out the month. I’m not a particularly friendly person. Oh, I’m pleasant enough and can, when motivated, skillfully navigate social milieus. If you work with me you might even think I am your friend. I’m not! I’m reluctant to form deep friendships. Entering my sixties I can count my real friends on my fingers. Carl was the best friend I ever had. For nearly twenty years, from high school, until the birth of my daughter, Carl was a happy presence in my life.

Carl was a happy presence in the lives of everyone that got to know him and could tolerate his manifold eccentricities. He was everyone’s crazy uncle and he relished the role. Since leaving western Canada I moved out of Carl’s orbit. We’d trade the odd letter, email, and in recent years, Facebook posts. On the few times that I passed through Calgary I’d always look him up. We’d fall right back into old days blithering. Hearing that he was dying of the nasty form of prostate cancer was a jolt.

Mali petitioned me to call him right away. “You better call before it’s too late.”

I was reluctant because I don’t know what to say to the dying. I missed saying goodbye to my paternal grandfather and I botched my last conversation with my mother. Christopher Hitchens recounts some of the awkward conversations he had with friends and colleagues while dying of cancer in his last book Mortality. Many avoided the sword overhead while others swung it callously. Hitchens didn’t know what to feel. None of their goodbyes really helped.

I called Carl more for myself than him. Carl took a few moments to recall who I was. He was understandably more depressed than I had ever heard him. He was also vague and struggling to respond. Pain medications dull more than pain. I asked if family and friends were about. We aren’t comfortable with people dying alone which is strange because no matter how many friends and family members surround us we all die alone. The only hint of the old Carl that came out our chat was when he referred to Nazar: a very old friend that we both enjoyed making fun of.  Before hanging up I told Carl I would check in on him later but later never came. He died less than two weeks later.

Care to guess what really happened?

Carl in his card counting days. He actually cashed in retirement savings to play Black-Jack in Vegas. He was a very good friend but not exactly your financial go to guy.


Death is the most serious thing that will ever happen to us. It forecloses on what’s next. We simply cease! There is no childish heaven or burning hell. We are not reincarnated and we don’t see ghosts or talk to the dead. If you think or believe otherwise you are simply wrong. I won’t argue with you. I am tired of your irrational objections, your contemptible myths, and your weakness in the face of oblivion. The human machine wears out and breaks down. We don’t imagine afterlives for our cars so why do we indulge such fantasies for ourselves. As much as I would like to see Mahin, Carl, my mother, or anyone of the hundred billion people who have already died, again I won’t and neither will you!

Ferguson and Dark Matter

For the last month the big story here in St. Louis has been Ferguson. At least that’s what media hucksters have sold as the big story. You will have to excuse me; my interests rarely align with “the news.” I don’t watch broadcast TV, listen to the radio, or pay for newspapers. Despite my media starvation diet I am better informed than many broadcast addicts. What’s my secret?

Everyday I read scores of news stories from many Internet sources. When I gave up broadcasting a few years ago I worried that I might miss something.  Actually the exact opposite has occurred. I usually learn of things long before they are “discussed” in main stream outlets. Contemporary broadcasters remind me of short wave Radio Moscow transmissions in the 1970s. Radio Moscow was an insipid utterly predictable propaganda outlet. Listening to what they said was unnecessary. I could predict with nearly 100% accuracy what their slant on any topic would be. They only surprised me by what they didn’t cover. With propaganda only silence is news. I bet you feel the same about that broadcaster you despise. Is it really necessary to listen to MSNBC, FOX, BBC  or the CBC? Their ideological positions are almost as predictable as cold war era Radio Moscow. If your news is not surprising it’s not news; it’s noise.

Ferguson is a classic case of surprise free news. Everyone agrees that a white police officer shot and killed a black teenager. Such a lovely bare-bones tableau invites creative interpretation and that’s exactly what we got. There are hundreds of predictable Ferguson authorities. You can Google any viewpoint you want. What’s lacking in all the hysteria is what skeptics call reliable data.

To me the officer’s guilt is similar to your favorite type of Dark Matter. The case for Dark Matter is very strong. Something out there is undeniably affecting the rotation and formation of galaxies. Similarly, we have a dead teenager, and a literal smoking gun, in Ferguson. Now pay attention; this is where we separate skeptically informed thought from raving ideological demagoguery. The case for Dark Matter is strong but the case for particular types of Dark Matter is close to nonexistent. Detection experiments are ongoing, inconclusive or outright failures. Yeah, the Dark Matter science is not settled. Similarly, the case for the officer’s quilt is far from ironclad. Perhaps that explains the grand jury’s verdict. If you disagree offer incontrovertible evidence.

Of course missing incontrovertible evidence is never a problem for people who have made up their minds. People believe all sorts nonsense, ghosts, demons, past lives, central banking efficacy, but, as I often say, “belief is a bullshit word; you know or you don’t know.”  If you cannot mount a rigorous case that withstands the harshest and meanest of skeptics please calm down and refrain from looting and arson.

Incoherent Interstellar


Don’t look for plot points in Black Holes!

Christopher Nolan has made some excellent commercially successful films like Inception, The Dark Knight, and Mememto. When word got out that he was working on a serious science fiction film expectations got out of hand. Those of us old enough to remember the first screenings of 2001 thought maybe, just maybe, we might see something comparable to Kubrick’s masterpiece. Well I am sorry to report that Interstellar is no 2001; it’s not even a Blade Runner or Nolan’s own Inception. Interstellar is a giant, moderately entertaining, incoherent mess.

Much has been made of Kip Thorne’s involvement with Interstellar. The Black Hole depicted in Interstellar is based on General Relativity calculations. Apparently the CGI animators uncovered something unexpected in how a spinning Black Hole drags light around it. We are told the Black Hole in Interstellar is the most technically accurate ever seen in the movies. Unfortunately, it’s the only technically accurate part of the whole damn movie.

There is no point going over the boners in Interstellar. They are numerous, annoying, glaring, and embarrassing. If you must torture yourself the Bad Astronomer has catalogued Intersellar’s most egregious violations. Now I know what you’re thinking. John, it’s a freaking sci-fi movie, lighten up! You’re going on like a character on the Big Bang Theory.

My answer to such ankle biters is simple.  Science fiction is as an Art Form.  An art form has two equal components: art and form!  Art, without form, is usually effete garbage, and form, without art, is an income tax return. Greatness only emerges when the two are in perfect balance. The first step in achieving balance is honoring the basic elements of the form.  So what are the basic elements of the serious science fiction form?

I’ve gone over this before but clearly you weren’t paying attention. Serious science fiction differs from fantasy in the way it treats reality. Serious science fiction is allowed a few departures from physical reality. You can assume wormholes connect different parts of the universe and that it’s possible to safely traverse them but that’s it cowboy!  Outside of wormholes it’s physics as usual! This is the science part in science fiction. Great science fiction strictly follows this mandate. Take 2001, the exemplar of how this is done, anything non-obelisk related in 2001, including HAL 9000, is completely and absolutely plausible. The obelisk is the singular departure from reality in 2001.

Interstellar bombs because it often departs from physical reality for the basest of reasons: advancing a clunky nap inducing plot. I cannot abide such transgressions. It’s like watching a prima donna ballerina stop in the middle of Swan Lake, drop her tutu, and take a dump on stage. Now prima donna dumps may be entertaining but they’re not ballet. Similarly, Interstellar has its good parts but it’s not serious science fiction. In my opinion Interstellar is a bigger disappointment than Transcendence and it makes we wonder if anyone in Hollyweird is capable of making serious science fiction these days.

I Voted for Nothing

I have just returned from another biannual exercise in futility: voting in mid-term US elections. Once again my preferred candidate, none of the above, was not on the ballot; so, once again, I held my nose and did my best to sabotage the fetid dreams of the rotting things that were. Any barely sentient ape knows that voting, especially in the corrupt US system, is almost a complete waste of time. The choices we’re presented with have been exquisitely gamed by armies of conniving manipulative hacks all hell-bent on not asking the important questions.

Behold your ballot: the primary process purged principled people leaving a scummy residue of stunted subhuman choices. Most are ignorant delusional leftists, grasping smarmy right-wing car salesmen, or outright apolitical narcissist psychopaths. I wouldn’t cross the street to pee on any of them. Where are assassins when you need them?

As bad as voting is the alternative, despotic rule, is still worse so I always haul my cynically enlightened ass to the polls and do what I can to erect roadblocks. I always vote for divided government by carefully selecting bitter opposites. I will vote for Democrats, Republicans, Libertarians, Independents, even Communists and Greens provided I can pit them against someone they detest, abhor, loathe, hate, and want dead on the other side. I want to minimize agreement and maximize conflict because conflict always results in doing nothing, and nine times out of ten, doing nothing is vastly superior to political something. So, until we get some qualified candidates – don’t make me laugh it hurts; it’s critical to vote in people who want to smash in their opponents heads.

The original republican design of the US government recognized the fundamental importance of doing nothing and divided government into three coequal branches to prevent the political class from doing things just for the hell of it!  This basic design has degenerated into an out-of-control executive, a cowardly self-serving congress, a swamped navel gazing judiciary, and an ever-expanding underachieving overpaid bureaucracy that fills in the details on massive template laws like Obamacare. It doesn’t take much of a constitutional scholar to see that this is not what the originators intended. Today we endure a massive, bloated, expensive, inept, corrupt and stupid government run by simpletons, parasites, and criminals.

Fortunately for us, government isn’t nearly as important as political assholes of all stripes think it is! As long as ordinary citizens can buy off and safely ignore government they will tolerate its existence. So keep doing nothing boys and girls; it will keep your heads off the bloody spikes you probably deserve.


Photospheres Away!

Photographers are notorious gear-heads. Everybody has a favorite lens, camera or tripod and, no matter how much gear we have, more is always better! Hey, somebody has to stimulate the moribund Obama economy. Gear lust is not entirely irrational. Good cameras really do take “technically” better pictures than mediocre or poor cameras but here’s the infuriating thing. Technical merit does not make the image!

Image quality depends of many things and changes over time. If you doubt this imagine you have an old locket that frames a Daguerreotype of your great-great-grandmother. I’d bet that every single portrait on your shiny new iPhone is “technically” superior to that old Daguerreotype but I’d also bet you’d chuck your iPhone before giving up that old Daguerreotype.  An image’s quality goes far beyond MTF curves and pixel counts.  Photography and geometry are both bereft of royal roads.

In the early days of photography gear was a serious constraint. Getting a good picture was a technical and artistic struggle. Imagine shooting a panorama using large glass plates covered with a home-brewed ASA 0.5, (you read that right ASA 0.5), blue light-sensitive emulsion. Despite such limitations early photographers managed to create some great images. Imagination and gumption have always been the most important photographic tools with good lenses as a distant third.  Well, the 150 year reign of the lens has ended, software has displaced the lens as the primary modern photographic tool and Google’s Photosphere cell phone application neatly demonstrates this technological shift.

A photosphere is a panorama on steroids. It’s a complete 360 degree look around image. The Google photosphere app derives from Google Maps street view. Street views are shot with special multi-lens cameras that look everywhere at once but some bright spark in Google realized that you could get roughly the same result from a single lens if the photographer was willing to endure a vertigo inducing dance. Shooting a photosphere takes at least three twirling 360 degree passes. You have to shoot the ground, horizon and sky.  It takes about twenty frames to build a photosphere.

There is nothing new about multi-frame panoramas. Photographers started shooting multi-frame panoramas shortly after the camera’s invention. I shot them when I was teenager. Panorama software isn’t new either; it’s been around for decades. Two “new” developments make photospheres possible: photosphere viewers and cameras (cell phones) that are more software than camera.


I shot this panorama in the 1960’s. I rotated in my grandfather’s driveway shooting an entire roll of film with my Instamatic camera. Many years later I scanned the prints and used panorama software to stitch the images together. Click on the image for more panoramas.

The sphere cannot be mapped onto the plane without distortion. This mathematical fact limits how wide-angle your wide-angle shots can be before they are unnaturally distorted. A flattened 360 degree view of common rectilinear subjects looks wonderfully, or horribly, weird: straight lines become curves and areas lose proportionality. Many natural vistas can tolerate such torments but average street views cannot.  Photosphere viewers fix this problem by simulating how we look around.  Every time you browse a Google street view you are running a photosphere viewer.

Shooting panoramas and photospheres is like any other type of photography. It takes lots of practice! It was hard to “practice” shooting such images before cell phones could run stitching and viewing software because you couldn’t see what you had until you took your twenty frames back to the “lab” and tediously put them together.  Ten years ago panorama software required a lot of manual intervention. I spent hours putting three or four frames together. I didn’t put a twenty frame panorama together until I snapped my first iPhone Photosphere.

The iPhone lens is a pretty crappy short focal length lens. Any decent camera lens easily outclasses it yet I cannot shoot photospheres with my expensive Nikon’s while my cruddy little cell phone can.  What’s the difference?  Software!

iPhone Google photospheres are fun but they’re flaw ridden. You can easily see stitching errors, blending artifacts, ghost people, and other blemishes. Now that we can shoot photospheres the race is on to shoot quality photospheres! Software will dominate but hardware has to catch up to make this happen.  Before long you will be able to buy a special multi-lens photosphere ball camera that you can literally throw into the air. This ought to fix the viewpoint problem for people with good pitching arms and the rest of us can drop the little sucker from a drone. Photospheres away!

Panono Ball Camera

The Panono ball camera is a small multi-lens camera that shoots 360 photospheres by simultaneously capturing images from all its lenses and stitching the result together. It is designed to be thrown into the air. Photosphere baseball is going to be huge!

Pandora’s Star: a Grand Sprawling Entertainment

pandoras-starIn my fevered youth I was an avid fan of science fiction but as I crossed the Rubicon of middle age I read less and less of the genre. For years I preferred nonfiction: mostly science with a smattering of history and biography. Then, about five years ago, I started reading science fiction again.

What kept me away? Most of the authors of my youth had died: Heinlein, Clarke, Asimov, Bradbury, Anderson, Herbert and Dick – all gone! I had to find new – to me – authors. I knew and loved Neal Stephenson, the author of Cryptonomicon, Diamond Age, Snow Crash, and Anathem, but after four or five thousand pages of Neal it was time to move on. My first post Stephenson, new to me, author was Iain M. Banks.

Banks specialized in what’s often called alien infested space opera. His universes are overflowing with life. Aliens are everywhere, inhabiting niches that most biologists would poo-poo as impossible. I prefer more empty and serene universes but Banks’ books like The Algebraist, Surface Detail, and Consider Phelbas whet my appetite for his crowded milieus. I was looking forward to following Banks for years but it wasn’t to be. Iain M. Banks died of cancer, at the ridiculously premature age of 59, leaving fans all over the word wanting. There is no greater outrage than mortality!

After Banks’ death I looked around for other operatic authors; it didn’t take me long to find Peter F. Hamilton and Pandora’s Star. Pandora’s Star is a huge, highly entertaining, example of what I call restrained science fiction.

Restraint is what separates science fiction from fantasy. Fantasy tolerates an anything goes mishmash of logical inconsistencies. Literature has a term for this: Deus ex machina. Modern fantasy is a veritable high-tech Deux ex machina factory churning out beta-male vampires that take implausible romantic interests in their food, prepubescent wizards jerking off in boarding school, (Oh it happened), fireproof maximum babes with pet dragons, and armies of oxymoronic brain-dead brain eating zombies. Only scripture piles on more logical nonsense than fantasy.

I enjoy fantasy as much as the next nerd but it’s not science fiction. Proper restrained science fiction admits a small number of “magic suppositions” but otherwise rigorously adheres to what we know about physical reality. You need some damn science in your science fiction people. The universe of Pandora’s Star presumes a few impossibilities; it assumes wormholes and faster than light (FTL) travel. FTL is a standard plot enabling device. Civilizations spanning thousands of light years simply cannot exist, on human time scales, without it. Pandora’s Star makes three more “impossible” assumptions which I will not divulge because ruining good books should be a capital crime. Aside from these allowed departures from reality the universe of Pandora’s Star sticks to scientific bricks and unfolds with lovely consistency.

Most science fiction writers make impossible assumptions but great ones take them in unexpected directions. Consider wormholes. Wormholes have been a staple of science fiction forever. Three, not entirely restrained, TV series had contemporary soldiers marching through them every week for years. They’ve popped up in every two-bit tale that needed quick point A to B plumbing. Wormholes are a cliché and their presence often signals unimaginative hackery. If you’re going to confront me with wormholes you better damn well show me something new or I’m outta your lame book. The opening chapter of Pandora’s Star is one of the most humorous and imaginative use of wormholes in science fiction. A few pages later Hamilton sends trains through wormholes. It’s Sheldon Cooper’s wet dream: trains in space. I had to smile and keep on reading. Pandora’s Star is a big book, almost one thousand pages, but like all great sprawling books it’s too damn short. Fortunately, there’s a second book, Judas Unchained,  that keeps the story rolling. I haven’t had this much fun with a science fiction since Dune. It’s that good.


There will never be a Literature of Reality

The first job of the myth maker is to shrink the universe. The shrinkage occurs along all dimensions: time, space and mind. You see this everywhere. Immortal vampires and wizards are, at most, a few thousand years old: younger than many real trees. Grand science fiction star empires seldom span a single galaxy in a universe of hundreds of billions. Time travelers never set their machines to zero or infinity; they have no interest in creation or ultimate ends. The demigods, super AI’s, and other transcendent beings brood unproductively, without original ideas, for millennia. Remember when Elrond reminisced about “being there” when Sauron was struck down three thousand Middle Earth years ago. In all that time the great, wise, and nearly immortal, elves of Middle Earth did not move beyond simple swords. How quickly the battle for Gondor would have gone with a few dozen Abrams tanks and a couple squadrons of Warthogs.

It’s hard to fault human authors for shrinking the universe. The real universe, the utterly vast and ancient universe slowly discovered by modern science, is beyond all human scales. Imagine a book that devoted one word to every star in the observable universe. The number of stars in the observable universe is a unit that has a name. It’s called a Sagan in honor of Carl Sagan. The Sagan is currently estimated to be around 7 x 1022.  Assuming there are one thousand words per page our book would have 7 x 1019 pages.

If we read nonstop at a rate of one hundred pages per hour it would take about 80 billion years, or six times the estimated age of the universe, to read this book. Yet, vast as this book is, it leaves out pretty much everything. There’s no mention of the Earth or other planets: so many pages and not even a Hitchhiker’s Guide to the Galaxy’s “mostly harmless.” Reality overwhelms imagination. The magic of Harry Potter is piffle compared to cosmic inflation. Haldane was right. The universe is probably queerer than we can imagine and certainly deeper than we can grasp. Our meager fictions must float in serene little bubbles lest they overrun our short impatient lives. There will never be a literature of reality.

Parsing the Bitcoin Genesis Block with J

The genesis block is the first block on the Bitcoin blockchain. Satoshi Nakamoto, the mysterious entity that created Bitcoin, mined the genesis block on January 3, 2009. It’s been five years since the genesis block’s birth and Satoshi is still unknown, Bitcoin is bigger than ever, and the blockchain is longer than 300,000 blocks and growing.

One of the most important features of the blockchain is its immutability. After the Bitcoin network accepts a block and adds it to the blockchain it can never be altered. This makes Bitcoin blocks rare durable binary artifacts. The cryptographic hash algorithms that underpin the Bitcoin protocol enforce block immutability. If someone decides to tinker with a block, say maliciously flip a single bit, the block’s hash will change and the network will reject it. This is what makes it almost impossible to counterfeit Bitcoins. Bitcoins have been lost and stolen but they have never been successfully counterfeited. This sharply contrasts with funny money like the US dollar that is so routinely and brazenly counterfeited that many suspect the US government turns a blind eye.

The exceptional durability of Bitcoin blocks, coupled with the mysterious origins of Bitcoin, makes the genesis block one of the most intriguing and important byte runs in the world. This post was inspired by the now defunct post 285 bytes that changed the world. I would love to give you a link but this post has vanished. A secondary, but excellent reference is John Ratcliff’s How to Parse the Bitcoin BlockChain. I am adapting John’s nomenclature in what follows.

When programmers start exploring Bitcoin they often cut their teeth on parsing the genesis block. If you Google “blockchain parsing” you’ll find examples in dozens of programming languages. The most popular are C, C++, Java, PHP, C#, JavaScript, and the rest of the mainstream suspects. What you will not find, until now, are J examples.

So what does J bring to the table that makes yet another genesis block parser worth a look? Let’s take a look at Bitcoin addresses. The following is the Bitcoin address of this blog’s tip jar. Feel free to send as many Satoshis and full Bitcoins as you like to this address.

   tip=. '17MfYvFqSyeZcy7nKMbFrStFmmvaJ143fA'

There is nothing deep or mysterious about this funny string of letters; it’s just a plain old number in Bitcoin base 58 clothing. So, what is this number in standard format? Here’s how it’s calculated with J.

   BASE58=. '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz'
   dfb58=. 58x #. BASE58 i. ]
   dfb58 tip

The second line that defines dfb58, (decimal from base 58), is the complete J program! That’s it folks. You can troll the internet for days looking at base 58 to big integer converters and it’s unlikely you will find a shorter or more elegant conversion program. Not only is the J version short and sweet it’s also fast and versatile. Suppose you wanted to convert ten thousand Bitcoin addresses. The following converts ten thousand copies of tip.

   dfb58 10000 # ,: tip
 1709618896654985460726422911112500711652231559804656492485 17096188966549854607264...

At this point fanboys of mainstream programming languages typically pipe up with something like, “changing number encodings is inherently trivial; what about something more demanding like going the other way, say converting Bitcoin public keys to the base 58 address format?”

The public key in the genesis block is encoded in what many call the “challenge script.” Here is the genesis block’s challenge script in hex.

41 04 67 8A FD B0 FE 55 48 27 19 67 F1 A6 71 30 B7 10 5C D6 
A8 28 E0 39 09 A6 79 62 E0 EA 1F 61 DE B6 49 F6 BC 3F 4C EF 
38 C4 F3 55 04 E5 1E C1 12 DE 5C 38 4D F7 BA 0B 8D 57 8A 4C 
70 2B 6B F1 1D 5F AC

Public keys take a number of forms in the blockchain. John Ratcliff’s post summarizes the many forms you will run into. The genesis block uses the 65 byte ECDSA form. Converting this form to base 58 requires taking SHA-256 and RIPEMD-160 hashes. These hashes are available in OpenSSL which is conveniently distributed with J 8.02 JQT. Here’s how to convert the genesis block’s public key to base 58 with J.

   load 'c:/bitjd/scripts/sslhash.ijs'

   Base58frKey65=:3 : 0

   NB.*Base58frKey65 v-- 65 byte public Bitcoin key bytes to base 58.
   NB. monad:  clB58 =. Base58frKey65 clBytes

   ekey=. (0{a.) , sr160 s256 y
   csum=. 4 {. s256 s256 ekey
   Base58Check ekey,csum

   Base58frKey65 }. }: ChallengeScript

The ChallengeScript noun holds the bytes given in hex above. The verbs sr150, s256 and Base58Check are available in the J scripts sslhash and ParseGenesisBlock that I have put in the jacks repository on GitHub.

The following J verb ParseGenesisBlock reads the first full node Bitcoin block file and then extracts and checks the genesis block. ParseGenesisBlock tests the various verbs, (functions), it employs. As a side effect it clearly describes the layout of the genesis block and provides test data for anyone that’s interested.

If this post peeks your curiosity about J a good place to start learning about the language is the recently released New Dictionary of J. You can download a version of J for Windows, Linux, OS/X, IOS, and Android at Jsoftware’s main site.

ParseGenesisBlock=:3 : 0

NB.*ParseGenesisBlock v-- parse and check Bitcoin genesis block.
NB. monad:  clMsg =. ParseGenesisBlock clBlockFile
NB.   file=. 'c:/bitjd/blocks/blk00000.dat'
NB.   ParseGenesisBlock file

NB. fetch genesis block data
dat=. read y

NB. first 4 bytes are "sort of" block delimiters
MagicID=: (i. offset=. 4) { dat
'MagicID mismatch' assert 'F9BEB4D9' -: ,hfd a. i. MagicID

NB. next 4 bytes gives following block length
offset=. offset + 4 [ BlockLength=: _2 ic (offset + i. 4) { dat
'BlockLength mismatch' assert 285 = BlockLength

NB. next 4 bytes block format version - has changed
offset=. offset + 4 [ VersionNumber=: _2 ic (offset + i. 4) { dat

NB. next 32 bytes is previous block hash - genesis block
NB. has no previous hash and all bytes are set to 0
offset=. offset + 32 [ PreviousBlockHash=: (offset + i. 32) { dat
'PreviousBlockHash mismatch' assert (32#0) -: a. i. PreviousBlockHash

NB. next 32 bytes is the Merkle tree root hash
offset=. offset + 32 [ MerkleRoot=: (offset + i. 32) { dat
grh=. '3BA3EDFD7A7B12B27AC72C3E67768F617FC81BC3888A51323A9FB8AA4B1E5E4A'
'MerkleRoot mismatch' assert grh -: ,hfd a. i. MerkleRoot

NB. next 4 bytes is a unix epoch timestamp - rolls over 7th feb 2106
NB. there is no timezone information - it is interpreted as utc
offset=. offset + 4 [ TimeStamp=: _2 ic (offset + i. 4) { dat
'TimeStamp mismatch' assert 2009 1 3 18 15 5 -: ,tsfrunixsecs TimeStamp

NB. next 4 bytes represents block target difficulty
offset=. offset + 4 [ TargetDifficulty=: _2 ic (offset + i. 4) { dat
'TargetDifficulty mismatch' assert 486604799 = TargetDifficulty

NB. next 4 bytes is a random number nonce
offset=. offset + 4 [ Nonce=: (offset + i. 4) { dat
'Nonce mismatch' assert '1DAC2B7C' -: ,hfd a. i. Nonce

NB. next 1 to 9 bytes is the transaction count stored as a variable length integer
NB. see:
offset=. offset + vlen [ 'vlen TransactionCount'=: vint (offset + i. 9) { dat
'TransactionCount mismatch' assert TransactionCount = 1  NB. (*)=. vlen

NB. next 4 bytes transaction version number
offset=. offset + 4 [ TransactionVersionNumber=: _2 ic (offset + i.4) { dat
'TransactionVersionNumber mismatch' assert 1 = TransactionVersionNumber

NB. next 1 to 9 bytes is the number of transaction inputs
offset=. offset + vlen [ 'vlen TransactionInputNumber'=: vint (offset + i. 9) { dat

NB. next 32 bytes is the hash of the input transaction
offset=. offset + 32 [ TransactionHash=: (offset + i. 32) { dat
'TransactionHash mismatch' assert (32#0) -: a. i. TransactionHash

NB. next 4 bytes is the input transaction index
offset=. offset + 4 [ TransactionIndex=: _2 ic (offset + i. 4) { dat
'TransactionIndex mismatch' assert _1 = TransactionIndex

NB. input script length is next
offset=. offset + vlen [ 'vlen InputScriptLength'=: vint (offset + i. 9) { dat
'InputScriptLength mismatch' assert 77 = InputScriptLength

NB. script data
offset=. offset + InputScriptLength [ InputScript=: (offset + i. InputScriptLength) { dat

NB. sequence number 4 bytes
offset=. offset + 4 [ SequenceNumber=: ,hfd a. i. (offset + i. 4) { dat
'SequenceNumber mismatch' assert 'FFFFFFFF' -: SequenceNumber

NB. output count 1 to 9 bytes
offset=. offset + vlen [ 'vlen OutputCount'=: vint (offset + i.9) { dat

NB. output value - number of satoshis sent
offset=. offset + 8 [ OutputSatoshis=: (offset + i.8) { dat  NB. 64 bit unsigned integer
'OutputSatoshis mismatch' assert '00F2052A01000000' -: ,hfd a. i. OutputSatoshis
OutputSatoshis=: ]`(_3&ic)@.IF64 OutputSatoshis

NB. challenge script length
offset=. offset + vlen [ 'vlen ChallengeScriptLength'=: vint (offset + i.9) { dat
'ChallengeScriptLength mismatch' assert 67 = ChallengeScriptLength

NB. challenge script - contains elliptic curve signatures
offset=. offset + ChallengeScriptLength [ ChallengeScript=: (offset + i. ChallengeScriptLength) { dat
'ChallengeScript mismatch' assert GenesisBlockChallengeScript -: ,hfd a. i. ChallengeScript

NB. challenge script is 67 bytes drop first and last byte to
NB. compute the familiar Bitcoin base 58 address - compare with block explorer
OutputAddress=: Base58frKey65 }. }: ChallengeScript
'Genesis Block address mismatch' assert GenesisBlockOutputAddress -: OutputAddress

NB. last 4 bytes lock time
TransactionLockTime=: (offset + i.4) { dat
'TransactionLockTime mismatch' assert 0 0 0 0 -: a. i. TransactionLockTime

'Genesis Block Parsed and Checked'