SWAG a J/EXCEL/GIT Personal Cash Flow Forecasting Mob

While browsing in a favorite bookstore with my son, I spotted a display of horoscope themed Christmas tree ornaments. The ornaments were glass balls embossed with golden birth signs like Aquarius, Gemini, Cancer, et cetera, and a descriptive phrase that “summed up” the character of people born under a sign. Below my birth sign golden text declared, “Imaginative and Suspicious.”

I said to my son, “I hate it when astrological rubbish is right.”

I am imaginative and suspicious; it’s a curse. When it comes to money my “suspicious dial” is permanently set on eleven. I assume everyone is out to cheat and defraud me until there is overwhelming evidence to the contrary. Paranoia is generally crippling but when it comes to cold hard cash it’s a sound retention strategy.

Prompted by an eminent life move, I found myself in need of a cash flow forecasting tool. Normal people deal with forecasting problems by buying standard finance programs or cranking up spreadsheets; imaginative and suspicious people roll their own.

SWAG

SWAG, (Silly Wild Ass Guess), is a hybrid J/EXCEL/GIT mob1 that meets my eccentric needs. I wanted a tool that:

  1. Abstracted away accounting noise.
  2. Was general and flexible.
  3. Used highly portable, durable, and version control friendly inputs and outputs.
  4. Reflected what ordinary people, not tax accountants, actually do with money.
  5. Is open source and unencumbered by parasitic software patents.

Amazingly, my short list of no-brainer requirements eliminates most standard finance programs. Time to code!

SWAG Inputs

The bulk of SWAG is a JOD generated self-contained J script. You can peruse the script here. SWAG inputs and outputs are brain-dead simple TAB delimited text tables. Inputs consist of monthly, null-free, numeric time series tables, scenario tables, and name cross-reference tables. Outputs are simple, null-free, numeric time series tables. Input and output time series tables have identical formats.

A few examples will make this clear. The following is a typical SWAG input and output time series table.

 Date        E0      E1       E2      E3  E4     E5  E6  E7  E8  EC  EF  Etotal   I0       I1  I2  I3  I4  I5  IC  Itotal   R0         R1        R2  R3  Rtotal     D0  D1  D2  D3  D4  Dtotal  BB       NW         U0         U1  U2  U3
 2015-09-01  912.00  1650.00  100.00  0   50.00  0   0   0   0   0   0   2712.00  4800.00  0   0   0   0   0   0   4800.00  130000.00  25000.00  0   0   155000.00  0   0   0   0   0   0       2088.00  157088.00  155000.00  0   0   0 
 2015-10-01  912.00  1656.88  100.00  0   50.00  0   0   0   0   0   0   2718.88  4806.00  0   0   0   0   0   0   4806.00  130054.17  25062.50  0   0   155116.67  0   0   0   0   0   0       2087.13  159291.79  0          0   0   0 
 2015-11-01  912.00  1663.78  100.00  0   50.00  0   0   0   0   0   0   2725.78  4812.01  0   0   0   0   0   0   4812.01  130054.17  25062.50  0   0   155116.67  0   0   0   0   0   0       2086.23  161378.02  0          0   0   0 
 2015-12-01  912.00  1670.71  100.00  0   50.00  0   0   0   0   0   0   2732.71  4818.02  0   0   0   0   0   0   4818.02  130054.17  25062.50  0   0   155116.67  0   0   0   0   0   0       2085.31  163463.33  0          0   0   0 
 2016-01-01  912.00  1677.67  100.00  0   50.00  0   0   0   0   0   0   2739.67  4824.05  0   0   0   0   0   0   4824.05  130054.17  25062.50  0   0   155116.67  0   0   0   0   0   0       2084.37  165547.70  0          0   0   0 
 2016-02-01  912.00  1684.66  100.00  0   50.00  0   0   0   0   0   0   2746.66  4830.08  0   0   0   0   0   0   4830.08  130054.17  25062.50  0   0   155116.67  0   0   0   0   0   0       2083.41  167631.12  0          0   0   0 
 2016-03-01  912.00  1691.68  100.00  0   50.00  0   0   0   0   0   0   2753.68  4836.11  0   0   0   0   0   0   4836.11  130054.17  25062.50  0   0   155116.67  0   0   0   0   0   0       2082.43  169713.55  0          0   0   0 

The first header line is a simple list of names. The first name “Date” heads a column of first of month dates in YYYY-MM-DD format. The SWAG clock has month resolution and dates are the only nonnumeric items. Names beginning with “E” like E0, E1, …, are aggregated expenses. Names beginning with “I” like I0, I1, I2 … are income totals. “R” names are reserves: basically savings, investments, equity and so forth. “D” names are various debts. BB is basic period balance, NW is period net worth and “U” names are utility series. Utility series facilitate calculations. Remaining names are self-explanatory totals. Be aware that this table has been formatted for this blog. Examples of raw input and output tables can be found here.

The next ingredient in the SWAG stew is what many call a scenario. A scenario is a collection of prospective assumptions and actions. In one scenario you buy a Mercedes and assume interest rates remain low. In another, you take the bus and rates explode. When forecasting I evaluate five basic scenarios, grim, pessimistic, expected, optimistic, and exuberant. Being a negative Debbie Downer type I rarely invest time in exuberant scenarios. I concentrate on grim and pessimistic scenarios because once you are mentally prepared for the worst anything better feels like a lottery win.

The following is a typical SWAG scenario table. Scenario tables, like time series tables, are simple TAB delimited text files.

 Name         Scenario On Group       Value  OnDate     OffDate    Method   MethodArguments                                                Description                                                                     
 reservetotal s0          assumptions 0      2015-09-01 2015-10-01 assume   RSavings=. 0.5 [ RInvest=. 3 [ REquity=. 3 [ ROther=. 1        annual nominal percent reserve growth or decline during period                  
 car          s0                      50     2015-09-01 2035-08-01 history                                                                 annualized car maintenance until first death                                    
 house        s0                      912    2015-09-01 2016-08-01 history  BackPeriods=.1                                                 current rent until move                                                         
 insurance    s0                      100    2015-09-01 2035-08-01 history                                                                 car insurance                                                                   
 living       s0                      1650   2015-09-01 2044-01-01 history  YearInflate=.5                                                 normal monthly living expenses                                                  
 salary       s0                      4800   2015-09-01 2016-08-01 history  BackPeriods=.4 [ YearInflate=.1.5                              maintain net monthly income until move                                          
 reservetotal s0                      25000  2015-09-01 2015-10-01 reserve  Initial=.1 [ RInvest=. 1                                       stock value at model start                                                      
 reservetotal s0                      130000 2015-09-01 2015-10-01 reserve  Initial=.1                                                     savings at model start                                                          
 salary       s0                      4200   2016-08-01 2023-07-01 history  BackPeriods=.4 [ YearInflate=.1.5                              reduced net income after move until retirement                                  
 house        s0          move        2000   2016-08-01 2016-09-01 history                                                                 moving expenses                                                                 
 house        s0                      100    2016-08-01 2044-01-01 history                                                                 incidental housing expenses after move                                          
 house        s0                      100    2016-08-01 2044-01-01 history                                                                 home owners association payments                                                
 house        s0                      150    2016-08-01 2044-01-01 history                                                                 property taxes                                                                  
 reservetotal s0          buy house   110000 2016-08-01 2016-09-01 reserve  Initial=.1 [ REquity=. 1                                       down payment added to house equity initial setting prevents double spend        
 loan         s0          buy house   150000 2016-08-01 2023-02-01 borrow   Interest=. 4.5 [ YearTerm=. 30 [ DHouse=.1  [ LoanEquity=.1    30 year mortgage rate on house until inheritance                                
 reservetotal s0          buy house   110000 2016-08-01 2016-09-01 spend                                                                   down payment on house from savings                                              
 annuities    s0                      250    2018-07-01 2035-08-01 history                                                                 monthly retirement and other annuity payments end date is unknown               
 annuities    s0                      50     2018-07-01 2035-08-01 history                                                                 any government pension payments                                                 
 reservetotal s0          assumptions 0      2020-01-01 2044-01-01 assume   RSavings=. -2.5 [ RInvest=. -5.0 [ REquity=. -5.0 [ ROther=. 2 market tanks government introduces negative interest                            
 loan         s0          buy car     10000  2020-07-01 2025-07-01 borrow   Interest=. 5 [ YearTerm=. 5 [ DCar=.1                          pay balance of car at 5% for 5 years                                            
 reservetotal s0          buy car     7000   2020-07-01 2020-08-01 spend                                                                   car down payment from savings                                                   
 reservetotal s0          inherit     180000 2023-01-01 2023-02-01 reserve  Initial=.1                                                     inheritance to savings                                                          
 reservetotal s0          buy house   150000 2023-03-01 2023-04-01 transfer Fee=. 1500 [ DHouse=.1                                         pay off remaining mortgage balance after inheritance fee is closing cost        
 salary       s0                      1400   2023-07-01 2035-08-01 history                                                                 estimated social security payments spread over expected life                    
 insurance    s0                      700    2023-07-01 2024-12-01 history                                                                 medical insurance in the gap between retirement and spouse medicare eligibility 
 annuities    s0                      100    2024-12-01 2044-01-01 history                                                                 any retirement pension payments to spouse                                       
 annuities    s0                      100    2035-08-01 2044-01-01 history                                                                 any us social security survivor benefit after first death                       

Again the first header row is a simple list of names. Most scenario names are self-explanatory but four OnDate, OffDate, Method, and MethodArguments merit some explanation. SWAG series methods assume, history, reserve, transfer, borrow, and spend are modeled on what people typically do with cash.

  1. assume sets expected interest rates and other global assumptions for a given time period. SWAG series methods operate over a well-defined time period. The period is defined by OnDate and OffDate.
  2. history looks at past periods and estimates a numeric value that is projected into the future. Currently, history computes simple means but the underlying code can use arbitrary time series verbs.
  3. reserve manages savings, investments, equity and other cash-like instruments.
  4. borrow borrows money and sets future loan payments. borrow supports simple amortization loans but is also capable of reading an arbitrary payment schedule that can be used for exotic2 loans.
  5. transfer moves money between reserves, debts, expenses and income series.
  6. spend does just what you expect.

SWAG series methods adjust all the series affected by the method. As you might expect SWAG arguments methods are detailed. MethodArguments uses a restricted J syntax to set SWAG arguments. Argument order does not matter but only supported names are allowed. Many examples of SWAG MethodArguments can be found in the EXCEL spreadsheet tp.xlsx. I use EXCEL as a scenario editor. By setting EXCEL filters, you can manage many scenarios.

The final SWAG input is a name cross-reference table. It is another TAB delimited text file that defines SWAG names. You can inspect a typical cross-reference table here.

Running SWAG

To run SWAG you:

  1. Prepare input files.
  2. Start J, any front-end jconsole, JQT or JHS will do, and load the Swag script.
  3. Execute RunTheNumbers.
  4. Open the EXCEL spreadsheet swag.xlsx, click on the data ribbon and then press the “Refresh All” button.

Let’s work through the steps.

Prepare Input Files

By far the most difficult step is the first. Here you review your financial status which means checking bank balances, stock values, loan balances and so on. Depending on your holdings this could take anywhere from minutes to hours. I call this updating actuals. Not only is updating actuals the most difficult and time-consuming step it is also the most valuable. Money that is not closely watched leaks away.

I store my actuals in a simple tabbed spreadsheet. Each tab maintains an image of a text file. I enter my data and then cut and paste the sheets into a text editor where I apply final tweaks and then save the sheets as TAB delimited text files.

Monthly income, expenses and debts are easy to update but some of my holdings do not offer monthly statements. The verb RawReservesFromLast in Swag.ijs fills in missing months with the last known values. When I’m finished preparing input files I’m left with four actual TAB delimited files, RawIncome.txt, RawExpenses.txt, RawReserves.txt, and RawDebts.txt. You can inspect example actual files here.

Start J and load the Swag script.

The SWAG script is relatively self-contained. It can be run in any J session that loads the standard J profile. Load Swag with the standard load utility.

 load 'c:/pd/fd/swag/swag.ijs'

Here SWAG is loaded in JHS.

jhsswag

Execute RunTheNumbers

RunTheNumbers sets the SWAG configuration, loads scenarios, copies actuals to each scenario, and then evaluates each scenario. Scenarios are numbered. I use positive numbers for “production” scenarios and negative numbers for test scenarios. It sounds more complicated that it is. This is all you have to do to execute RunTheNumbers

 RunTheNumbers 0 1 2 3 4 

The code is simple and shows what’s going on.

RunTheNumbers=:3 : 0

NB.*RunTheNumbers v-- compute all scenarios on list (y).
NB.
NB. monad:  blclFiles =. RunTheNumbers ilScenarios
NB.
NB.   RunTheNumbers 0 1 2 3 4

NB. parameters sheet is the last config sheet
ModelConfiguration_Swag_=:MainConfiguration_Swag_
parms=. ".;{:LoadConfig 0
scfx=. ScenarioPrefix

ac=. toHOST fmttd ActualSheet 0
ac write TABSheetPath,'MainActuals',SheetExt

sf=. 0$a:
for_sn. y do.
  ac write TABSheetPath,scfx,(":sn),'Actuals',SheetExt
  sf=. sf , parms Swag sn [ LoadSheets sn
end.

sf 
)

RunTheNumbers writes a pair of TAB delimited forecast and statistics files for each scenario it evaluates.

Open swag.xlsx and press “Refresh All”

The spreadsheet swag.xlsx loads SWAG TAB delimited text files and plots results.3 I plot monthly cash flow, estimated net worth and debt/equity for each scenario. The following is a typical cash flow plot. It estimates mean monthly cash balance over the scenario time range.

meanbalance

The polynomial displayed on the graph is an estimated trend line. Things are looking bleak.

Here’s a typical net worth plot.

networth

In this happy scenario, we die broke and leave a giant bill for the government.4

So far SWAG has met my basic needs and forced me to pay more attention to the proverbial bottom line. As I use the system I will fix bugs, refine rough spots, and add strictly necessary features. Feel free to use or modify SWAG for your own purposes. If you find SWAG useful please leave a note on this blog or follow the SWAG repository on GitHub.


  1. What do you call dis-integrated collections of programs that you use to solve problems? Declaring such dog piles “systems” demeans the word “system” and gives the impression that everything has been planned. This is not how I roll. “Mob” is far more appropriate. It conveys a proper sense of disorder and danger.
  2. When borrowing money you should always plan on paying it all back. Insist on a complete iron clad repayment schedule. If such a schedule cannot be provided run like hell or prepare for the thick end of a baseball bat to be rammed up your financial ass.
  3. It may be necessary to adjust file paths on the EXCEL DATA ribbon to load SWAG TAB delimited text files.
  4. They can see me in Hell to collect.

What is Required: Print Captions on the back of Photographs

I am tired of waiting for “the market” to exploit painfully obvious opportunities so I will now provide guidance, in the form of milliblog entries, that tell our less imaginative entrepreneurs just what some of us would buy if we could.

Here’s a deep request. How about printing captions on the back of photographs?

I just went through the excruciating ordeal of ordering about one hundred prints from SmugMug. SmugMug prints meet my high standards but their online ordering software is clunky and clearly geared toward very small print runs. If you have hundreds of pictures, of varying and custom sizes, the software will punish you. The obvious steps of setting a paper type, selecting all your images, and then letting the software work out the paper size from image aspect ratio is not available. You must go image by image setting one easily derived parameter after another. I would print a lot more if it wasn’t such a frigging chore.

As annoying as SmugMug print-ordering is at least I get the prints I want with one major exception. If you check out my online photographs you’ll see I use captions like nanoblog entries. Many people have told me they really like my captions. One of my pet peeves is unlabeled photographs. I have wonderful hundred-year-old photographs of elegantly posed complete strangers because nobody left a clue on the back.

So, in addition to printing timestamps and file names on the back of photographs, include captions as well!

P.S. It takes about five years for the market to meet my obvious requirements; don’t hold your breath.

Milliblog: Religous and Comic Origin Stories

joseph smith powers up small

Joseph Smith gets his superpowers.

Have you ever noticed that the “origin stories” of religious figures and comic superheroes have a lot in common? Green Lantern is given a powerful ring. Moses is handed magic tablets. Buddha, a wealthy patrician, is horrified by injustice and suffering and decides to fix things: ditto for Batman. A powerful, otherworldly being, comes to Earth to save us. Are we talking about Christ or Superman? The only difference between comic book heroes and religious figures is: comic book origin stories are more plausible. There is a teeny tiny chance that being bitten by a radioactive spider will give you some powers, probably a tolerance for certain arachnid proteins, but there is absolutely no chance that long dead John the Baptist will poof back from the dead to baptize your semi-literate backwoods ass.


  1. A milliblog is a short blog entry that makes a single point and then gets out of the reader’s face.

American Star Chamber creates Foreign Passport Business Opportunity

Good news citizens: the putrid rogue intellects in Washington D.C. have created a brand new shiny Star Chamber. What’s a Star Chamber you ask? It’s basically a secret kangaroo court. The US system of injustice already employs an alphabet soup of Star Chambers. Perhaps you’ve heard of FISA and its outstanding work of rubber stamping the surveillance of dangerous American citizens – oops terrorists. Well, I’m happy to report that the self-aggrandizing assholes that drool us are no longer content with monitoring our terrorist naughty bits; now they’re going for what they covet most: our money.

The new Star Chamber law plugs an egregious hole that citizens – oops terrorists – have mercilessly exploited in the past. You see, when an American – oops terrorist – determines that residing in the US and maintaining American citizenship is no longer worth it, said American could simply gather up his holdings, leave the country, and renounce his citizenship. Remember Eduardo Saverin? He was a cofounder of Facebook who did a little arithmetic and figured out that dumping his US citizenship would save him many millions of tax dollars. Eduardo did what any intelligent human being would do. He analyzed the cost-benefit ratio of his citizenship and made the correct financial decision.

Of course, the entire financial position of the US government is predicated on making sure that citizens do not get in the habit of making such cost-benefit analyzes. In the past, the old tropes of patriotism, honor, giving back, national pride, and so forth kept many of us from firing up our computers but that’s the past. This is the age of hope and change.  And many little Eduardo’s hope to get out of here with some of their change!

The years of hope and change have led to an ever-increasing number of Americans fleeing the country. It’s only a trickle now but wait until Pussydent Hildabeast takes over. The trickle will turn into a flood and that’s a problem because without taxpayers how will Pussydent Hildabeast pay off her cronies, reward her allies, and punish her enemies? It takes a lot of money to bribe – oops run – government.

Fortunately, the new Star Chamber fixes the exile problem. Under the new regime, if the government thinks you owe them back taxes1 they can refuse to issue you a passport. You’re a flight risk you ungrateful terrorist – oops citizen – scum.  It gets better! If the government thinks you’re aiding foreign terrorists they can also refuse to issue you a passport. Of course, what constitutes “aiding foreign terrorists” is completely unspecified. Better not send any attaboy ISIS tweets, they might be construed as “aiding foreign terrorists.”  And, because we are so hope and changey these days if you dare ask why the government thinks you are aiding terrorists this shiny new law allows them to withhold evidence because: terrorists!  Due process is so white privilege! With such an idiotic and oppressive law in place, it’s just a matter of time before the US harbors thousands of internal tax and terrorist aiding exiles. They’ll be joining a larger cohort of hundreds of thousands of former American drug offense prisoners that are also, oddly enough, frequently denied US passports.

This is all good news! What the morons in Washington have created is a gigantic business opportunity for any country that will issue Americans passports for cash.  If you don’t already have another passport contrive to get one as soon as possible!  You really don’t want to be imprisoned here if you run afoul of our increasing fascist authorities.


  1. Of course the IRS never incorrectly assesses a citizen’s – oops terrorist’s – taxes.

Euphoria: Review

euphoriabookLily King’s excellent new novel Euphoria derives from an incident in Margaret Mead’s life. Margaret Mead achieved fame as a young woman with her 1928 book Coming of Age in Samoa. Usually, scholarly works do not attract mass audiences but the good bits of Mead’s book read like soft-core porn and introduced the radical idea that sexual behavior in adolescence may have strong cultural overtones. Nowadays we lump such deep revelations in the “No Shit Sherlock” category!

Some of this is brilliantly alluded to in Euphoria. The strong female character (Nell) had written a popular book that her husband (Fen) envied and peers deprecated. The three main characters, Nell, Fen and Bankson, are social anthropologists doing field research in New Guinea in the 1930s. All three have serious doubts about what they are doing. They obliquely acknowledge the sheer conceit of foreign neophytes descending on an unfamiliar culture and, without speaking the language, being familiar with the environment, or knowing jack shit about the local economy, “decode a people,” in a few short months.

Early social anthropologists liked to cast themselves as “anti-missionaries.” Euphoria echoes this sentiment in a few passages. Anthropologists were there to learn about a culture not obliterate it with Christian sky fairy fantasies. The admirable agnosticism of social anthropologists, you cannot take one myth seriously when you have studied hundreds, is still blunted by an infantile dedication to the absolute primacy of culture. We are not animals but Rousseauian “blank sheets” that our culture scribbles on. Many contemporary social scientists of the left, “Are there any other kind?” bitterly dismiss criticism of this ludicrous axiom as “White Privilege.” The social anthropologists of Mead’s day may have been a bit delusional and naïve, but they didn’t create utter bullshit like Critical Race Theory or, I kid you freaking not, Microaggression Theory.

My only complaint about Euphoria is that it romanticizes a “soft pseudo-science.”  Anthropology has two major branches: physical and social. Physical anthropology deals with things like comparative anatomy, radioisotope dating, geological layering, and DNA; it is very much a real science! Social anthropology is all squishy, personal, and non-verifiable; it is not a real science.  It’s not even, to use Rutherford’s exquisite burn, “stamp collecting.”  Euphoria makes this all clear to scientifically literate readers. In many ways, Euphoria is a better introduction to Mead than Mead herself: recommended.

How Dante Can Save Your Life: Review

dantesavelifeDante’s Commedia may save your life, but I wouldn’t bet on this book doing the same. How Dante can Save Your Life is both interesting, annoying, and ultimately disappointing. If I had stopped in the middle of this book I would have rated it higher. It certainly started out well but, what can only be described as the author’s whining, slowly degraded my view.

The seriously religious do not perceive reliable approximations of reality. They are drifting with their phantoms, looking for things that cannot be rationally demonstrated to exist. Though I admire the discipline and restraint many intelligent religious people exhibit it’s simply impossible to take their cherished beliefs seriously. Those of us that demand verifiable reasons for accepting propositions will never accede to the belief that the purpose of life is to return to God. The author repeatedly returns to this theme as he reads Dante and shares his own life.

The author, Rod Dreher, and his family endured serious grief. The best part of this book is his retelling of his sister’s death from cancer in her forties and her community’s outpouring of love and support. I don’t think the author would disagree that his sister’s death, and the book he wrote about it, greatly contributed to his career as a writer.

It was at this point the author had a crisis that lead to Dante. Cemeteries are for the living not the dead, as is myth. Dante created an extravagant and great myth and like all great classics his epic poem has much to offer readers in any age. The author uses it as a type of self-help book to work through his family problems.

His problems are common. Many of us have seen loved family members die horribly, many of us have suffered crippling injuries, many of us have distressing careers, and many of us have family members that are struggling with themselves and us. Yet some of us are tough enough to see life as a random clash of blameless atoms and that whining will not fix anything.

In Dante’s view, this is the great sin of pride that unchecked leads to Hell. Lucky for us Hell and Heaven are myths. Art, however great, is not reality.

Turning JOD Dump Script Tricks

Have you ever wondered how extremely prolific bloggers do it? How is it possible to crank out thousands of blog entries per year without creating a giant stinking pile of mediocre doo-doo? Like most deep medium mysteries it’s not very deep and there are no mysteries. The spewers, people who post like teenage girls tweet, use two basic strategies:

  1. Multiple authors: The heroic image of the lone blogger waging holy war against a sea of Internet idiocy is largely a myth. Many popular prolific blogs are the work of many hands. The editor at Analyze the Data not the Drivel eschews this tactic. Apparently he’s an incontinent and argumentative prima donna that sane people steer clear of.
  2. Content recycling: In elementary school this was called copying. Now that we’re all grown up we use terms like, “excerpting”, “abstracting”, and my favorite “re-purposing.” The basic idea is simple. Take something you’ve written elsewhere and repackage it as something new. Hey, all the cool kids are doing it!

The following is a slightly edited new appendix I have just added to the JOD manual. I am working to properly publish the JOD manual mostly so I can say that I’ve written a legitimate, albeit strange and queer, book.

I created this post by running the \LaTeX code of the manual appendix through the excellent utility pandoc, tweaking the resulting markdown, and then using pandoc again to generate html for this blog. pandoc is a great “re-purposing” tool!  

Finally, re-purposing is not entirely cynical. The act of moving material from one medium to another exposes problems. I found a few editing errors while creating this post that eluded my \LaTeX eyes. If you find more this is your chance to tell me what a moron I am.

Turning JOD Dump Script Tricks

Dump script generation is my favorite JOD feature. Dump scripts serialize JOD dictionaries; they are mainly used to back up dictionaries and interact with version control systems. However, dump scripts are general J scripts and can do much more! Maintaining a stable of healthy JOD dictionaries is easier if you can turn a few dump script tricks.1

  1. Flattening reference paths: Open JOD dictionaries define a reference path. For example, if you open the following dictionaries:
       NB. open four dictionaries
       od ;:'smugdev smug image utils'
    +-+-----------------------+-------+----+-----+-----+
    |1|opened (ro/ro/ro/ro) ->|smugdev|smug|image|utils|
    +-+-----------------------+-------+----+-----+-----+

    the reference path is /smugdev/smug/image/utils.

    When objects are retrieved each dictionary on the path is searched in reference path order. If there are no compelling reasons to maintain separate dictionaries you can improve JOD retrieval performance and simplify dictionary maintenance by flattening all or part of the path.

    To flatten the reference path do:

       NB. reopen the first three dictionaries on the path
       od ;:'smugdev smug image' [ 3 od ''
    +-+--------------------+-------+----+-----+
    |1|opened (ro/ro/ro) ->|smugdev|smug|image|
    +-+--------------------+-------+----+-----+
    
       NB. dump to a temporary file (df)
       df=: {: showpass make jpath '~jodtemp/smugflat.ijs'
    +-+---------------------------+-----------------------+
    |1|object(s) on path dumped ->|c:/jodtemp/smugflat.ijs|
    +-+---------------------------+-----------------------+
    
       NB. create a new flat dictionary
       newd 'smugflat';jpath '~jodtemp/smugflat' [ 3 od ''
    +-+---------------------+--------+--------------------+
    |1|dictionary created ->|smugflat|c:/jodtemp/smugflat/|
    +-+---------------------+--------+--------------------+
    
       NB. open the flat dictionary and (utils)
       od ;:'smugflat utils'
    +-+-----------------+--------+-----+
    |1|opened (rw/ro) ->|smugflat|utils|
    +-+-----------------+--------+-----+
    
       NB. reload dump script ... output not shown ...  
       0!:0 df

    The collapsed path /smugflat/utils will return the same objects as the longer path. It is important to understand that the collapsed dictionary smugflat does not necessarily contain the same objects found in the three original dictionaries smugdev, smug and image. If objects with the same name exist in the original dictionaries only the first one found will be in the collapsed dictionary.

  2. Merging dictionaries: If two dictionaries contain no overlapping objects it might make sense to merge them. This is easily achieved with dump scripts. To merge two or more dictionaries do:
       NB. open and dump first dictionary
       od 'dict0' [ 3 od ''
    +-+--------------+-----+
    |1|opened (rw) ->|dict0|
    +-+--------------+-----+
       df0=: {: showpass make jpath '~jodtemp/dict0.ijs'
    +-+---------------------------+--------------------+
    |1|object(s) on path dumped ->|c:/jodtemp/dict0.ijs|
    +-+---------------------------+--------------------+
    
       NB. open and dump second dictionary
       od 'dict1' [ 3 od ''
    +-+--------------+-----+
    |1|opened (rw) ->|dict1|
    +-+--------------+-----+
       df1=: {: showpass make jpath '~jodtemp/dict1.ijs'
    +-+---------------------------+--------------------+
    |1|object(s) on path dumped ->|c:/jodtemp/dict1.ijs|
    +-+---------------------------+--------------------+
    
       NB. create new merge dictionary
       newd 'mergedict';jpath '~jodtemp/mergedict' [ 3 od ''
    +-+---------------------+---------+---------------------+
    |1|dictionary created ->|mergedict|c:/jodtemp/mergedict/|
    +-+---------------------+---------+---------------------+
    
       NB. open merge dictionary and run dump scripts
       od 'mergedict'
    +-+--------------+---------+
    |1|opened (rw) ->|mergedict|
    +-+--------------+---------+
    
       NB. reload dump scripts ... output not shown ...  
       0!:0 df0  
       0!:0 df1

    Be careful when merging dictionaries. If there are common objects the last object loaded is the one retained in the merged dictionary.

  3. Updating master file parameters: When a new parameter is added to jodparms.ijs it will not be available in existing dictionaries. With dump scripts you can rebuild existing dictionaries and update parameters. To rebuild a dictionary with new or custom parameters do:
       NB. save current dictionary registrations
       (toHOST ; 1 { 5 od '') write_ajod_ jpath '~temp/jodregister.ijs'
    
       NB. open dictionary requiring parameter update 
       od 'dict0' [ 3 od ''
    +-+--------------+-----+
    |1|opened (rw) ->|dict0|
    +-+--------------+-----+
    
       NB. dump dictionary and close
       df=: {: showpass make jpath '~jodtemp/dict0.ijs'
    +-+---------------------------+--------------------+
    |1|object(s) on path dumped ->|c:/jodtemp/dict0.ijs|
    +-+---------------------------+--------------------+
    
       3 od ''
    +-+---------+-----+
    |1|closed ->|dict0|
    +-+---------+-----+
    
       NB. erase master file and JOD object id file
       ferase jpath '~addons/general/jod/jmaster.ijf'
    1
       ferase jpath '~addons/general/jod/jod.ijn'
    1
    
       NB. recycle JOD - this recreates (jmaster.ijf) and (jod.ijn) 
       NB. using the new dictionary parameters defined in (jodparms.ijs)   
       (jodon , jodoff) 1
    1 1
    
       NB. re-register dictionaries
       load jpath '~temp/jodregister.ijs'
    
       NB. create a new dictionary - it will have the new parameters
       newd 'dict0new';jpath '~jodtemp/dict0new' [ 3 od ''
    +-+---------------------+---------+-------------------+
    |1|dictionary created ->|dict0new|c:/jodtemp/dict0new/|
    +-+---------------------+---------+-------------------+
    
       od 'dict0new'
    +-+--------------+--------+
    |1|opened (rw) ->|dict0new|
    +-+--------------+--------+
    
       NB. reload dump script ... output not shown ...
       0!:0 df  

    Before executing complex dump script procedures back up your JOD dictionary folders and play with dump scripts on test dictionaries. Dump scripts are essential JOD dictionary maintenance tools but like most powerful tools they must be used with care.


  1. Spicing up one’s rhetoric with a double entendre like “turning tricks” may be construed as a microaggression. The point of colored language is to memorably make a point. You are unlikely to forget turning dump script tricks.