Fixpoint

2020-04-28

Build system overhaul for bitcoind

Filed under: Bitcoin, Historia, Software — Jacob Welsh @ 21:19

Background

The Real Bitcoin's build system for some years has consisted at the top level of a number of GNU Makefiles and a thing called "Rotor", building on an earlier "Stator". According to its 2015 introduction by Stanislav Datskovskiy, it served to compile the "bitcoind" executable deterministically and with full static linking, given a reasonable starting environment. The key to this magic was "buildroot", essentially a miniature, non-self-hosting Linux distribution designed for cross-compiling embedded systems.

The determinism came from capturing a full set of dependencies all the way down to the compiler. This came at the cost of adding considerable complexity to the process, as what was formerly a mere application took on all the potential problems involved in bootstrapping an operating system from an unpredictable environment, in addition to the already intricate build systems of the required libraries Boost, Berkeley DB (BDB) and OpenSSL. In an early sign of trouble, Michael Trinque found it wouldn't build BDB on his system without some CPU architecture specific configuration. In my own experience, I got it to work once, but when demonstrating to some friends on fairly similar Gentoo systems I'd built for them, it failed in multiple different ways. Ultimately I couldn't be bothered to track them all down, partly because of how unbearably slow it was: to try any change you would have to repeat the whole toolchain bootstrap.

The V cryptographic source code management system was introduced, with Bitcoin as its first user, shortly after Rotor; somehow the Rotor scripts and patches didn't end up in the V-tree proper, meaning that they in addition to the library and toolchain sources had to be rounded up in order to do offline (i.e. reliable) builds or study the code.

Finally, having already taken on the publication of a Linux distribution with similar static linking and deterministic bootstrapping goals, but going further in providing a self-sufficient system with native compilers, I had little desire to be stuck maintaining two different such beasts.

The vpatch

Thus I now present bitcoin_system_compiler.vpatch (with seals on the shelf). Building on my previous raw transactions patch, it:

  • Rewrites the Makefiles almost entirely. This includes the "makefile.unix" inherited from earlier developers, greatly simplifying it and eliminating historical baggage such as dynamic linking tweaks, Ubuntu bug workarounds, linking of "libssl" and way too much "sed" magic. Additions include compiler warning flags (resulting in quite a bit of warning spew, some of which might be interesting) and building "test_bitcoin" by default. Automatic header dependency analysis is preserved.
  • Removes some minor GNUisms like "tar xvfz" in order to work on BusyBox systems.
  • Brings "openssl-004-musl-termios.patch" formerly found in the external rotor sources into the tree.
  • Adds "boost-no-demangler.patch", discussed below.
  • Removes the various "src/obj" directories and moves all compiler output to the "build" directory. (For instance, this makes it easier to "grep" or "diff" the code without tripping on binary files.)
  • Avoids copying the "bitcoind" binary all around and removes the "bin" subdirectory: one place is enough.
  • Corrects the oversight that a library build failure would be ignored on a second "make" invocation because the mere extracted directories were used as the targets for dependency calculation.
  • Fixes parallel "make" by forcibly serializing the recursion into OpenSSL's ever-so-special custom build system.
  • Avoids recursing into "deps" on a top-level "make clean" so that dependency tarballs won't need to be re-downloaded. (Ultimately these need to get cleaned up and imported directly to the tree.)
  • Tweaks the BDB configuration to prevent "libtool" from attempting to build unwanted shared librarires.
  • Tweaks the Boost "bjam" invocation (the "compression" module gets built at install time if suppressed only at build time) and removes some "|| true" constructs that caused failures to be ignored.

It still supports the "make ONLINE=1" mode to download out-of-tree dependencies into the "deps" subdirectory from deedbot; these are reduced to the three essentials (Boost, BDB, OpenSSL).

In short, it makes both development and deployment much less painful, with a sane starting system as the price of admission.

The undemangling

Special mention is in order for the new boost-no-demangler.patch. Gales Linux uses an older branch of GCC that didn't receive fixes when the long-existing "stack clashing" (archived) family of attacks was stirred up in 2017, meaning some applications could end up vulnerable, especially those using the hazardous yet popular "alloca" or variable-length array features.

As a first step in investigating this, I enabled a number of warnings by default in the GCC configuration relating to excessive or dynamic stack frame size. While these warnings produce many false positives, they've done nicely to illuminate some suspicious code, such as binutils/libiberty/cp-demangle.c. Got that all read? Me neither... but so what, that "libiberty" is just an internal part of the toolchain, or so say the docs, right? And it's "well-known" that you don't want to feed untrusted input to the linker and friends. But wait: the GCC build system copies that code into libstdc++; from there it gets linked into C++ programs. This happens even if the program doesn't use the nonstandard "__cxa_demangle" extension it provides, by way of the default exception handler ("terminate called after throwing an instance of std::whatever").

So my gcc-4.7.4-demangler-amputation.patch, included in the Gales toolchain, removes __cxa_demangle along with the copying of cp-demangle.c, and simplifies the termination function to print raw symbol name, as it previously did anyway for the case of demangler failure (hah! - we learn that they knew their code doesn't always work). These names can be fed to "c++filt" to decode them manually if need be. It then turns out that Boost contains a couple uses of __cxa_demangle - none of them in components actually needed by bitcoind, harrumph - so the Boost patch simply cuts out the assumption that GNU compilers support it.

Stats

$ diffstat bitcoin_system_compiler.vpatch
 .gitignore                          |   21 ----
 Makefile                            |   21 ----
 bin/Makefile                        |   13 --
 bin/Manifest.sha512                 |    1
 build/Makefile                      |   97 +++++++++++++--------
 build/Makefile.rotor                |   56 ------------
 deps/Makefile                       |  166 +-----------------------------------
 deps/Manifest.sha512                |   17 ---
 deps/boost-no-demangler.patch       |   49 ++++++++++
 deps/openssl-004-musl-termios.patch |   46 +++++++++
 manifest                            |    1
 src/makefile.unix                   |  145 -------------------------------
 src/obj-test/.gitignore             |    2
 src/obj/.gitignore                  |    2
 src/obj/nogui/.gitignore            |    2
 src/obj/test/.gitignore             |    2
 verify.mk                           |    5 -
 17 files changed, 168 insertions(+), 478 deletions(-)

Looking only at the "make" code, 471 lines across seven files is reduced to 112 lines across three: quite the improvement I should think!

Future directions

Some work that sorely needs doing, as suggested earlier, is getting those external libraries under control, through some combination of pruning their code to just the necessary parts, replacing their build systems, and importing to the tree, or changing bitcoind code to eliminate them.

2020-04-10

The missing Adacore public download index, vintage 2018, while it lasts

Filed under: Historia, Software — Jacob Welsh @ 21:16

I usually browse the web with JavaScript disabled, if present at all: it's bad for your computer and it's bad for your mind. That it's bad for your computer should be clear if you've dug at all into browser security in the past, say, 15 years. Failing that, consider the intended behavior of the thing: to allow any page to consume unlimited processing and network resources even once loaded. It's as if the mailman, upon completion of a delivery, strolls right into your house and helps himself to whatever is in the fridge. As for why it's bad for your mind, one aspect is that it leads you to believe that a page "works" or contains some piece of information when it in fact does not, like a mirage, or the screensavers of the cathode-ray era, falling away as soon as you reach out to touch it.(i)

Being thus grounded in reality has its difficulties, such as when those around you mistakenly believe a certain link points to something other than a blank page, even expecting you to have an opinion on it. Worse is when you actually want the promised item. On these occasions you can of course choose to let in the hungry mailman; sometimes he'll even give you helpful instructions on how to do so (as if you'd need them, having barred the door in the first place). But sometimes, a little effort can reveal the secrets of the code and allow you to possess the thing for yourself. Some might call it "reverse engineering", but isn't that a strong term for what's really just reading ? It's reading some text that was foisted on you, albeit not in the manner the foisters would like for it to be read.

So it went the last time I was looking into the Ada programming language, in 2018. GNAT, the foremost public implementation, as I understand, was mostly developed as a frontend for the GNU Compiler Collection at US taxpayer expense by a company presently known as Adacore.(ii) But their public download site demanded use of a JavaScript menu tree to filter options by platform and other categories, and offered no readily accessible listing of files or links otherwise.

A survey of the page source didn't directly reveal the hidden list, but turned up a number of scripts by reference. Skipping over some well-known "framework" wads left a promising "script.js", which straightaway pointed to a JSON "feed" containing a sizable collection of file metadata. A search in the code for how it turned this into URLs, and then some custom coding to do the same in a controlled context, was all that remained to produce a usable index. This done, I downloaded a couple GNAT binary releases for x86_64 Linux(iii) and added them to my archives.

Sadly, at the time I had neither a blog on which to boast of my exploits nor the kind of social engagement to motivate it. A year and change went by and I rectified that, but I didn't substantially revisit the Ada bootstrapping process until now, and had forgotten all about the indexing work.

Through looking into the publications of the now disbanded Republic and asking around, I found a series of recipes, notes, more notes, and a partial collection of dependencies, but little assurance that I'd obtained all necessary ingredients. In particular, I realized that the initial binary required to bootstrap the process had not been nailed down, but reports indicated the 2016 version was known to work. After the back-and-forth over what pieces someone might have on hand, I decided what I really wanted was access to the full collection.

In the unsurprising heathen manner, Adacore had broken their download links, apparently in the course of a move to Amazon's Cloudfront delivery network. Looking for the new locations, I was again greeted by the non-index page, felt the deja vu, and unearthed my old work. It appeared the "feed" format was unchanged and the new URL format was easily constructed from the old data. More surprisingly, the feed URL itself still pointed to the vanished "mirrors.cdn.adacore.com" hostname; as far as I could see, downloads wouldn't be working even with full JS. Behold the incredible bandwidth savings realized by moving to the Cloud! On the bright side, my existing SHA1 checksums provide some assurance that the historical files have not been diddled since 2018.

The code

Despite the tree-structured JSON format, the feed is essentially tabular. We'll use Python so as to accurately parse the JSON and convert to a more manageable comma-separated format (taking care that the separator characters don't occur in the values themselves).

import json

# http://mirrors.cdn.adacore.com/gpl_feed
rels = json.load(open('gpl_feed.json'))['feed']['releases']

fields = (
	'name',
	'id',
	'size',
	'sha1',
	'type',
	'client',
	'component',
	'date',
	'display_name',
	'display_order',
	'kind',
	'platform',
	'platform_display_name',
	'platform_display_order',
	'release_date',
	'release_name',
	'title',
)

table = [[str(obj[f]) for f in fields] for obj in rels]
table.sort()

for rec in table:
	for val in rec:
		assert ',' not in val and '\n' not in val

print ','.join(fields)
for rec in table:
	print ','.join(rec)

The output includes column headers and can be processed by any number of standard tools. How about a shell one-liner to produce the full URL listing?

awk -F, 'NR>1 { print "https://community.download.adacore.com/v1/" $4 "?filename=" $1 }' | uniq

Note that downloads are not uniquely identified by filename!

The data

And no, I don't intend to go mirroring the whole 21 GB of it, though if you do, I'll gladly link it here. Once I'm more clear on what parts are truly needed, I'll probably host those.

Enjoy!

  1. Terms pertaining to this effect, ranging from technical to marketing, include XHR, AJAX, and Web 2.0. [^]
  2. At some point the code was imported by the main GCC project, but for reasons I haven't yet ascertained, that version was considered broken for the purposes of The Most Serene Republic, so Adacore remained the source of record while efforts were made for the Republic to take over that role. [^]
  3. Namely, one dated 2007, being the oldest available, and 2014, for reasons I forget but perhaps because the next one came with a precipitous size increase. [^]

2020-03-31

Adventures in the forest of V

Filed under: Historia, Software, V — Jacob Welsh @ 19:11

It started as what I thought a simple enough job: take the existing SHA512 v.pl I'd been using to press the Bitcoin code, or rather the VTree that grew from it, swap out the hash with my own keksum so as to avoid a hefty and otherwise unnecessary GNAT requirement, add my version of the classic vdiff modified likewise, bundle up a "starter" to cut the bootstrapping knot, and publish the bunch as my own tested and supported offering for wherever a V may be needed.

Such a thing would still require Perl, itself not an insignificant liability. While work had been underway to replace that, the results fell short of completeness, and from the ensuing discussion I decided it would be best to shore up my own grounding in the historical tools before venturing deeper into the frontier. I suppose I should be glad, because I got even more of that grounding - or swamping, more like - than I had asked for.

I.

One pitfall I already knew was that file header lines in the "unified diff" format used by V, which begin with "---" and "+++", cannot be accurately distinguished from deleted lines beginning "--" and inserted lines beginning "++", if parsing linewise and statelessly as done by the original "one-liner" vdiff. This was discovered in practice through an MP-WP patch containing base64-encoded images, and the potential damage is hardly restricted to that; for instance both SQL and Ada programming languages use "--" as comment marker. This was part of the motivation behind vtools, which took the approach of avoiding the system's existing "diff" program in favor of a stripped-down version of the GNU codebase with integrated hashing. My own approach had been more lightweight: tightening up the awk regex to at least reduce the false positive cases. It wasn't too satisfying, but had worked well enough so far.

II.

The first surprise I hit (stupidly late in the process, after I'd already signed my patch and starter) was that the Busybox version of "diff -N" replaces the input or output file path with "/dev/null" for the cases of creation and deletion respectively.

This reflects a larger reservation I have about Busybox code: it's been hacked extensively toward the goal of minimizing executable and memory footprint, which sometimes but only sometimes coincides with clear code and sensible interfaces. In this case, from brief inspection I surmise that it literally uses /dev/null so as to avoid some kind of null check in the downstream code that compares and emits the header. It's clever, but breaks compatibility with the GNU format in unforeseen ways.(i) In fairness to Busybox, the format was poorly specified in the first place - and nobody involved with V apparently found this important enough to remedy either.

III.

Another surprise for me was that the sloppy parsing affects not just diffing but pressing too. At least v.py and v.pl exhibit the same sort of blind regexing in extracting antecedent information from vpatches. (I'd guess that use of somewhat tighter regexes has prevented this from causing trouble in practice yet.) Further, v.pl extracts file paths only from the "---" part of the header which suggests it would indeed be broken by "/dev/null" style patches. On the extended vtools side, vfilter makes yet another assumption not backed by either such documentation as exists for the format or the Busybox version: a line showing a diff pseudo-command at the start of the header.

IV.

Finally, I've noticed what strikes me as a design problem affecting all V implementations, which I haven't seen mentioned before: it's not possible to have the same (path, hash) pair as an output of two different patches in the same VTree. More simply put, you can't have a patch that changes a file back to a previous state, contrary to the suggestion that "adding and removing the null character from the manifest file in every other patch would work" seen in the manifest spec. The reason is that both patches would end up in the antecedent set of a patch referencing either version of the file, in some cases producing a cyclic graph.(ii)

Stay tuned for the aforementioned patch and starter that make progress on a few of these fronts.

  1. A related annoyance I've had is Busybox "diff -qr" doesn't report added or removed directories, while adding -N replaces "Only in ..." messages with the less helpful "Files ... differ". [^]
  2. At this point I must say I wonder why V wasn't made to simply include in the header of each patch the hash of its antecedent patch as a whole. It would have neatly bypassed all these problems, forcing a tree topology and simplifying implementation. Would it have smelled too much like Git, or what? [^]

2020-03-07

JFW's 130 top Trilema picks to date

Filed under: Bitcoin, Hardware, Historia, Lex, Paidagogia, Philosophia, Politikos, Software, Vita — Jacob Welsh @ 16:25

Inquiring minds have asked of me to please shed a bit more light on what this Republic thing and that Popescu fellow in particular are all about. Is there more to it than the ravings that first meet the eye, of sluts and slaves and scandalous sexual predations and every "ism" and trigger word known to man or woman? What's the value I see in it that keeps me coming back? And what's the plan for this world domination thing anyway?

I gave the most accurate response I could, if not the most helpful: see, all you gotta do is read a couple thousand articles in multiple languages averaging maybe a thousand words each, a couple times over, and likely a bunch of the imported cultural environment and extensive chat logs besides, and then all will become clear! At least as clear as it can be so far. At least I think it will. But what would I know, I'm a long ways from being there.

Well great, so couldn't I at least give an executive summary? Not exactly an easy task either. Short of that, here's an attempt at picking some of the especially interesting, informative or significant articles on Trilema from my reading so far, a map of sorts of enticing entries to the rabbit hole.

The very unfair process that articles went through to make this list was as follows:

  1. I extracted an initial set of 957 items from my presently accessible browsing history, using some CLI magic.(i)
  2. I narrowed the list to those where I believed I recalled something of the article, going off the title alone. This brought it down to 424.
  3. I further selected based on roughly the above "interesting, informative or significant" standard in my subjective perception, again by memory from title alone.(ii) I also ended up skipping some that would have met this by way of having especially horrified me; not sure if I've done anyone any favors thus, but there it is.

The ordering within each publication year is merely alphabetical (because I can't quite see a pressing need to do it better in this context).

Enjoy... if you dare. What can I say, it's not for everyone.

2012

2013

2014

2015

2016

2017

2018

2019

2020

  • The slap and human dignity
  • Fin.

    1. You know Firefox keeps this in a SQL database, yes? Because they told you about it in the manual, and documented the schema and all? [^]
    2. At times I was overpowered by the temptation to go check, with the inevitable expenditure of time on re-reading which, useful as it can be, I hadn't planned on getting drawn into just now. And while my shiny tools got this down to a minimal "this button to keep, that button to skip" flow, they were entirely powerless to speed up the thinking. [^]

2019-12-14

Uruguay parte 4: el turismo es trabajo

Filed under: Historia, Politikos, Vita — Jacob Welsh @ 16:11

Concluding from Parte 3.

On Sunday my host generously treated me to a five or six hour walking tour of the city, starting at the Feria de Tristan Narvajo, a weekly flea market named after the street it centers on in the Cordon neighborhood, where we browsed a while, then headed west along Avenida 18 de Julio through Centro to Ciudad Vieja (old town) and the seaport, then back and northeast to El Obelisco and finally back southeast to Pocitos. His extensive time on the pavement and accumulated trove of information became all the more apparent. He figured we did about ten miles on foot on this part of the day alone and noted it was certainly more than whatever else passes for tours these days.

The previous night he had made his rounds near the site of previous post-election destruction and observed light tank, helicopter and searchlight presence, apparently effective at deterring a third weekly recurrence.

Once I had managed to rouse myself, grab a bite and walk to his place, we proceeded by cab to be sure to catch the Feria. As at the airport, the orange-and-white themed cab system is far more orderly than Panama's, with taxi stands (this one unoccupied presumably due to being Sunday, but we didn't have to wait long for one to stop by), metering, divider between front and back seats and a better kept (or at least more consistently so) fleet from what I saw. Cost is higher but still reasonable.

uy-44

uy-45

Exiting Pocitos, looking northbound down the Boulevard General Artigas if I'm not mistaken.

uy-46

Things get a bit less polished in Cordon.

uy-47

Arriving at the entrance to the Feria.

uy-48

My illustrious guide.

uy-49

The Feria had all manner of things from varyingly finished woodwork...

uy-50

to outright junk...

uy-51

to antiques such as typewriters...

uy-52

guns...

uy-53

rotary phones, some sporting the Antel brand which has enjoyed its monopoly from the early days...

uy-54

treadles...

uy-55

kerosene lamps...

uy-57

mechanical calculators...

uy-58

and more guns...

uy-59

to clothing, plants, birds...

uy-56

uy-62

to cameras new and old...

uy-60

to some nice amethyst, one of the few items of local origin...

uy-61

and more generic polished rocks.

uy-63

Not pictured were stands with fresh produce, empanadas and similar prepared bites, jewelry, books (mainly well-used and yellowing paperbacks from what I saw), and paraphernalia for Mate consumption.(i) Something to do with cannabis too, its being fully legalized now: some "art" but I don't recall whether any consumables.

uy-67

Exiting the Feria onto 18 de Julio where we find an Universidad de la Republica. The stand in front with red handwriting was promoting a hunger strike (better pictured).

uy-64

Some live percussion on the corner...

uy-66

Then a national library...

uy-65

where Cervantes, or perhaps his ingenious Don, reminds us that he who reads much and walks much, sees much and knows much.

uy-68

Many shops here were closed for Sunday, leaving a good view of the graffiti on the roll-down metal covers protecting the glass within.

uy-69

McD's will use old buildings if needed to get that corner real estate.

uy-70

The local Communists and friends, known as the Frente Amplio, fly an inverted Russian flag.

uy-71

uy-72

uy-73

uy-74

Aaron takes note of a new message scrawled on the Ministry of Social Destruction by someone who doesn't want her rights pissed on. Er, stepped on.

uy-75

The Bronze Statue of Man on Horse series begins.

uy-76

uy-77

The Intendencia de Montevideo, a government office and one of the taller buildings with Mirador public observatory from which we were going to "see the entire city (if not up close)"; sadly it was closed that day, for reasons the posted officers didn't know.

uy-78

uy-79

uy-80

uy-81

A particularly elegant cupola, I thought, with neighboring buildings of entirely differing character.

uy-82

uy-83

The lamppost builletin board is alive and well.

uy-84

I would have liked to see if the arcade machines were as old as the signage evokes.

uy-85

Park with some epic struggle depicted, fountain not operational...

uy-86

But side fountains were.

uy-87

A look toward the Palacio Legislativo, suggesting this is from Plaza Fabini.

uy-88

Four blocks further and we reach the sizable Plaza Independencia, marking the transition from Centro to Ciudad Vieja.

uy-89

uy-90

uy-91

uy-92

Presidential office gets a nice building. Whatever's to the right, not so much.

uy-93

Nor do the offices look much more pleasant inside.

uy-98

Radisson.

uy-94

uy-99

uy-95

uy-96

The biggest statue is for national hero Artigas, who died in exile. I'm told that ashes are kept below and might even be his.

uy-97

Old city gate.

uy-100

There's this rainbow-filled park-of-sorts tucked around a corner.

uy-101

Pictures and sign read "Trans Law Now", "Fight for Diversity", "Constructing the future with love".

uy-102

Back to park-like parks; some National Party colors.

uy-103

uy-104

First glimpse of the port down a street in Ciudad Vieja.

uy-105

It's a mix of the very old and the new.

uy-106

uy-107

Instituto Nacional de Colonizacion: I'm told there's an active homesteading program for unused land in the interior.

uy-108

uy-109

Caribaldi, chief of naval forces of the Republic, 1842-1848.

uy-110

Across the water to the Antel tower, tallest building in the country at 158m if the Internet is to be believed.

uy-111

Port facilities.

uy-112

Holding pen for containers, and a cruise ship.

uy-113

uy-114

View across the water to what might be a refinery, past some wreckage I hadn't noticed at the time.

uy-115

uy-116

uy-117

Radar, presumably for maritime traffic control.

uy-120

Fortress atop a distant hill.

uy-119

uy-118

There was this nice semi-indoor plaza, the one exception to the uniform restaurant cost thing.

uy-121

The town has its run-down parts.

uy-122

It's a peninsula, with water visible in both directions.

uy-123

uy-124

I hadn't recalled seeing an evergreen yet.

uy-125

As-yet unidentified man on horse. And sheep.

uy-126

uy-127

Juniper if I'm not mistaken.

uy-128

uy-129

Itau and BBVA branches.

uy-130

Some fine woodwork on the Palacio Santos, home of the ministry of foreign relations.

uy-131

En esta plaza, el dia 24 de abril de 1925 el fisico Albert Einstein mantuvo un dialogo con el filosofo uruguayo Carlos Vaz Ferreira.

Homenaje del Consejo de Educacion Tecnico Profesional (UTU) y el Gobierno Departamental de Montevideo 30 de junio de 2005, a los 100 años de la Teoria de la Relatividad

If the bench were traveling half the speed of light, would it still fit within the chains?

uy-133

Lavalleja.

uy-134

In Ciudad Vieja and on the way back we were approached two or three times by beggars: bold, persistent, sad stories at the ready, and ungrateful when I did once offer a coin against my better judgement.

uy-135

Back at Cordon, the Feria was packing up. Below, one of the air-cooled VWs that provide an inexpensive transport option as many were produced nearby (Brazil?) and imports otherwise face steep tariffs.

uy-136

Classic Plymouth soft-top, apparently in decent shape, a rare sight here.

uy-137

We made our way to this freestanding radio tower, marking an approximate center of the city and visible from many directions due to height.

uy-138

Mega-flag and mega-cross; Aaron tells me there were three when the Pope made his one and only visit.

uy-139

The promised obelisk: A los constituyentes de 1830. Bystanders included at the base for scale.

uy-140

Heading back to Pocitos; there's a hospital complex behind the row of trees. Perhaps you'll recall this boulevard from the cab ride.

uy-141

Petrobras, soon to be leaving the country.

uy-142

Pole painted in Commie colors; private school in the background.

uy-143

uy-144

uy-145

Church of Christ, Scientist. No sign of Scientologists though.

uy-146

This time we dined at a Club de la Papa Frita. Determined to get in some beach time, we went home to change and, at least in my case, rest up a bit from the day's mileage.

A waxing gibbous hangs above the Rambla just before dusk.

uy-148

Hitting the beach, with sunset peeking through the distant showers.

Health hazards are sometimes flagged for the water e.g. due to city runoff after a rain or cyanobacterial blooms, but no problems this time.

uy-149

uy-150

uy-151

We walked the surf for a ways then back. Among topics that came up were DDoS attacks; Aaron reflected on how they had varied depending on time and context, comparing things to a shallow beach where waves can travel far and steeper beach where they break.

uy-152

uy-153

180+ saved shots yet only one of the cameraman; I oughta take more initiative about requesting these.

uy-154

A look at the hotel room prior to rolling out Monday morning.

uy-155

The balconies were a nice touch, at least on this side that faces street rather than concrete wall.

uy-156

Kitchenette stocked with plates, glasses and cutlery; works for leftovers if not quite for cooking.

uy-157

Bathroom with separate bidet, which might be the first time I'd seen this type IRL. (Panama does this hose on the wall thing, cheaper I suppose.)

uy-158

At center, the World Trade Center tower that once housed a datacenter, the failure of which set in motion the whole chain of events that led me to this spot.

uy-159

uy-160

The cab driver tuned to a radio program on which there happened to be some extended chatter about Bitcoin y Blockchain en Uruguay. I didn't make out much but it sounded like someone promoting a conference.

uy-161

uy-162

uy-163

uy-164

uy-165

uy-166

Chilling at the gate, this time with plenty of time to spare. The wait for check-in was reasonable; security was noticeably less obnoxious just for that seemingly small difference of leaving shoes and coats alone.

uy-167

Eat enough of those Uruguayan portions of mozzarella and you too could transition from underwear model to pear.

uy-169

Perhaps airplane photos are a cliche but I still enjoy them...

uy-168

uy-170

uy-171

Likely the Hipodromo de Moroñas, from a glance at the map.

uy-172

uy-173

uy-174

uy-175

Unlike Panama, Uruguay supplied real butter with the airplane dinner. Aaron also tells me that unlike Europe, they produce consistently unadulterated olive oil.

Near Peru, some Andes poking through the clouds.

uy-176

uy-177

uy-178

uy-179

uy-180

Approaching Panama.

uy-181

The decent beaches here are a ways out from the city.

uy-182

uy-183

Clockwise from bottom: the causeway islands of Amador; the old town Casco Viejo with bypass on the water, the coastal Avenida Balboa, Punta Paitilla.

uy-184

Mouth of the canal.

uy-185

Punta Paitilla and the manmade Punta Pacifica, with recently added islands.

uy-186

On arrival I had what seemed like at least a kilometer to walk from gate to immigration; the moving walkways along the way were out of service just as they'd been my last several trips. Despite walking briskly and getting in the short line for residents, by the time I got through the bags from my flight had already been unloaded from the belt and stood in a row. I suppose they think this is helpful, or necessary to make room; it does tend to confuse the newbies.

I faced another vague list of things requiring customs declaration, citing all sorts of old and recent laws and decrees, and decided I could argue that my stuff qualified for exemption if need be while declaring might draw additional scrutiny. It probably would have been about the same either way. As I'd mentioned, the servers attracted some curiosity on the X-ray belt, but no further trouble once it was clarified they were my own, not TVs, used, relevant to occupation, or whatever other checksums the agent was looking for.

Congratulations: we've made it to the end. I hope you've enjoyed the tour of the tour; I certainly enjoyed the thing itself and the recounting. Till next time, que les vaya bien!

  1. Hot infusion of Yerba Mate, the local caffeinated drink of choice and apparently something of a ritual involving special gourds and straws, though I didn't get to witness or try. [^]

2019-12-11

Uruguay parte 2: llegada y primeras vistas

Filed under: Historia, Politikos, Vita — Jacob Welsh @ 20:12

Continued from Parte 1.

My text having overtaken the start of photography, I'll have to backtrack a bit to Montevideo's Aeropuerto Internactional de Carrasco (MVD) which was looking quite shiny and new. Bag claim (evidently I misremembered: there were three on the international side, though just the one active):

uy-2

Aduanas (customs). That bienes de ingreso/egreso temporal would seem vague enough to cover just about anything if they felt like it; fortunately they didn't give a second glance (perhaps even first glance) to my scandalous screwdriver and packing materials.

uy-1

Free at last, but not quite home.

uy-5

I would have picked up a local SIM but the booth was closed for the night. It turned out my Panama SIM worked on roaming, at least briefly, which it hadn't in the US.

uy-3

The bit of the world that is me thanks Uruguay for the welcome.

MUNDO, BIENVENIDO A URUGUAY

In contrast to Panama, there was no crowd of taxi syndicate reps soliciting eagerly. Instead it's an orderly racket; you go to the taxi counter and arrange a ride with prepayment and receipt. Having been warned the cab would be around US $55, I held out for the $13 shuttle bus, taking the wait time to replace that stolen sunscreen, collect my thoughts and decompress a bit. I found myself tired but alert and relieved.

The only exterior shot I managed of the airport, so it'll have to do:

uy-6

Some Himpton by Halton thing near the airport with well-lit street:

uy-7

First shuttle stop was at the Motel Bahamas:

uy-8

A pleasant nighttime drive down the coast and another one or two stops later and I'd made it to my destination in the relatively nice Pocitos neighborhood.

uy-10

uy-9

The buildings here cap out around 10-12 stories due to zoning. Most are mixed-use, with shops at ground level and apartments above. My first impression of the area compared to most of Panama City was of something older (turns out many buildings date to the 1930's if I recall), more stable (as opposed to wreckage and new construction everywhere), cleaner, and far more pedestrian friendly (wide and not entirely treacherous sidewalks). Aaron pointed out that this does not apply to the whole city, with outlying neighborhoods ranging from more typically LatAm to outright favela (though these not walled off as in Brazil).

uy-11

Right around the block was an ANCAP gas station with rare 24-hour convenience store and deli, which served me for breakfasts, rather dreadful espresso (they couldn't believe I didn't want sugar, which probably says it all), and a printed map so as to navigate free of any "mobile device" nonsense.

First daytime views of the coastal Rambla, supporting vehicle, bike and pedestrian traffic and beach access, as I made my way to meet my host.

uy-12

uy-13

Oh yes, the street signs serve advertising; it does seem to help keep them in good shape.

uy-14

One of the larger mini-parks opposite the beach, near the Avenida Brasil.

uy-15

Battery scooters for hire.

uy-16

"Por la vida y la convivencia" : La Policia seem to like their mottos...

uy-17

There's a lazy tourism option.

uy-18

I have no idea. It didn't seem animate.

uy-19

"Orgullosamente blanco" - "proudly white" - referring, I gather, to the party colors of the recently victorious Partido Nacional rather than something racial.

uy-20

Either Ave. Brasil or Espana, two thoroughfares that converge at the coast.

uy-21

Corner florists seem to be thriving...

uy-22

Corner locksmiths not so much.

uy-23

Heading inland a bit; some gym/yoga place.

uy-24

uy-25

Apparently they don't need no education at the Center of Foreign Tongues. Aaron tells me the buildings generally don't get repainted much because maintenance work is taxed the same as new construction.

uy-26

Perhaps this would have been the spot for a better coffee.

uy-27

uy-28

There's minimal piped natural gas infrastructure (as in Panama, though there it's often provided building-wide and refueled by tanker trucks).

uy-29

Pizzeria Trouville

Another florist, and some of the typical sycamores lining the streets.

uy-31

To be continued with rare electronics and proper tourism.

2019-12-03

Keccak background

Filed under: Bitcoin, Historia, Software — Jacob Welsh @ 18:52

"Keccak" is a cryptographic hash function, or rather, some primitives for constructing such functions in a desired size and shape, of relatively recent design as these things go. It was brought to the attention of the forum in early 2016 in the context of contemplating changes to the Bitcoin protocol,(i) (ii) (iii) and subsequently differentiated from SHA3.(iv)

Compared to the prevailing standards at the time - mostly variants on the MD4 concept, processing blocks of input through an iterated compression function - Keccak is based on a large pseudorandom permutation (1600 bits, though the spec also defines smaller variants). As this is readily invertible, the desired "one-way" property is provided by a "sponge construction", mixing in blocks of input and extracting output while iterating the permutation and keeping some number of its bits secret as internal state. This number is called the capacity (or by complement the rate, the two summing to the permutation bit width) and can be tuned for the desired balance of security and computational intensity. The construction can take unlimited input, or produce unlimited output as a kind of stream cipher.(v)

I started out in 2017 playing with a C implementation found in the wild, supposedly a "readable and compact" version written by the original team. With some cleanup I got it into a state that could be described as compact, but I couldn't get very far in reading it, at least without having first digested the spec. And it had the unfortunate limitation of requiring the full input and output to exist in memory, no streaming. My confidence as an applied cryptographer was growing and I soon implemented a number of classical hash functions, but set Keccak aside as not being an immediate necessity. Meanwhile, Diana Coman produced and incrementally published a very nice and documented reference implementation in Ada, which was adopted for use in V and soon became non-optional.

While I was well convinced by the Republican rationale for Ada, I was much less keen on introducing GNAT, the flagship implementation, into my environment. It was a million-plus-line-of-code beast that I wouldn't stand a chance to ever really understand; making matters worse, it was a "Thompsonism", a circular dependency requiring existing binaries in order to build from source and thus dubiously "open source" at all. While I already depended on one such thing - the C compiler - I was hoping to somehow keep this to ONE thing, or at least ensure a way to work with the crucial V on existing machines without pulling all this in.

Stay tuned for the result.


  1. mircea_popescu: actually i wouldn't go to war over keccak.
    mircea_popescu: letting bitfury & friends eat 100mn in unrecoupable engineering costs would provide exactly the correct lesson as to what it's a good idea to say and when it's a good idea to shut the fuck up and toe the line.

    [^]

  2. The necessary prerequisite for any change to the Bitcoin protocol [^]

  3. mircea_popescu: http://log.bitcoin-assets.com/?date=01-02-2016#1393026 << at least it wasn;t fucking developed by teh nsa.
    assbot: Logged on 01-02-2016 19:29:18; ascii_butugychag: ;;later tell mircea_popescu in what sense is adoptinc keccak a rejection of usg standards? it was actually adopted as sha3...
    mircea_popescu: as far as we know. whatevs. minor point.
    ascii_butugychag: btw between that thread and now i went and read the keccak spec
    ascii_butugychag: it is mighty spiffy.
    ascii_butugychag: accordionizes to size.
    mircea_popescu: :)
    mircea_popescu: i don't need to explain what i meant by not finite then ?
    ascii_butugychag: aha.
    ascii_butugychag: other hashes also accept infinite bits but they eat where they shit.
    mircea_popescu: quite.
    mircea_popescu: and mind that while in no means do i propose this is "Asic resistant", from a designer perspective you must appreciate i'm giving you a fun job to do.
    mircea_popescu: at least therer's that.
    mircea_popescu: always make sure everyone's having fun.
    ascii_butugychag: quite! nobody will be plagiarizing old verilog from fpga docs to bake this one.
    ascii_butugychag: very asian-resistant.
    ascii_butugychag: which is a mega-plus.

    [^]


  4. asciilifeform: holyshit the original keccak www is gone
    asciilifeform: replaced with some horrorshow
    asciilifeform: ada code -- gone
    asciilifeform: fortunately still on my hdd
    asciilifeform: check this out, keccak.noekeon.org now forwards to buncha tards
    asciilifeform: https://archive.is/GkmgU < original
    shinohai: Notice that happened after nist.gov declared their spec
    asciilifeform: shinohai: not immediately , iirc was still intact last yr
    asciilifeform: incidentally shinohai keccak != usg.sha3
    asciilifeform: they adopted ~particular params~ of keccak as the new national whatever
    asciilifeform: orig is ~family~ of functions.
    asciilifeform: see also https://archive.is/lViVh << since 'unhappened' article
    asciilifeform: ' The SHA-3 version of Keccak being proposed appears to provide essentially the same level of security guarantees as SHA-2, its predecessor. If we are going to develop a next generation hash, there certainly should be standardized versions that provide a higher security level than the older hash functions! NIST, in the original call for submissions, specifically asked for four versions in each submission, with at least two that would
    asciilifeform: be stronger than what was currently available, so it's hard to understand this post-competition weakening.'
    asciilifeform: didjaknow.
    asciilifeform: notice how 'everyone' nao thinks 'oh, keccak? that's called sha3 nao' [^]
  5. Since state is still finite, output will of course repeat eventually; one would hope this cycle length approaches that order of 21600. [^]

2019-11-27

Early history of me, part 6

Filed under: Ego, Historia, Paidagogia, Vita — Jacob Welsh @ 18:22

Continued from part 5

Another eventually-successful parental negotiation involved my music studies. While my violin skills had advanced substantially from ages six to twelve, both solo and in orchestra, and I enjoyed performing, I had never quite accepted the burden internally, and the rigors of daily practice continued to grate. It probably didn't help that my parents weren't demonstrating much musical discipline themselves. If you want to raise a Wolfgang Mozart, it helps to be a Leopold Mozart, y'know?

At the same time, I'd dabbled a bit with the piano, because it was there, and it called out to be tackled properly. I convinced them to let me switch; we found a local teacher (at greater expense, if I recall, for having to look outside the organization) and I pursued the study with vigor. Unfortunately this only lasted about two years until we couldn't seem to make time for it among the increasing demands of school.

Some words about extended family would seem in order to round out an overview of my childhood. There was one set of grandparents surviving, my mother's side, who had retired about an hour north (a seemingly interminable drive at that age) in Gettysburg, Pennsylvania.(i) We'd visit every month or two. I liked them better than my parents did, probably due to less historical baggage on one hand and their inclination to spoil me on the other. When I slept over I'd be able to watch cartoons and play with Grandpa's Mac (with color display!) for hours. They had an affinity for the Arab world, having spent their careers as professors at the American University of Beirut. "Sittou" as we called her was the only churchgoer (Lutheran) in the clan, while Grandpa was a kind of tolerant non-believer. There was an uncle with family that I'd usually see at the grandparents' place.

On my father's side there was an elder aunt and family in Maine; due to the distance we'd see them yearly, at least in the good years when we could afford the vacation. They had picked up the tab on a coastal summer cottage that had been in the family a few generations; I remember with great fondness the change of scenery, climate and pace afforded by these trips; the smell of pine forests and ocean.

While all sorts of details could be relevant to the story of childhood, I will close this series with one that made a distinct mark on me and my generation: the events of the morning of September 11, 2001 and subsequent descent into war on an emotion. It was a school day in the sixth grade. The administration's first reaction was to say nothing, but by lunchtime a growing list of names was being called to report for early pickup, and rumor spread: "the country is under attack!" The superficial facts became clear soon enough, if not the interpretation. Following my parents I was skeptical of the official narrative; LaRouche had even spoken of the possibility of a "Reichstag fire" i.e. false flag event, before it happened. Whatever the Bush/Cheney administration's negligence or even complicity may have been, things played right into their hands. There was an upswell of patriotic fervor, with the songs, "United We Stand" posters and "Fight Terrorism" bumper stickers. I noted the blue skies vacant of contrails as civilian flight was suspended in the following weeks, and the later conversion of airport "security" from this quaint thing with X-ray machines to the complete exercise in humiliation that the inmates now take for granted. As the war whoops escalated, the average low-information voter didn't seem to perceive a difference between supposed Saudi hijackers, Taliban, or Saddam Hussein. Someone had to pay and it didn't much matter who. It marked the beginning of an end of innocence, both in the culture as a whole and my relationship to it.

  1. Perhaps most famous for its battlefield, regarded as the turning point of the American Civil War. [^]

2019-11-26

Early history of me, part 5

Filed under: Ego, Historia, Paidagogia, Vita — Jacob Welsh @ 17:13

Continued from part 4

Lest I paint too bleak a picture of a flat landscape in the public school system, special-needs options started to be offered around the fourth grade for those afflicted by working brains, namely honors classes and once-weekly programs with pompous acronyms whose meaning nobody remembered like FUTURA and SPECTRUM. These provided welcome relief, but remaining surrounded by a crowd that was none too interested in that whole learning thing, and probably resentful of being subjected to it, was still draining. And even in the honors classes, I found the ever-expanding homework burden full of silly, pointless or repetitive drudgery. Around the eighth grade I chose to drop my "straight-A" record to make more time for my interests outside school, which by that time had gravitated toward computer programming.

I sometimes complained to my parents about the situation. Why not the local private school where my friends from the organization went (by financial support from extended family)? Why not home-school? Such entreaties would be dismissed in the "yes, but" style.(i) While they did help pressure teachers and administrators into better supporting me, questioning the system itself was off the table. I see it as a kind of passivity from assumption of helplessness, lacking adequate consideration of what might have been possible or weighing of longer-term costs among proximate ones. When high school came around, there was finally a more serious option of a full-time magnet school;(ii) the proximate cost was being in the next county over with a lengthy bus commute. They rejected this on the first pass, hoping the local situation would improve. To their credit, they came around once it clearly wasn't improving and I got in as a sophomore transfer; unfortunately this meant having missed out on a number of freshman bonding experiences.

To be continued

  1. That "well, yes, but..." was a phrase often cited by LaRouche regarding the avoidances of potential recruits. [^]
  2. Thomas Jefferson High School for Science and Technology. [^]

2019-11-25

Early history of me, part 4

Filed under: Ego, Historia, Paidagogia, Vita — Jacob Welsh @ 17:31

Continued from part 3

There was none of that IV drip of brain sugar known as television in the household. My parents were quite concerned with the developmental effects of screen time, especially of that aimless and passive sort; movies and video games were fairly restricted as well. As intended, this directed my entertainment desires to books. My father's reading to me became a cherished evening tradition, and I eagerly took up reading myself as I became able, with interests tending toward fantasy adventure and a bit of science fiction.

While I loved the family time, a sore spot for me is the amount of time spent being not-raised by not-them. Like many - I'd venture to say most - American kids of this era I was "institutionalized", with daycare from an early age (around one year, if I recall) feeding right into preschool, kindergarten, then school proper. I'm not too clear on how this compares to global or historical norms, but my understanding is that the crowded environment makes individual attention difficult and the constant change in caregivers disrupts bonding. "Because I said so" and "life's not fair" are the typical explanations I remember from the preschool authorities. While the unfairness point is perfectly true, I see its usage more as code for "I feel overworked and underpaid and can't be bothered to help you think through your kiddie problems." Not that solving one's own problems isn't important either, but I don't know... how much reflection or social finesse can you really expect from four-year-olds, especially if it hasn't been well modeled? The typical justification for this outsourcing is Money; the only difference in my case is that it was the Mission.

I don't well recall if or how I expressed myself about the preschool environment at the time, but as time went on the deficiencies of the cookie-cutter approach of the school system became evident. One story (handed down as I didn't recall it myself) was the third grade teacher asking, "What solid has the same shape on every side?" An eager me: "Actually, there are five:", proceeding to rattle off the 'hedrons with correct pronunciation that my eighth grade geometry teacher later wouldn't manage. Teacher: "Yes Jacob; but the third grade answer is the cube." Peer: "That's right Jacob, this is third grade!!11" Me: "That's right. Third grade, not kindergarten." Then in the sixth grade, there was that "science" teacher who earnestly believed the moon orbited the earth every 28 hours and criticized me as "argumentative" (something my parents were delighted to hear).

To be continued

Older Posts »

Powered by MP-WP. Copyright Jacob Welsh.