Next steps in wallet planning

Filed under: Bitcoin, Software — Jacob Welsh @ 18:40

The planning in my last article left a fair amount to the imagination, and in particular several pending decisions without sufficient detail to move discussion forward. The picture coming into view is that through a habit of insufficiently structured communications between engineering and management, I've been attempting to bear more weight of decision-making than is my due, trying to optimize things from the technical point of view which, while not unimportant, does not always align with the needs of a business to survive and grow in a dynamic market. I'll also take the opportunity to encourage present or future management to lean on me for support should it happen that I wander off into my own world.

With that in mind, let's expand a bit on the remaining tasks and decisions.

Switching to the now-builtin bignum support

I noted this as something that could potentially be deferred due to time constraints, because it does work as is. Unfortunately upon testing I've found that performance is noticeably degraded from what it used to be, at 14 seconds to derive an address from a private key on my dev system where I recall this having been closer to 5 seconds on earlier versions of the interpreter. (Transaction signing would be several times this.) I suspect a big part of the slowdown is the now-redundant layers of type and overflow checking; I'm unsure how to confirm this other than just trying the change. FWIW, I don't anticipate it taking all that long; maybe I take an hour and see if I can get it done.

Unspent output accounting; import of inbound or previously unknown outbound payments

The wallet tracks three types of data: private keys, the public keys and addresses derived from them, and transactions affecting its addresses. Queries such as balance and unspent outputs can be resolved by scanning the list of transactions, or tracked separately. As I see it, the "full scan" will have to work anyway so separate tracking, while desirable from an algorithmic scalability standpoint, would be a premature optimization at this stage. Without any such derived data to maintain, the "import" operation can simply mean appending lines containing encoded transaction data to a text file. Which brings us to:

External representation of transaction data (S-expression or maybe JSON)

I'm presently thinking S-exprs are the way to go; it keeps things simple on the Scheme side, and the Python side would only need to emit them, not parse them. What I'm not in a position to see is whether interoperability with other JSON-based tools would be desirable. I do have a parser in Scheme for a simplified form of JSON - omitting some Unicode details - that would suffice here.

Transaction input selection (menu-based?)

A possible cut here is to skip any sort of interface and use a simple selection strategy like FIFO. A manual interface could be added later. The idea here was to print a numbered listing of outputs by address/age/value and prompt for which ones to spend until sufficient value is accumulated.

ECDSA S-value normalization?

Turns out I've already got this: "As (r, -s mod n) is also a valid signature, the result is canonicalized to use the lesser of the two possible s-values." Thanks, 2017-me! (Some background.)

Fetching blocks from TRB (new RPC? Bitcoin protocol?)

In fact this could be done using the existing "dumpblock", perhaps using a named pipe. An RPC to return a block directly still seems preferable to me, but patching TRB isn't free. The "Bitcoin protocol" angle here meant connecting to port 8333 and using the p2p protocol commands to fetch a block, which is probably overkill if this is to be used in a batched or "pull" style.

Pushing raw transactions to TRB (likewise)

This one I don't think can be avoided.

Tracking watched addresses and synchronization state (SQL?)

The simple approach here, if indeed we're going for a batch model, would be a text file listing addresses and another that gives the last scanned block number/hash. When you run a scan, new transactions affecting the given addresses are emitted to a text file for transfer offline, and the last-scanned-block file is rewritten. If whatever data gets lost, you just rescan from an older block. Dealing with reorgs is where it could all get messier; one approach could be to insist on, say, 6 confirmations, and do a full rescan in the rare event of a deeper fork. How exactly one detects this and what could happen if one doesn't seems to warrant further consideration.


Gales Bitcoin Wallet: status, preliminary work plan and code dump

Filed under: Bitcoin, Software — Jacob Welsh @ 21:01

The item in question is the software component of a combination of hardware, software and operational practices with the goal of sending and receiving Bitcoin while excluding the possibility of remote compromise of keys (so long as underlying cryptographic assumptions hold). Proper airgapping is the foundation, but it's easier said than done when it comes to Bitcoin because of the dependency on data from the outside world for the inputs of a transaction. Popular "cold storage" solutions may involve storing keys offline, but have windows of vulnerability when the keys are generated and funds are swept using an online (i.e. insecure) machine.

Our approach is to allow data to jump the gap as selectively as possible, using a simple and low-speed channel (serial port) with optical isolation and manually switched simplex operation (one way at a time). The system is intended for manual, low-frequency usage; thus we're not especially concerned with performance or timing side channels.(i) At the same time, we're very concerned with correctness, which informs a preference for straightforward "textbook" implementations when possible, minimizing external dependencies and overall system complexity, maximizing auditability (e.g. avoiding any sort of processor-specific "crypto acceleration"), and publishing code.(ii)

Design points include:

  • An offline machine, running the offline component of the wallet. The machine is to have no swap, be operated on battery power during key operations, and powered off after use.
  • A set of private keys, generated offline by explicit request and associated with names of the operator's choice, stored in a tmpfs during operation and GPG-encrypted on disk using a strong password when at rest.
  • A true (i.e. hardware) random number generator, used for all key generation including the ephemeral keys used in ECDSA signing.
  • Transactions constructed offline (potentially reducing round trips through the airgap).
  • An online machine running a TRB node and the online component of the wallet, which pushes outbound transactions, scans for sufficiently-confirmed new transactions affecting the set of watched addresses, and produces minimal data structures for updating state of the offline component.


I've used my own Scheme system for implementation of the offline component so far, after an earlier prototype in the much larger (albeit more battle-tested) Python. Much of the code falls into one of two categories: math, or shuffling data from one format to another.

On the first category, I've implemented the following algorithms, with reference primarily to Menezes, van Oorschot and Vanstone and the SECG standards.

  • Bignum arithmetic: addition/subtraction, multiplication, division/remainder. My first pass here was intended more as an exercise, a "userspace" (pure Scheme) implementation supporting only unsigned values. I subsequently redid this, with signed integers and plugging into the generic arithmetic operators, still mostly in Scheme. The wallet code is presently still using the former. This is a conventional arbitrary-precision framework and thus does not support constant-time operations.
  • Modular inversion (extended Euclidean GCD).
  • Unbiased random integer in a given range.
  • Elliptic curve point addition and scalar multiplication, for the secp256k1 curve required by Bitcoin. Scalar multiplication is the most intensive operation, done once for key generation and several times for signing, and comes out noticeably slow; one optimization was to precompute iterated doublings of the generator point.
  • ECDSA signing.
  • SHA256 and RIPEMD160 hash functions.

On the second, the following, with reference primarily to the TRB and OpenSSL code (and sometimes the Bitcoin Wiki which was almost always a mistake):

  • Base58 and Base58Check
  • Little-endian hex and octet encoding
  • Fixed and variable-length integer byte encodings as seen in Bitcoin
  • The OpenSSL EC point (public key) encoding
  • The subset of ASN.1 DER as used by OpenSSL for ECDSA signatures
  • Bitcoin script, with code to construct p2pkh and p2sh outputs(iii)

These are presently wrapped up into commands by which one can generate private key files, the associated addresses, and (in case of emergency) export private keys to the "WIF" format for import to other wallet software.

The aforementioned Python prototype also implements the somewhat convoluted Bitcoin transaction signing algorithm, and I have Python code for deserializing blocks and transactions.

Remaining tasks

This list will surely need some revision upon contact with reality. The practical deadline so as not to wreck my own holiday is December 13.

For the offline component:

  • Porting transaction signing to Scheme
  • External representation of transaction data (S-expression or maybe JSON)
  • Switching to the now-builtin bignum support
  • Unspent output accounting; import of inbound or previously unknown outbound payments
  • Transaction input selection (menu-based?)
  • ECDSA S-value normalization?
  • Frontend commands
  • Shell scripts for "opening/closing" the wallet (GPG operations)
  • More tests

The online component, for which I'll probably stick to Python and make use of available libraries:

  • Fetching blocks from TRB (new RPC? Bitcoin protocol?)
  • Pushing raw transactions to TRB (likewise)
  • Tracking watched addresses and synchronization state (SQL?)
  • Export of new payment data for transfer offline
  • Alert for unexpected outbound payments

Partial code dump

Python prototype:

Offline component, Scheme code:

  • pkg.scm - basic import/export mechanism used by the remaining
  • bignum.scm - Unsigned "userspace" bignum library
  • ecdsa.scm
  • bit-ops.scm - some rather sad complexity for implementing fixed-width arithmetic and bitwise operations, which in hindsight would have been better done as interpreter extensions
  • hashes.scm - the RIPEMD and SHA families, based on the above
  • wallet.scm
  1. Certainly it would be preferable to eliminate these. [^]
  2. At least when it comes to cryptography, you don't gain much from obscuring your algorithms or implementation, because chances are you're not as clever as you think: all the ways to mess it up are already known to someone and will eventually be probed. [^]
  3. The latter might be a contentious point; it does not involve any support for "multisig", "segwit" or similar on my side, but enables one to transact with those who insist on these Gavinistic 3-addresses. [^]


Blogging photos and chat logs: some handy scripts

Filed under: Software — Jacob Welsh @ 20:37

For last night's article I had to pay the debt of having deferred my ever so carefully planned dig into photo blogging for quite so long. I started from Popescu's recipe and made it into a script, including doing the necessary jiggling so I wouldn't have to recall the details every time.

One detail I wasn't sure about was the function of the "size-large" and "wp-image-###" classes on the <img> tag, so I left them out. If I wanted to style an individually numbered image, I can't imagine why I'd want to do it in a global CSS file rather than a local style attribute but perhaps there's some other reason. Ideally I should have asked in advance.

Without further ado, the blog-photos script:(i)

echo -n "Output file prefix: "
read pfx
root="/wp-content/uploads/`date +%Y/%m`"
let cnt=0
for f in "$@" ; do
	let cnt++
	echo -n "<a href=\"$root/$pfx-$cnt.jpg\">"
	echo "<img src=\"$root/$pfx-$cnt-720px.jpg\" alt=\"$pfx-$cnt\" title=\"$pfx-$cnt\" width=\"720\" class=\"aligncenter\" /></a>"
	echo # paragraph break for spacing between photos
	convert "$f" -auto-orient -geometry 1920 "$pfx-$cnt.jpg"
	convert "$f" -auto-orient -geometry 720 "$pfx-$cnt-720px.jpg"

The chief dependency here is of course ImageMagick. Typical usage, from a directory full of pre-selected images:

blog-photos *.JPG > draft.html

You could of course use "tee" instead of redirection if you'd like to see the progress. Once you've uploaded the resulting images to the proper directory, paste the draft as a new article (I like "xclip" for this; of course a GUI text editor would work too) and load the preview; this allows you to identify image numbers by mouseover so as to find where to put your text.

Next up, a simpler one for importing chat logs in the presently prevailing style from a browser, with proper escaping of special HTML characters, logotron2wp:

sed -re 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g' -e 's/^([^: ]*)/<strong>\1<\/strong>/'

Come to think of it, the first three substitutions there also work for any sort of code import that might contain those pesky &<> characters - such as this one right here.

The comment box awaits any questions, comments, flames etc.

  1. And now we'll get to see how badly long code lines will render... [^]

Hiking La India Dormida

Filed under: Vita — Jacob Welsh @ 08:43

El Valle de Anton is a town situated in a dormant volcanic crater, about two hours' drive from Panama City (assuming you time it to avoid traffic). One of the surrounding rock formations is known as La India Dormida, because it resembles a woman lying on her back, with trees and rivers for hair, peaks for nose, chin and breast, and more gradual slopes for legs. I couldn't quite see it at least from the vantage points we got, but either way it made a great hike and a way to escape the swelter of the city and enjoy the cooler mountain air.

Crossing the Pacific mouth of the famed canal on the Puente de las Americas.




On the right, a typical Diablo Rojo. On the left, the Saturday inbound commute.


Highway passing through the suburbs.


November is the end - and peak - of the rainy season. The grass looks happy.


Off the highway and heading inland.


First view of mountains.



Looks like they're doing something with it...





On the outskirts of town there was a strong presence of chinos (convenience stores run by Nth-generation Chinese). At this one, a taste of the local classifieds: hard drives and rabbits.





We stayed at Hostal La Casa de Juan; I wouldn't recommend it. The perhaps nicer Bodhi was all booked.



Indeed they don't seem to worry about much. Nor do the dogs that roam freely around the town, one of which we encountered taking a nap in the middle of the road. We didn't get to trying the pool table.


The bikes were "free"; maybe one of them had a working brake. The bedroom had a thick odor of mildew, and we soon found why: when it rained, the roof leaked right onto one of the beds. When we pointed this out to the staff they acted surprised. Fortunately there were enough extra beds.


Setting out on the adventure.


Perhaps the botanists in the audience can fill me in?





After the first ascent.





This would be why it's called El Valle.



Sniping some folks on the next peak with the 40x optical zoom on my newish Lumix.






Defunct vineyard?


After the final ascent; the view is almost straight down. (They say I like a few hundred feet of rock under foot!)








There's an excellent waterfall pool we dipped in to cool off. To get there we had to scrabble a ways up the rather slippery rocks, while it was raining, so I wasn't risking the camera, but here it's kinda visible through the trees.


There must be some story to this massive thing...


Across the bridge it opens back to the road and we've completed a 3-hour loop.


Afterwards we grabbed a meal at Charlie's Empanadas y Pizza y something-or-other. Food was tolerable and service was surprisingly attentive, compared to what we'd heard of other places. Afterward we stopped for groceries at a Rey that looked just like they do in the city, made some proper piƱa coladas back at the Hostal, and played a few rounds of rummy, taught by Robinson as the rest of us had been missing out.

And we're back home on Sunday.



yrc, a Unix IRC client: genesis

Filed under: Software, yrc — Jacob Welsh @ 18:44

The problem I chose to tackle in February 2017 was the need for secure real-time communications in my planned business. "Secure" here didn't mean in the cryptographic sense, but rather having a guarantee, or at least as close to one as could be practically had, that there exists no "magic packet" that could be sent from the outside network to compromise the terminal, at least through this particular hole.

Many of the traditional IRC clients were written in C, a notoriously poor language for providing any sort of guarantee, and inevitably seemed to be loaded with "features" I didn't need, and thus potential for nasty surprises, and reams of code to read if I wished to audit. On the ultra-minimalist end of the spectrum there was ii, but it was way to little, for instance in providing awareness of connection state or tracking multiple channels. (It was telling that they suggested a far more complex program, "multitail", for this.)

At the same time, I was interested in sharpening my skills in network and UI programming; I'd done the latter in various environments including Visual Basic, Qt, PyQt, and urwid, but hadn't found the elusive sweet spot. I decided to take a shot at writing my own client using a high-level language and low-level terminal facilities. "How hard can it be, it's just an event loop, displaying text, parsing a simple protocol, an input state machine and so forth." Hah.

I did make the practical compromise of using Python, due to its fairly robust system interface, rather than, say, the Scheme interpreter I'd been hacking together around the same time. Initially I used the bindings for the classic "ncurses" library. One nice thing it does is track screen state so you can render however inefficiently and screen updates will be sent differentially: originally good for slow serial connections and still good for slow network connections. The other supposed benefit was portability across terminal types. I soon found it to be nothing but trouble, an additional layer of crufty complexity and the supposed portability not being watertight. So I switched my reference to the original DEC VT100 user guide, from which modern Unix terminals are more or less derived, and found the experience altogether more pleasant.

Some 2400 lines of code and a couple revisions later, I have something that I find more or less usable.

Major things that it does NOT intend to support include Unicode and SSL. Notable missing features it ought to have, quoting the manual, include:

  • Prompt history, kill/yank commands
  • Indication of nick mentions (ring bell + bold the window name?)
  • Tab completion (nicks, slash commands)
  • Formatting for displayed messages (e.g. bold sender, highlight own nick)
  • Finish mode/op and close-net commands
  • Filtering of inbound PMs (/ignore or similar)
  • Scrolling based on rendered lines rather than messages
  • Scrolling + more efficient navigation of window list

A fairly detailed manual is included, so see within to learn more.

Download Vpatches

(Hash algorithm is Keccak with 256-bit capacity and 512-bit output. This is aping the previous vpatches I've seen; I do not know the rationale for these parameters.)

Remedial time management

Filed under: Tempus fugit — Jacob Welsh @ 09:12

I find myself in what I'd like to interpret optimistically as a regressive phase of that messy process of growth.

After some years of a life carefully arranged so as not to burden me with much in the way of deadlines, where I could pursue whatever line of thought or activity seemed interesting or important and at whatever length I pleased, I chose to again submit to the rigors of the time-sensitive way of being. To my younger self these had seemed unwanted impositions from above; and this could even be true, if you take "above" to mean "reality", which doesn't care in the slightest about your wants. In this messy and entropic world, the time-sensitive way is the surest way to survive, to collaborate effectively, and to make the most of the limited time each one of us gets, and is a key part of the elusive "growing up" or "getting one's act together". (I've heard it said that women tend to figure this out earlier, from biological necessity, and I tend to believe it.)

Is it any wonder, after a period of youthful indolence, that my time management skills - which I don't recall having ever been made particularly explicit, relative to their importance - have atrophied? I loaded my schedule up, and things started falling off and making messes. While time management is not my only problem, I do believe it's a big one and would like to take this time while it's fresh in mind to set down what I've learned about it thus far, and primarily for my own reference.

1. Your time is a scarce resource, like any other, but highly non-renewable in that once spent it can never be earned back, and has no substitute.

2. You can never solve all your problems. You can solve some; they will be replaced with more; eventually the small ones will accumulate, or one big one will come along, you will be unable to cope, and you will die. The choices you get are which ones you're willing to suffer with, and how vigorously you will act to keep them at bay.

3. All those tools you know you ought to be using: plans, calendars, agendas, schedules... yes, you need to use them. Take care of them and let them take care of you.

4. Scheduling works by explicitly reserving minimum time blocks for important tasks (and honoring them).

5. Don't "snowball": if you drop one thing, avoid giving up or letting it spill over to other things too.

6. Buffers: allocate extra time to allow for overruns, and all the larger when it's unfamiliar territory.

7. Expect the unexpected: make time for the unplanned opportunities or challenges that arise so frequently.

8. When things don't fit, re-evaluate correctly what the gains and losses are from a given decision.


The road to Ossasepia, part 6

Filed under: Ego — Jacob Welsh @ 09:02

Continued from part 5.

With my worries put to rest as best they could be, I ended up following Robinson in applying. Since I had a number of articles to write already, which I had thought to be chiefly my technical backlog, I first wrote a plan to list and briefly summarize them. The plan itself took longer than anticipated, and the content raised alarms: it painted a picture of unbalance, like a man with one arm much stronger than the other, in a similarity to asciilifeform's case. The Marquess further emphasized the connection of technology and politics, in particular, that just publishing code and documentation would not suffice due to the large unknown of who I am. To start to address this she proposed an interview, which was completed; as I wrote afterwards,

giving a high-level view of my life story, with some back-and-forth, then answering questions for which my reading and mulling had at least somewhat prepared me. After over three hours of conversation, I found myself with a new Master and, not to waste any time, a blogging assignment drawn from my own plans to recount my steps in setting up the blog, a subsequent one to reflect on my month in the channel, and advice on the writing process.


My present understanding of the WoT, part 2

Filed under: Philosophia, Politikos — Jacob Welsh @ 09:00

Continued from part 1

In the envisioned society in which all parties one might want to do business with participate in the WoT, there are many ways to gather deeper information from it beyond merely what is said in the ratings; for instance, how well connected a subject is to the evaluator's own network, relative to how that subject ought to be connected based on his claimed history. Or one might draw completely different conclusions about the subject from what a rater claims, based on knowledge of the rater. To not be present in the WoT at all, once of a sufficient maturity to protect one's key, is to be an un-person, an outcast, perhaps even more so than today's "undocumented" or "stateless" persons.

Even in its present form, many of the benefits of the WoT are already available; it has enabled substantial volumes of online commerce, and indeed it serves as a kind of filter for people who are both sufficiently intelligent and aware to find out about it, and who seek to accept the burdens and reap the rewards of a world of personal responsibility. If the concept works as well as is claimed and participants use it diligently, along with the other elements of economy, then it's not a stretch to imagine WoT-society continuing to grow in wealth and power until it's the only game in town.


My present understanding of the WoT, part 1

Filed under: Philosophia, Politikos — Jacob Welsh @ 23:41

The Web of Trust (WoT) as conceived by Mircea Popescu is a set of concepts and practices for publishing and finding information pertinent to making trust decisions about people, or more generally, finding sources of such information for further exploration. "The WoT" can also refer to the mathematical, technological and human machinery implementing the concept in a given context.(i) Unlike the commonplace wishful-thinking based attempts at "reputation systems", it is understood not to provide mechanical answers to questions such as "is X trustworthy" (even when properly qualified as "for Y purpose"); trust exists in the people rather than in the tools, but the tools provide the structure for making one's own judgements.

The WoT is represented as a graph, with nodes being the public keys of the participants, corresponding to private keys generated by the individuals themselves; exclusive control of a private key is one's sole claim to identity in the WoT world. The edges of the graph represent the trust ratings between individuals, inbound and outbound, and consist of a sign (overall positive or negative trust), a number reflecting the rater's degree of certainty that the assessment will not change, and optional comment.

~ To be continued(ii) ~

  1. Presently: identity provided by the RSA cryptosystem as implemented in GPG, and a ratings database available through deedbot. [^]
  2. The installment style lately was assigned as an exercise in improving my abysmal writing speed by publishing whatever I can get out in one hour, with 300-word minimum target, spreading the pressure across smaller segments rather than allowing it be deferred. [^]


The road to Ossasepia, part 5

Filed under: Ego — Jacob Welsh @ 22:24

Continued from part 4.

My hesitation at this point came from three causes as far as I'm presently aware. For the first, there were my strongly independent tendencies that resisted the idea of taking up a harness to join someone else's team. I did not take these lightly, as I believed they had done well enough for me in many ways, from de-prioritizing the largely dull schoolwork of my late childhood years in favor of my own technical pursuits, to taking an interest in free-market economic theories over the objections of my family and undergraduate community, to getting out of the proverbial Mom's basement as soon as practical, to stepping down from a comfortable-enough suburban IT job to pursue my deeper interests in programming as well as Bitcoin, which I saw as one of those rare truly disruptive innovations, while trying out life in a foreign country. While I didn't manage to express or perhaps even quite understand this at the time, it was anticipated well enough:

diana_coman: if this is of the sort "I think I'm better off on my own" then it's quite easy to sort out too - set yourself a big goal and a deadline for it - e.g. "I'll make it to lordship on my own steam by April 2021 or I'll go on 1st of May 2021 on my knees to whoever will take me and do something useful with me"

It became clear to me that the "try my best and see what happens, on my own" approach was not going to get very far, at least given how far behind I was starting.

For the second cause, there was the question of Robinson. A day came where he definitively stated he'd be joining that evening. As darkness fell, I didn't see a join, and a phone call failed to connect, my imagination tended toward worst-case possibilities and I fired off some impassioned pleas by IRC. Sure enough he joined, and I had a prime situation to exercise that openness thing:

jfw: dorion: though we know it's but the first step, I'm proud of you for showing up here.
jfw: We've been stuck in some kind of a mutual encouragement yet avoidance-enablement pact.
jfw: I'd be witholding information not to say that I was crying like a little girl earlier, telling him over irc why he needs to join, and Now.
jfw: The neighbors are probably wondering who died; as far as I could tell it was gonna be either my 5-year best friends relationship with him, my potential relationships with ~everyone here, or a particularly stubborn bit of our own stupidity.

For the third, there was:

diana_coman: jfw: fwiw you seem to want rock-solid under your foot (and preferably a few hundred meters of it downwards too) before you even step anywhere; assuming that's true, it's like everything else, it comes with its own set of advantages and disadvantages, not a "wrong" or a "right" thing.
jfw: yes I am very much of a 'look before you leap' tendency.

On this count, remembering the imperative of asking any necessary questions, I had attempted to inquire into the philosophical foundations and nature of the Republic, but found I lacked the language or background to get very far with this; still, some pertinent principles had come out, including:

diana_coman: jfw: realise that there is no right/wrong thing by itself; it's only about doing whatever you do well or not.
diana_coman: jfw: for a core principle of mine (and I'd say not only mine but I don't really talk for others): focus on figuring things out for what they are rather than for what you'd want them to be/ find convenient to be
diana_coman: jfw: and at every juncture, choose the thing that needs to be done rather than your comfort

~ To be concluded ~

« Newer PostsOlder Posts »

Powered by MP-WP. Copyright Jacob Welsh.