Draft gbw-node frontend, part 1

Filed under: Bitcoin, Software — Jacob Welsh @ 01:18

The database schema for the "node" part of Gales Bitcoin Wallet covered, we now proceed to the frontend program that puts it to work: collecting data from bitcoind, parsing its various binary encodings and extracting something useful.

Source: draft1/ draft2/ (and the previous schema-node.sql).

You'd be well advised to read the downloaded thing prior to executing, especially since it's in an unsigned draft state. As for what's necessary to graduate to a vpatch I'd be ready to sign, my thinking is that it's this review and annotation process itself, plus whatever important changes come out of it, and the previously suggested schema tweaks (since changing that is the most obnoxious part once deployed).

At present there's not much of an installation process and the database is initialized manually. I'd suggest creating some directory to hold the two sources. Then from that directory:

$ chmod +x
$ mkdir ~/.gbw
$ sqlite3 ~/.gbw/db < schema-node.sql
$ ./ help

In preparing this code for publication I observed that I had continued by force of habit (and editor settings) with the Python style guidelines of four-space indents and some fixed line width limit, in opposition to Republican doctrine. I've attempted to clean it up such that line breaks occur only for good reasons, though I can't say I'm happy with how my browser wraps the long lines. And it's not like I expect the poor thing to know good indentation rules for every possible programming language now... wut do?!


We start with the usual Pythonistic pile of imports. The ready libraries are a big reason the language is hard to beat for getting things working quickly, and at the same time a dangerous temptation toward thinking you don't need to care what's inside them.

# J. Welsh, December 2019

from os import getenv, open as os_open, O_RDONLY, O_WRONLY, mkdir, mkfifo, read, write, close, stat
from stat import S_ISDIR, S_ISFIFO
from sys import argv, stdin, stdout, stderr, exit
from socket import socket
from threading import Thread, Event
from binascii import a2b_hex, b2a_hex
from base64 import b64encode
from struct import Struct
from hashlib import sha256 as _sha256
from decimal import Decimal
from inspect import getdoc
import errno
import signal
import string
import json
import sqlite3
from sqlite3 import IntegrityError

The above are all in the standard library, assuming they're enabled on your system. The ones that stick out like sore thumbs to me are threading and decimal; more on these to come.

As the comments say:

# Safety level: scanning stops this many blocks behind tip

# There's no provision for handling forks/reorgs. In the event of one deeper than CONFIRMS, a heavy workaround would be:
#   $ sqlite3 ~/.gbw/db
#   sqlite> DELETE FROM output;
#   sqlite> DELETE FROM input;
#   sqlite> DELETE FROM tx;
#   sqlite> .exit
#   $ gbw-node reset
#   $ gbw-node scan

At least a semi-automated and lighter-touch recovery procedure would certainly be nice there.

gbw_home = getenv('HOME') + '/.gbw'
bitcoin_conf_path = getenv('HOME') + '/.bitcoin/bitcoin.conf'

# Further knobs in main() for database tuning.
db = None

This Is The Database; Use It.

For reasons I don't quite recall (probably interpreting hashes as integers, combined with pointer type punning - an unportable C programming practice common in Windows-land), bitcoind ended up reversing byte order compared to the internal representation for hex display of certain things including transaction and block hashes. Thus we have "bytes to little-endian hex" wrappers.

b2lx = lambda b: b2a_hex(b[::-1])
lx2b = lambda x: a2b_hex(x)[::-1]

Not taking any chances with display of monetary amounts, a function to convert integer Satoshi values to fixed-point decimal BTC notation. The remainder/modulus operators have varying definitions between programming languages (sometimes even between implementations of the same language!) when it comes to negative inputs, so we bypass the question.

def format_coin(v):
	neg = False
	if v < 0:
		v = -v
		neg = True
	s = '%d.%08d' % divmod(v, 100000000)
	if neg:
		return '-' + s
	return s

Preloading and giving more intelligible names to some "struct" based byte-packing routines.

u16 = Struct('<H')
u32 = Struct('<I')
u64 = Struct('<Q')
s64 = Struct('<q')
unpack_u16 = u16.unpack
unpack_u32 = u32.unpack
unpack_u64 = u64.unpack
unpack_s64 = s64.unpack
unpack_header = Struct('<i32s32sIII').unpack
unpack_outpoint = Struct('<32sI').unpack

Some shorthand for hash functions.

def sha256(v):
	return _sha256(v).digest()

def sha256d(v):
	return _sha256(_sha256(v).digest()).digest()

An exception type to indicate certain "should not happen" database inconsistencies.

class Conflict(ValueError):

For reading a complete stream from a low-level file descriptor; experience has led me to be suspicious of Python's file objects.

def read_all(fd):
	parts = []
	while True:
		part = read(fd, 65536)
		if len(part) == 0:
	return ''.join(parts)

Ensuring needed filesystem objects exist.

def require_dir(path):
	except OSError, e:
		if e.errno != errno.EEXIST:
		if not S_ISDIR(stat(path).st_mode):
			die('not a directory: %r' % path)

def require_fifo(path):
	except OSError, e:
		if e.errno != errno.EEXIST:
		if not S_ISFIFO(stat(path).st_mode):
			die('not a fifo: %r' % path)

RPC client

Bitcoind uses a password-authenticated JSON-RPC protocol. I expect this is one of the more concise client implementations around.

class JSONRPCError(Exception):
	"Error returned in JSON-RPC response"

	def __init__(self, error):
		super(JSONRPCError, self).__init__(error['code'], error['message'])

	def __str__(self):
		return 'code: {}, message: {}'.format(*self.args)

Some of this code was cribbed from earlier experiments on my shelf. The fancy exception class above doesn't really look like my style; it may have hitchhiked from an outside JSON-RPC library.

The local bitcoin.conf is parsed to get the node's credentials. This is done lazily to avoid unnecessary error conditions for the many commands that won't be needing it.

bitcoin_conf = None
def require_conf():
	global bitcoin_conf
	if bitcoin_conf is None:
		bitcoin_conf = {}
		with open(bitcoin_conf_path) as f:
			for line in f:
				line = line.split('#', 1)[0].rstrip()
				if not line:
				k, v = line.split('=', 1)
				bitcoin_conf[k.strip()] = v.lstrip()

Side note: I detest that "global" keyword hack. It's "necessary" only because variable definition is conflated with mutation in the single "=" operator, and completely misses the case of a nested function setting a variable in an outer but not global scope. ("So they added 'nonlocal' in Python 3, solves your problem!!")

def rpc(method, *args):
	host = bitcoin_conf.get('rpcconnect', '')
	port = int(bitcoin_conf.get('rpcport', 8332))
	auth = 'Basic ' + b64encode('%s:%s' % (
		bitcoin_conf.get('rpcuser', ''),
		bitcoin_conf.get('rpcpassword', '')))
	payload = json.dumps({'method': method, 'params': args})
	headers = [
		('Host', host),
		('Content-Type', 'application/json'),
		('Content-Length', len(payload)),
		('Connection', 'close'),
		('Authorization', auth),
	msg = 'POST / HTTP/1.1\r\n%s\r\n\r\n%s' % ('\r\n'.join('%s: %s' % kv for kv in headers), payload)
	sock = socket()
	sock.connect((host, port))
	response = read_all(sock.fileno())
	headers, payload = response.split('\r\n\r\n', 1)
	r = json.loads(payload, parse_float=Decimal)
	if r['error'] is not None:
		raise JSONRPCError(r['error'])
	return r['result']

I could see removing the "parse_float=Decimal", and thus the corresponding import, as we won't be calling here any of the problematic interfaces that report monetary values as JSON numbers. But then, I'd also see value in one RPC client implementation that can just be copied for whatever use without hidden hazards.

Bitcoin data parsing

Now things might get interesting. To parse the serialized data structures in a manner similar to the C++ reference implementation and hopefully efficient besides, I used memory views, basically bounds-checking pointers.(i)

# "load" functions take a memoryview and return the object and number of bytes consumed.

def load_compactsize(v):
	# serialize.h WriteCompactSize
	size = ord(v[0])
	if size < 253:
		return size, 1
	elif size == 253:
		return unpack_u16(v[1:3])[0], 3
	elif size == 254:
		return unpack_u32(v[1:5])[0], 5
		return unpack_u64(v[1:9])[0], 9

def load_string(v):
	# serialize.h Serialize, std::basic_string and CScript overloads
	n, i = load_compactsize(v)
	return v[i:i+n].tobytes(), i+n

def vector_loader(load_element):
	# serialize.h Serialize_impl
	def load_vector(v):
		n, i = load_compactsize(v)
		r = [None]*n
		for elem in xrange(n):
			r[elem], delta = load_element(v[i:])
			i += delta
		return r, i
	return load_vector

def load_txin(v):
	# main.h CTxIn
	i = 36
	txid, pos = unpack_outpoint(v[:i])
	scriptsig, delta = load_string(v[i:])
	i += delta
	i += 4 # skipping sequence
	return (txid, pos, scriptsig), i

load_txins = vector_loader(load_txin)

def load_txout(v):
	# main.h CTxOut
	i = 8
	value, = unpack_s64(v[:i])
	scriptpubkey, delta = load_string(v[i:])
	return (value, scriptpubkey), i+delta

load_txouts = vector_loader(load_txout)

def load_transaction(v):
	# main.h CTransaction
	i = 4 # skipping version
	txins, delta = load_txins(v[i:])
	i += delta
	txouts, delta = load_txouts(v[i:])
	i += delta
	i += 4 # skipping locktime
	hash = sha256d(v[:i])
	return (hash, i, txins, txouts), i

load_transactions = vector_loader(load_transaction)

def load_block(v):
	# main.h CBlock
	i = 80
	head = v[:i]
	version, prev, root, time, target, nonce = unpack_header(head)
	hash = sha256d(head)
	txs, delta = load_transactions(v[i:])
	return (hash, prev, time, target, txs), i+delta

The code dig to come up with this magic for identifying standard pay-to-pubkey-hash outputs and extracting the enclosed addresses was ugly.

def out_script_address(s):
	if len(s) == 25 and s[:3] == '\x76\xA9\x14' and s[23:] == '\x88\xAC':
		return s[3:23]
	return None

To be continued.(ii)

Updated for errata.

  1. I'm just now noticing these were added in 2.7, ugh... sorry, 2.6 users. [^]
  2. My blog will be going on hiatus as far as new articles until early January. There's quite a ways to go on this file and I might not make it all the way through on this pass. If the suspense gnaws, you can always keep reading the source! [^]


Draft gbw-node schema

Filed under: Bitcoin, Software — Jacob Welsh @ 22:09

This series will present my initial implementation of a Bitcoin wallet built up from loosely-coupled parts ("simple droplets"). This follows from necessity as much as design philosophy, in that: 1) the only practical way to secure keys against remote compromise is with dedicated offline hardware; 2) additional online code is needed to support this; and 3) the existing bitcoind is a costly and treacherous thing to hack on. Further background on this work is at:

Today we will look at the data model for the awk-like part, which has shaped up as a line-mode command suite run on demand to collect validated block data, index and make accessible the parts needed to track funds and issue transactions.

Source: schema-node.sql.

Compared to a key-value store like BerkeleyDB as used by bitcoind, using SQL allows formalizing the structure and relationships in the data, working at a high level and enforcing integrity along several dimensions. Besides simplifying implementation, this means the human operator can bypass any limitations of the frontend software and work directly with the database if need be.

Because SQL in practice isn't quite as standard as one might like, one needs to pick either a specific implementation or some abstraction layer to target. While I've worked with MySQL, PostgreSQL and SQLite, I'm most familiar with the last, and figure it has some nice properties for this application: simple to install, widely available, without daemon processes, network listeners, user accounts and passwords to configure. Don't be fooled by the "lite": it's still a large item with fairly comprehensive and optimized SQL engine, claiming robust concurrency and durability properties (ACID transactions) and large, well-defined and tested limits. It includes a command-line shell sqlite3 for manual operation, dump and restore. You are of course welcome to try other implementations and report findings; I haven't used inherently SQLite-specific features like dynamic typing.

A "watched address" model is used such that only transaction outputs and inputs affecting Bitcoin addresses of interest are recorded. The benefit is much lower storage requirement than indexing the full blockchain; the cost is having to rescan after importing addresses with existing history.(i) Addresses are grouped into sets (tags) such that one node can serve many wallets or include decoy sets.


--- Gales Bitcoin Wallet: node (online component) schema
--- J. Welsh, December 2019
--- Dialect: SQLite (3.7.0 for WAL)

SQLite originally used a writeback rollback journaling approach, which is terribly slow for write-heavy uses, at least on mechanical HDDs lacking battery-backed cache, due to requiring at least two fsync operations per transaction. It provides no way to relax durability without sacrificing integrity guarantees, which would precisely suit the needs of the scan process. We'll instead use the newer Write-Ahead Logging mode; once activated, it's remembered in the database file.

PRAGMA journal_mode=WAL;


The tx table tracks confirmed Bitcoin transaction metadata. All "_id" fields are database-assigned integers for efficiency of storage and lookup; hashes are stored as raw bytes.

	tx_id        INTEGER PRIMARY KEY,
	hash         BLOB    NOT NULL,
	block_hash   BLOB    NOT NULL,
	block_height INTEGER NOT NULL,
	n            INTEGER NOT NULL, -- position in block
	comment      TEXT,
	size         INTEGER NOT NULL,
	fee          INTEGER
CREATE UNIQUE INDEX i_tx_hash ON tx(hash);
CREATE UNIQUE INDEX i_tx_height_n ON tx(block_height, n);

The purist might note that this table isn't entirely normalized since many-to-one block information is stored in two fields; but then, there can also be upsides to avoiding excessive joins. It might be nice to track more block metadata anyway since it's not that big and bitcoind does it quite opaquely.

Comment and fee fields are unused at present: computing fees, while desirable, would seem to require a full transaction index; comments don't seem to fit with the concept of minimizing private information stored by the node.

Transactions can be found either by their 256-bit hash or the more concise block coordinates.

Outputs and inputs

The transaction output is the main structure of interest, holding a fixed parcel of spendable or spent coin.

The REFERENCES clause, besides making relationships explicit, creates a foreign key constraint. Unfortunately SQLite doesn't enforce these by default for compatibility reasons; it must be enabled by the application on each connection. To ensure the sqlite3 shell gets it too, I include in my ~/.sqliterc a PRAGMA foreign_keys=ON;, as well as a .headers on and .mode quote (one per line) for better handling the blobs.

	n          INTEGER NOT NULL, -- position in output vector
	address_id INTEGER NOT NULL REFERENCES address,
	value      INTEGER NOT NULL,

SQLite integers take up to 8 bytes as needed, thus can safely represent Satoshi values up to the maximum BTC that will ever exist (1e8 x 21e6 = 2.1e15; 263 > 1018).

	spent      INTEGER REFERENCES input(input_id),

An initial mistake was keeping the spent reference on the input side, following the raw transaction structure. The trouble with this is there's no efficient way to find unspent outputs: "the rows in this table that aren't referenced from another table" are necessarily not contained in that table's index! So instead we keep the link on this side, allowing NULL, and update the row when an input is seen that spends it.

	flags      TEXT
CREATE UNIQUE INDEX i_output_txid_n ON output(tx_id, n);
CREATE INDEX        i_output_addrid ON output(address_id);
CREATE UNIQUE INDEX i_output_spent  ON output(spent);

The flags field is presently unused: the idea was to track local state bits like seen or acknowledged.

	n          INTEGER NOT NULL -- position in input vector
CREATE UNIQUE INDEX i_input_txid_n ON input(tx_id, n);

Having a separate table for inputs is looking pointless now that it's a mere one-to-one reference from output.spent... ah, the joys of schema refactoring.

Addresses and tags

CREATE TABLE address (
	address    BLOB NOT NULL
CREATE UNIQUE INDEX i_address_address ON address(address);

Like transaction and block hashes, addresses are represented in raw byte form as opposed to Base58Check-encoded. This makes scanning faster but importing and formatting for display slower. "Version" is assumed to be zero, meaning P2SH / "3-addresses" aren't a thing.

CREATE UNIQUE INDEX i_tag_name ON tag(name);

Unlike the "accounts" seen in bitcoind, addresses can have multiple tags.

CREATE TABLE address_tag (
	address_id INTEGER NOT NULL REFERENCES address,
	PRIMARY KEY (address_id, tag_id)
CREATE INDEX i_addrtag_tag ON address_tag(tag_id);

Like many databases, SQLite uses B-tree storage. One implication is that in the case of composite keys (those spanning multiple fields as seen here), a dedicated index is not necessary for the first field as B-trees provide efficient prefix search. Subsequent fields however may benefit from them.


Finally, a "singleton" table to track scan state. A more extensible approach to global variables would be textual name and value fields, but I dunno, I like the explicit-ness of mapping variables to column rather than row.



Stay tuned for the code that puts it all to work. To run it you'll be needing Python 2 (2.7 tested) built with the sqlite3 module, the standalone sqlite3 shell (as loading the schema is manual for now), and of course, a Bitcoin node.

  1. A test scan of 607`849 blocks on 1`102 addresses finding 8`929`351 transactions took 13 hours producing a 1.8GB database file. [^]


Review of polarbeard_add_sendrawtransaction_rpc.vpatch

Filed under: Bitcoin, Software — Jacob Welsh @ 20:14

This patch to The Real Bitcoin was published somewhere around 2016 by someone who as I gather showed up to dump some fairly verbose and undocumented patches then didn't stick around to advocate for them or keep them updated for the meanwhile changing tree.

One of them fills a still-unmet requirement of any split wallet so I'm reviewing it in hopes of saving some time on having to redo from scratch. Full patch is here.

+Value sendrawtransaction(const Array& params, bool fHelp)
+    if (fHelp || params.size() < 1 || params.size() > 1)
+        throw runtime_error(
+            "sendrawtransaction <hex string>\n"
+            "Submits raw transaction (serialized, hex-encoded) to local node and network.");
+    vector<unsigned char> txData(ParseHex(params[0].get_str()));
+    CDataStream ssData(txData, SER_NETWORK, VERSION);

Above we see the three-argument form of the CDataStream constructor(i) with explicit arguments matching the defaults for the nTypeIn and nVersionIn parameters.

Some documentation of what if anything these parameters are for would be nice (though not a deficiency of this patch). For now it seems to me a reasonable demand that the defaults never change, or at any rate that the costs of such a change be borne by he who makes it. Thus their inclusion here would be redundant, and not really the helpful kind of redundancy at that i.e. the kind that catches mistakes. I had omitted them in my getrawtransaction patch.

+    CTransaction tx;
+    try {
+        ssData >> tx;
+    }
+    catch (std::exception &e) {
+        throw JSONRPCError(-22, "tx decode failed");
+    }

The -22 isn't seen elsewhere in the tree, but matches the number in the PRB implementation and named RPC_DESERIALIZATION_ERROR.

The condition of excess data in the stream after the transaction does not appear to be detected, which I figure would be nice to have.

+    CTxDB txdb("r");
+    uint256 txHash = tx.GetHash();
+    uint256 hashBlock = 0;
+    if (TransactionSeen(txHash, hashBlock))
+    {
+        throw JSONRPCError(-5, string("tx already ") +
+            (hashBlock != 0 ?
+                string("included in block ") + hashBlock.ToString() :
+                "in memory pool"));
+    }

Here it gets iffy in my opinion. Just because my node sees the transaction, whether in block or mempool, it doesn't follow that other nodes do too. What if I want to re-broadcast?

Secondarily, the use of zero as a sentinel seems mildly unhygienic as it's a legal hash value, though I suppose if a preimage were found then the whole system would be screwed anyway. Along with the use of argument mutation, I see it more as an indication that too much implicit behavior is being packed into this new TransactionSeen interface, such that its (ONE!) caller needs a hack to do what it actually wants.

+    bool fMissingInputs = false;
+    if (!tx.AcceptToMemoryPool(txdb, true, &fMissingInputs) || fMissingInputs)
+    {
+        throw JSONRPCError(-22, string("tx rejected") +
+            (fMissingInputs ? ", missing inputs" : ""));
+    }

Perhaps a nitpick but why bother having error codes if you don't use them to distinguish things? Is the human text part of API now?

Looking into AcceptToMemoryPool, it doesn't actually do anything with that fMissingInputs besides initializing to false!

By the earlier argument it's also questionable whether that second fCheckInputs parameter should be true. It activates what looks like assorted anti-spam code of dubious value in this context.

+    SyncWithWallets(tx, NULL, true);
+    CInv inv(MSG_TX, txHash);
+    RelayMessage(inv, tx);
+    return txHash.GetHex();

Oh yes, there's code to support handling wallets plural, not that anything is done with it.

The CInv boilerplate is demanded by the poor abstraction of RelayMessage. That function includes some 15-minute expiration mechanism whose purpose and workings aren't readily apparent to me.

Generally, it seems awkward that all these steps are needed here; shouldn't there be a function somewhere that handles transactions received from the network, that we'd just reuse? I'm almost afraid to look.

+// return wether a given transaction is found on mempool or already included in a block
+bool TransactionSeen(const uint256 &hash, uint256 &hashBlock)
+    CRITICAL_BLOCK(cs_mapTransactions)
+        if (mapTransactions.count(hash))
+            return true;
+    CTransaction tx;
+    CTxDB txdb("r");
+    CTxIndex txindex;
+    if (tx.ReadFromDisk(txdb, COutPoint(hash, 0), txindex))
+    {
+        CBlock block;
+        if (block.ReadFromDisk(txindex.pos.nFile, txindex.pos.nBlockPos, false))
+            hashBlock = block.GetHash();
+        return true;
+    }
+    return false;

All the objects getting passed around and mutated here are painful to follow. This appears to do what it says, though I'd much rather be rid of it.

Painfully apparent in the debts of the codebase is documentation of locking protocol. Getting this sort of thing wrong results in non-deterministic failures, either data corruption or deadlock depending on the direction of the mistake.

 // make sure all wallets know about the given transaction, in the given block
-void static SyncWithWallets(const CTransaction& tx, const CBlock* pblock = NULL, bool fUpdate = false)
+void SyncWithWallets(const CTransaction& tx, const CBlock* pblock, bool fUpdate)
     BOOST_FOREACH(CWallet* pwallet, setpwalletRegistered)
         pwallet->AddToWalletIfInvolvingMe(tx, pblock, fUpdate);

Parameter defaults move to the declaration; I gather C++ puts the burden of setting these on the generated calling code, which seems sensible for a statically-compiled language.

 void RegisterWallet(CWallet* pwalletIn);
 void UnregisterWallet(CWallet* pwalletIn);
+void SyncWithWallets(const CTransaction& tx, const CBlock* pblock = NULL, bool fUpdate = false);
+bool TransactionSeen(const uint256 &hash, uint256 &hashBlock);
 bool ProcessBlock(CNode* pfrom, CBlock* pblock);
 bool CheckDiskSpace(uint64 nAdditionalBytes=0);
 FILE* OpenBlockFile(unsigned int nFile, unsigned int nBlockPos, const char* pszMode="rb");

Perhaps polarbeard had other patches that make use of this TransactionSeen. I'd rather this sort of patch not introduce globally-visible interfaces unnecessarily.

TransactionSeen is introduced as a globally-visible interface so that the underlying mapTransactions object stays static, in contrast with my more liberal approach.

In conclusion, the patch appears safe to apply and should do for now for the sake of testing my offline wallet, which might help inform any changes needed here. After some testing I might even end up signing it, though I'd much prefer something simpler and more clearly defined.

  1. And here I long for a self-hosted and accurately cross-referenced i.e. language- and context-aware source browser, but thanks to jurov for this approximate one. [^]


Uruguay parte 4: el turismo es trabajo

Filed under: Historia, Politikos, Vita — Jacob Welsh @ 16:11

Concluding from Parte 3.

On Sunday my host generously treated me to a five or six hour walking tour of the city, starting at the Feria de Tristan Narvajo, a weekly flea market named after the street it centers on in the Cordon neighborhood, where we browsed a while, then headed west along Avenida 18 de Julio through Centro to Ciudad Vieja (old town) and the seaport, then back and northeast to El Obelisco and finally back southeast to Pocitos. His extensive time on the pavement and accumulated trove of information became all the more apparent. He figured we did about ten miles on foot on this part of the day alone and noted it was certainly more than whatever else passes for tours these days.

The previous night he had made his rounds near the site of previous post-election destruction and observed light tank, helicopter and searchlight presence, apparently effective at deterring a third weekly recurrence.

Once I had managed to rouse myself, grab a bite and walk to his place, we proceeded by cab to be sure to catch the Feria. As at the airport, the orange-and-white themed cab system is far more orderly than Panama's, with taxi stands (this one unoccupied presumably due to being Sunday, but we didn't have to wait long for one to stop by), metering, divider between front and back seats and a better kept (or at least more consistently so) fleet from what I saw. Cost is higher but still reasonable.



Exiting Pocitos, looking northbound down the Boulevard General Artigas if I'm not mistaken.


Things get a bit less polished in Cordon.


Arriving at the entrance to the Feria.


My illustrious guide.


The Feria had all manner of things from varyingly finished woodwork...


to outright junk...


to antiques such as typewriters...




rotary phones, some sporting the Antel brand which has enjoyed its monopoly from the early days...




kerosene lamps...


mechanical calculators...


and more guns...


to clothing, plants, birds...



to cameras new and old...


to some nice amethyst, one of the few items of local origin...


and more generic polished rocks.


Not pictured were stands with fresh produce, empanadas and similar prepared bites, jewelry, books (mainly well-used and yellowing paperbacks from what I saw), and paraphernalia for Mate consumption.(i) Something to do with cannabis too, its being fully legalized now: some "art" but I don't recall whether any consumables.


Exiting the Feria onto 18 de Julio where we find an Universidad de la Republica. The stand in front with red handwriting was promoting a hunger strike (better pictured).


Some live percussion on the corner...


Then a national library...


where Cervantes, or perhaps his ingenious Don, reminds us that he who reads much and walks much, sees much and knows much.


Many shops here were closed for Sunday, leaving a good view of the graffiti on the roll-down metal covers protecting the glass within.


McD's will use old buildings if needed to get that corner real estate.


The local Communists and friends, known as the Frente Amplio, fly an inverted Russian flag.





Aaron takes note of a new message scrawled on the Ministry of Social Destruction by someone who doesn't want her rights pissed on. Er, stepped on.


The Bronze Statue of Man on Horse series begins.



The Intendencia de Montevideo, a government office and one of the taller buildings with Mirador public observatory from which we were going to "see the entire city (if not up close)"; sadly it was closed that day, for reasons the posted officers didn't know.





A particularly elegant cupola, I thought, with neighboring buildings of entirely differing character.



The lamppost builletin board is alive and well.


I would have liked to see if the arcade machines were as old as the signage evokes.


Park with some epic struggle depicted, fountain not operational...


But side fountains were.


A look toward the Palacio Legislativo, suggesting this is from Plaza Fabini.


Four blocks further and we reach the sizable Plaza Independencia, marking the transition from Centro to Ciudad Vieja.





Presidential office gets a nice building. Whatever's to the right, not so much.


Nor do the offices look much more pleasant inside.







The biggest statue is for national hero Artigas, who died in exile. I'm told that ashes are kept below and might even be his.


Old city gate.


There's this rainbow-filled park-of-sorts tucked around a corner.


Pictures and sign read "Trans Law Now", "Fight for Diversity", "Constructing the future with love".


Back to park-like parks; some National Party colors.



First glimpse of the port down a street in Ciudad Vieja.


It's a mix of the very old and the new.



Instituto Nacional de Colonizacion: I'm told there's an active homesteading program for unused land in the interior.



Caribaldi, chief of naval forces of the Republic, 1842-1848.


Across the water to the Antel tower, tallest building in the country at 158m if the Internet is to be believed.


Port facilities.


Holding pen for containers, and a cruise ship.



View across the water to what might be a refinery, past some wreckage I hadn't noticed at the time.




Radar, presumably for maritime traffic control.


Fortress atop a distant hill.



There was this nice semi-indoor plaza, the one exception to the uniform restaurant cost thing.


The town has its run-down parts.


It's a peninsula, with water visible in both directions.



I hadn't recalled seeing an evergreen yet.


As-yet unidentified man on horse. And sheep.



Juniper if I'm not mistaken.



Itau and BBVA branches.


Some fine woodwork on the Palacio Santos, home of the ministry of foreign relations.


En esta plaza, el dia 24 de abril de 1925 el fisico Albert Einstein mantuvo un dialogo con el filosofo uruguayo Carlos Vaz Ferreira.

Homenaje del Consejo de Educacion Tecnico Profesional (UTU) y el Gobierno Departamental de Montevideo 30 de junio de 2005, a los 100 años de la Teoria de la Relatividad

If the bench were traveling half the speed of light, would it still fit within the chains?




In Ciudad Vieja and on the way back we were approached two or three times by beggars: bold, persistent, sad stories at the ready, and ungrateful when I did once offer a coin against my better judgement.


Back at Cordon, the Feria was packing up. Below, one of the air-cooled VWs that provide an inexpensive transport option as many were produced nearby (Brazil?) and imports otherwise face steep tariffs.


Classic Plymouth soft-top, apparently in decent shape, a rare sight here.


We made our way to this freestanding radio tower, marking an approximate center of the city and visible from many directions due to height.


Mega-flag and mega-cross; Aaron tells me there were three when the Pope made his one and only visit.


The promised obelisk: A los constituyentes de 1830. Bystanders included at the base for scale.


Heading back to Pocitos; there's a hospital complex behind the row of trees. Perhaps you'll recall this boulevard from the cab ride.


Petrobras, soon to be leaving the country.


Pole painted in Commie colors; private school in the background.




Church of Christ, Scientist. No sign of Scientologists though.


This time we dined at a Club de la Papa Frita. Determined to get in some beach time, we went home to change and, at least in my case, rest up a bit from the day's mileage.

A waxing gibbous hangs above the Rambla just before dusk.


Hitting the beach, with sunset peeking through the distant showers.

Health hazards are sometimes flagged for the water e.g. due to city runoff after a rain or cyanobacterial blooms, but no problems this time.




We walked the surf for a ways then back. Among topics that came up were DDoS attacks; Aaron reflected on how they had varied depending on time and context, comparing things to a shallow beach where waves can travel far and steeper beach where they break.



180+ saved shots yet only one of the cameraman; I oughta take more initiative about requesting these.


A look at the hotel room prior to rolling out Monday morning.


The balconies were a nice touch, at least on this side that faces street rather than concrete wall.


Kitchenette stocked with plates, glasses and cutlery; works for leftovers if not quite for cooking.


Bathroom with separate bidet, which might be the first time I'd seen this type IRL. (Panama does this hose on the wall thing, cheaper I suppose.)


At center, the World Trade Center tower that once housed a datacenter, the failure of which set in motion the whole chain of events that led me to this spot.



The cab driver tuned to a radio program on which there happened to be some extended chatter about Bitcoin y Blockchain en Uruguay. I didn't make out much but it sounded like someone promoting a conference.







Chilling at the gate, this time with plenty of time to spare. The wait for check-in was reasonable; security was noticeably less obnoxious just for that seemingly small difference of leaving shoes and coats alone.


Eat enough of those Uruguayan portions of mozzarella and you too could transition from underwear model to pear.


Perhaps airplane photos are a cliche but I still enjoy them...




Likely the Hipodromo de Moroñas, from a glance at the map.





Unlike Panama, Uruguay supplied real butter with the airplane dinner. Aaron also tells me that unlike Europe, they produce consistently unadulterated olive oil.

Near Peru, some Andes poking through the clouds.






Approaching Panama.


The decent beaches here are a ways out from the city.



Clockwise from bottom: the causeway islands of Amador; the old town Casco Viejo with bypass on the water, the coastal Avenida Balboa, Punta Paitilla.


Mouth of the canal.


Punta Paitilla and the manmade Punta Pacifica, with recently added islands.


On arrival I had what seemed like at least a kilometer to walk from gate to immigration; the moving walkways along the way were out of service just as they'd been my last several trips. Despite walking briskly and getting in the short line for residents, by the time I got through the bags from my flight had already been unloaded from the belt and stood in a row. I suppose they think this is helpful, or necessary to make room; it does tend to confuse the newbies.

I faced another vague list of things requiring customs declaration, citing all sorts of old and recent laws and decrees, and decided I could argue that my stuff qualified for exemption if need be while declaring might draw additional scrutiny. It probably would have been about the same either way. As I'd mentioned, the servers attracted some curiosity on the X-ray belt, but no further trouble once it was clarified they were my own, not TVs, used, relevant to occupation, or whatever other checksums the agent was looking for.

Congratulations: we've made it to the end. I hope you've enjoyed the tour of the tour; I certainly enjoyed the thing itself and the recounting. Till next time, que les vaya bien!

  1. Hot infusion of Yerba Mate, the local caffeinated drink of choice and apparently something of a ritual involving special gourds and straws, though I didn't get to witness or try. [^]


Uruguay parte 3: encontrar, destruir y proteger

Filed under: Hardware, Vita — Jacob Welsh @ 17:47

Continued from Parte 2.

Upon meeting at Aaron's castle, we chatted briefly then got down to the business of picking out my purchases from the pile and loading the rackmount items onto his trusty fold-up hand truck. I passed on the pile of cables, nuts and bolts as these could be procured easily enough anywhere and the weight adds up quickly. We gathered a complete set of Supermicro rails and he demonstrated their mechanical robustness compared to the more complex and jam-prone interlocks on the Dell rails. He also lent me a bathroom scale and set of finer screwdrivers to help with my packing job.

Then we rolled down the Rambla with the iron, by a route perhaps more direct and certainly with less curbs to navigate than I'd come by. I stupidly neglected to get pics of the intrepid couriers. We got some curious looks but no questions; Aaron explained that while the locals are friendly toward strangers, they're culturally inhibited from initiating contact.


The trip could have been done by cab, but with the pleasant weather, reasonable distance and helpful host there was really no need. On arrival to the hotel the concierge caught our attention; I was half-expecting some lecture about how we were using them the wrong way, but rather she was helpfully directing us to the parking entrance where there'd be fewer stairs to navigate.

As expected based on measurements, the first server fit in hardshell suitcase with just enough room to spare for some padding. Behold the sweet Opterons, RAID card and true random number generators (which turned out to be mounted by advanced technology of double-stick tape).


The second server fit likewise in softshell.


The highest-value item, whether by weight, volume or replacement cost, was a Russian teapot. Well actually, the contents of a box that once held a Russian teapot: another 18 of the aforementioned TRNGs.(i)


Aaron headed home while I worked on packing. He asked if I wanted to help with his duty of data destruction on former customers' unclaimed storage media. Of course I did, not least of all for the chance to learn from a pro (I learned he once worked for a data recovery firm). We reconvened and went out in search of a pentalobe - apparently the latest wave of screwdriver DRM - for SSD disassembly. We found none at three or four places, but his shop was well stocked otherwise with implements of creative destruction; the aluminum cases proved susceptible to prying and ripping with pliers.

The biggest surprise was the hard disk platters, which we'd both expected to be made of glass, that merely dented rather than shattering on hammer impact. We roughed up the surfaces as best we could with sandpaper then left them to cook outdoors in a mild chemical bath of tomato sauce (with an excellent smelling touch of basil!), Coca-Cola and salt (fluoridated at that, apparently a local thing).

The hammer also proved ineffective for the SSD boards, mostly just ripping the plastic bags we tried to contain them with. The pliers again prevailed as the boards would bend or crack after some flexing, exposing the thin IC packages which could be snapped into flakes.

This time I had stupidly left my camera back at the hotel so perhaps Aaron can provide destruction pics, but here's the setup the next day with chip remains soaking in their own salt bath, to be dispersed across multiple dumpsters.


Having worked up a good appetite we headed to Expreso Pocitos for dinner. It had recently been named the best in the area, mostly on account of the competition going out of business. Apparently the restaurant landscape in Soviet Uruguay is quite flat, with little variation in menu or prices. Their idea of a basic pizza is just crust and sauce; if you order with mozzarella they figure that means you want it drowning in the stuff. Portions are consistently generous. Water is only served bottled, despite the tap water being safe. Service at least here was not remotely attentive.


Here as generally I tried to ask my host as many pertinent questions as I could think of, and enjoyed the conversations. General topics included the country and city, their economy and politics, our own Republic, ourselves, and computing. I observed his ability to pick out interesting details from large topics and integrate information from differing fields. One topic I recall might be summarized as the usability hazards of insufficient context in text-only computing interfaces.

After dinner I resolved to get my packing done for real so we could enjoy Sunday for tourism. A particular difficulty was the Ubiquiti Networks EdgeSwitch, a 48-port beast with 500W supply for Power-over-Ethernet capability which weighed in at 6 kg. I didn't have an immediate need for such a thing but had picked it up cheap since I was making the trip anyway. My first thought was to move a server PSU to carry-on, after a pic to note the wiring and RAM sticks I removed to get at its screws:


This didn't quite save enough weight though, so next was opening the switch, which in a strange laptop-like style required removing a great many small screws.

Remember, kids: if you see an open power supply like this with mains-voltage capacitors that may have been recently powered, short out their terminals e.g. using a screwdriver with well insulated handle prior to handling, as they can pack a nasty shock.


As there was nothing heavy inside I could remove yet re-package adequately for carry-on, and the switch didn't seem worth the marginal cost in overweight fees, I opted to leave it behind for either the next guy or the dumpster divers ("nothing is ever really thrown away here", says Aaron.)

I disassembled the server rails a bit further, allowing me to bundle them up to minimize chances of damage. I had brought my roll of the proper kind of tape for these jobs, plain old Scotch magic tape.


Fully mummified rails:


One server got bubble wrap for its first layer; the other (not pictured) got my old mattress pad.


Some sound-absorbing foam pads I happened to have at home proved excellent secondary padding and space filler.


Some $3 pillows made fine space filler to finish it off.


To be continued with proper tourism.

  1. jfw: asciilifeform: do you know the origin of this zavarochnyi chainik box (if my translit serves)?
    jfw: otherwise, safe travels and enjoy that real olive oil, sadly I didn't manage to grab any
    asciilifeform: jfw: it's a teapot ( the kind where make concentrated tea and then can make quicker at teatime by pouring in cup + hot water )
    asciilifeform: standard item in household of tea maniacs
    asciilifeform: jfw: how didja end up with a ru teapot ?
    jfw: that's what I'm wondering! 'tis what the FGs were stowed in. BingoBoingo?
    jfw: (I didn't end up with the teapot, merely the box.)
    BingoBoingo: jfw: I acquired the teapot at Tienda Inglesa
    BingoBoingo: That's the box it came it.
    asciilifeform: pretty great
    BingoBoingo: It just happened to be the right size for FUCKGOAT packing
    jfw: BingoBoingo: ha, neat. RU teapot from English shop in Uruguay.
    BingoBoingo: Layering happens [^]


Uruguay parte 2: llegada y primeras vistas

Filed under: Historia, Politikos, Vita — Jacob Welsh @ 20:12

Continued from Parte 1.

My text having overtaken the start of photography, I'll have to backtrack a bit to Montevideo's Aeropuerto Internactional de Carrasco (MVD) which was looking quite shiny and new. Bag claim (evidently I misremembered: there were three on the international side, though just the one active):


Aduanas (customs). That bienes de ingreso/egreso temporal would seem vague enough to cover just about anything if they felt like it; fortunately they didn't give a second glance (perhaps even first glance) to my scandalous screwdriver and packing materials.


Free at last, but not quite home.


I would have picked up a local SIM but the booth was closed for the night. It turned out my Panama SIM worked on roaming, at least briefly, which it hadn't in the US.


The bit of the world that is me thanks Uruguay for the welcome.


In contrast to Panama, there was no crowd of taxi syndicate reps soliciting eagerly. Instead it's an orderly racket; you go to the taxi counter and arrange a ride with prepayment and receipt. Having been warned the cab would be around US $55, I held out for the $13 shuttle bus, taking the wait time to replace that stolen sunscreen, collect my thoughts and decompress a bit. I found myself tired but alert and relieved.

The only exterior shot I managed of the airport, so it'll have to do:


Some Himpton by Halton thing near the airport with well-lit street:


First shuttle stop was at the Motel Bahamas:


A pleasant nighttime drive down the coast and another one or two stops later and I'd made it to my destination in the relatively nice Pocitos neighborhood.



The buildings here cap out around 10-12 stories due to zoning. Most are mixed-use, with shops at ground level and apartments above. My first impression of the area compared to most of Panama City was of something older (turns out many buildings date to the 1930's if I recall), more stable (as opposed to wreckage and new construction everywhere), cleaner, and far more pedestrian friendly (wide and not entirely treacherous sidewalks). Aaron pointed out that this does not apply to the whole city, with outlying neighborhoods ranging from more typically LatAm to outright favela (though these not walled off as in Brazil).


Right around the block was an ANCAP gas station with rare 24-hour convenience store and deli, which served me for breakfasts, rather dreadful espresso (they couldn't believe I didn't want sugar, which probably says it all), and a printed map so as to navigate free of any "mobile device" nonsense.

First daytime views of the coastal Rambla, supporting vehicle, bike and pedestrian traffic and beach access, as I made my way to meet my host.



Oh yes, the street signs serve advertising; it does seem to help keep them in good shape.


One of the larger mini-parks opposite the beach, near the Avenida Brasil.


Battery scooters for hire.


"Por la vida y la convivencia" : La Policia seem to like their mottos...


There's a lazy tourism option.


I have no idea. It didn't seem animate.


"Orgullosamente blanco" - "proudly white" - referring, I gather, to the party colors of the recently victorious Partido Nacional rather than something racial.


Either Ave. Brasil or Espana, two thoroughfares that converge at the coast.


Corner florists seem to be thriving...


Corner locksmiths not so much.


Heading inland a bit; some gym/yoga place.



Apparently they don't need no education at the Center of Foreign Tongues. Aaron tells me the buildings generally don't get repainted much because maintenance work is taxed the same as new construction.


Perhaps this would have been the spot for a better coffee.



There's minimal piped natural gas infrastructure (as in Panama, though there it's often provided building-wide and refueled by tanker trucks).


Pizzeria Trouville

Another florist, and some of the typical sycamores lining the streets.


To be continued with rare electronics and proper tourism.


Una visita a la Republica Oriental del Uruguay, parte 1

Filed under: Politikos, Vita — Jacob Welsh @ 18:22

Having bought some of the remains of the historic but sadly liquidated Bitcoin firms No Such lAbs and Pizarro ISP, and with expected overseas shipping costs being comparable to a personal courier run, I seized the opportunity for some travel and networking. It's been a success on all three fronts: retrieving the gear, getting a taste of Montevideo, and meeting and spending some quality time with Aaron Rogier aka BingoBoingo, whom I'd previously known mainly as the humorously grandiose voice of Qntra and a thoughtful contributor to IRC discussions; I found him to demonstrate the same insight in person and be quite likable besides. I made it a three-night stay to allow one full day for the hauling and packing and one for tourism.(i)

My biggest mess-up of the trip as I see it was not allowing enough time for my initial departure from the "Hub of the Americas", Panama's recently expanded Tocumen International Airport - for which I'm starting to develop a hearty loathing - or the time to get there, my previous departures here having been either in the wee hours or from locations with better toll road access. I got stuck in the check-in cattle queue for the better part of an hour.(ii) By the time my turn came, I was informed that due to late check-in my bags would be subject to "voluntary separation" and might end up on the next flight. Since apparently I couldn't find out whether they made it until arrival, I worried and contemplated my options on the flight. At security, while not subjected to the gate-side mandatory gropings reserved for the US-bound, there were still US-inspired theatrics like shoe removal and inspecting my carry-on for liquids, confiscating my over-100ml sunscreen. Serves me right for being such a terrorist, huh.

Things went much smoother from there; immigration in Montevideo was a breeze at least for chip-enabled passport holders, there were no kilometers to walk to the airport's one baggage claim, and my bags had made it just fine. Having been warned about the pricey airport taxi service, I elected to wait for a shuttle, which departed once the next flight had dumped enough passengers to form a group. On exiting the airport (around 2am local time) I was welcomed by the delightfully cool, spring-like air: always a nice thing after months in the tropics, though my skin and nose didn't adjust to the dryness too well.

All the travel intel Aaron had given that I had chance to verify proved accurate, and the Punta Trouville hotel he recommended was the perfect fit for my needs: budget but clean, functional, well located and with 24-hour service. Power outlets and money proved easier than anticipated. The hotel had multi-format outlets; it's just as well I came prepared with adapters, as Aaron said those can be flaky, though they worked for me. While there are cambios all over for changing currency with around 4% spread, I never ended up needing one as the airport transit and the merchants I tried were all equipped and even glad to take my specie (well, USD) and give change in pesos Uruguashos; the local currency sees the sort of inflation that gets automatically priced into yearly contracts.

To be continued (and with photos).

  1. Not ideal for really getting to know a place, but I already had a longer holiday coming up and lots to get done before it. [^]
  2. The "web check-in" line turned out to move faster; I can't see any good reason as it doesn't save much time at the counter: you still need to get docs checked, bags weighed and tagged, and any overage paid. The main reason as far as I could tell was simply that they'd allocated more agents there and didn't rebalance until the line was entirely exhausted. [^]


Basic getrawtransaction patch proposal for bitcoind

Filed under: Bitcoin, Software — Jacob Welsh @ 17:35

I present a bitcoind patch, of my own authorship, for consideration: a basic but functional getrawtransaction RPC command. I expect the need for it is clear: if it's your node then you shouldn't accept any sort of "I'm sorry Dave, I'm afraid I can't do that" especially regarding the data it already has handy.

To speak plainly about some deficits, this time not my own: the Real Bitcoin project presently rests on some shaky foundations. Despite apparently intricate efforts, a Keccak based vtree has seen little progress, in that the original patch signers besides mod6 have not signed on to the regrinds of their work, and some recommended patches have not been reground at all, updated for the very necessary manifest spec, or included in the mainline collection. Further, the sorry state of the inherited code such as magic numbers everywhere has seen little improvement. Perhaps I will have to take up some of these burdens in time; for now I'll leave it as an entreaty to the elder hands to please find a way to make it happen!

The patch

As in the original introduced somewhere around PRB 0.7.0, this command takes a transaction ID (256 bits of hex), searches first the node's own memory pool then the confirmed transaction database (blkindex.dat) and returns a hex dump of the encoded transaction. Unlike the original it does not support a "verbose" option to give a JSON representation of the data. This task seems to me better suited to an external tool, but I could see including it here if the implementation is concise and obviously correct.

Backporting the original was not possible due to the many intervening changes, though I did consult it to confirm I hadn't missed anything important and matched its numeric error code.

Based on the overall idea in the V version control system of building yggdrasil, I'm breaking from one of the project's prior naming conventions by including "bitcoin" but not the author in the patch name; the author is still recorded in the enclosed manifest file. Due to the problems noted earlier with the prior patch tree it's also not a proper vpatch yet.

Download: bitcoin_getrawtransaction.draft.patch.

To try this you will need to:

  1. Perform a press to asciilifeform_whogaveblox.vpatch;
  2. Manually apply mod6_phexdigit_fix.vpatch (which could be missed otherwise due to lacking a manifest entry);
  3. Manually apply the patch in question.

In detail

I added a manifest entry for the phexdigit fix, to make its inclusion explicit:

--- a/bitcoin/manifest.txt
+++ b/bitcoin/manifest.txt
@@ -28,3 +28,5 @@
 542413 asciilifeform_aggressive_pushgetblocks asciilifeform Issue PushGetBlocks command to any peer that issues 'version' command
 542413 mod6_excise_hash_truncation mod6 Regrind of ben_vulpes original; removes truncation of hashes printed to TRB log file
 543661 asciilifeform_whogaveblox asciilifeform Record the origin of every incoming candidate block (whether accepted or rejected)
+606789 mod6_phexdigit_fix mod6 Fix decoding LUT which wrongly accepted certain invalid characters as hex
+606789 bitcoin_getrawtransaction jfw Add RPC to get transactions from memory pool or database (hex only)

The RPC itself is about as simple as it gets in this codebase. First we try the mempool, as this should be fast and may contain unconfirmed transactions.(i)

--- a/bitcoin/src/bitcoinrpc.cpp
+++ b/bitcoin/src/bitcoinrpc.cpp
@@ -1351,7 +1351,31 @@
     return entry;

+Value getrawtransaction(const Array& params, bool fHelp)
+    if (fHelp || params.size() != 1)
+        throw runtime_error(
+            "getrawtransaction <txid>\n"
+            "Get hex serialization of <txid> from memory pool or database.");

+    uint256 hash;
+    map<uint256, CTransaction>::iterator it;
+    CTransaction tx;
+    CDataStream ssTx;
+    hash.SetHex(params[0].get_str());
+    it = mapTransactions.find(hash);
+    if (it != mapTransactions.end())
+        tx = it->second;

Functions everywhere have to open their own database connections - though some have the luxury of getting one passed in - which then implies a whole caching mechanism so as not to be horribly inefficient. Odin knows why there couldn't just be one global (or at least per-thread) "This Is The Database; Use It" object.

+    else {
+        CTxDB txdb("r");
+        if (!txdb.ReadDiskTx(hash, tx))
+            throw JSONRPCError(-5, "Transaction not found in memory pool or database.");
+    }
+    ssTx << tx;
+    return HexStr(ssTx.begin(), ssTx.end());
 Value backupwallet(const Array& params, bool fHelp)
     if (fHelp || params.size() != 1)

Wiring the function into the RPC dispatch table (I don't recall how I chose where to insert it, as the list was already non-alphabetical; probably based on where it seemed sensible in the help listing):

@@ -1865,6 +1889,7 @@
     make_pair("getreceivedbyaccount",   &getreceivedbyaccount),
     make_pair("listreceivedbyaddress",  &listreceivedbyaddress),
     make_pair("listreceivedbyaccount",  &listreceivedbyaccount),
+    make_pair("getrawtransaction",      &getrawtransaction),
     make_pair("backupwallet",           &backupwallet),
     make_pair("keypoolrefill",          &keypoolrefill),
     make_pair("walletpassphrase",       &walletpassphrase),

The mempool object now needs to be visible between compilation units. I suggest doing a grep to verify this introduces no name conflicts.

--- a/bitcoin/src/main.cpp
+++ b/bitcoin/src/main.cpp
@@ -26,7 +26,7 @@

 CCriticalSection cs_main;

-static map<uint256, CTransaction> mapTransactions;
+map<uint256, CTransaction> mapTransactions;
 CCriticalSection cs_mapTransactions;
 unsigned int nTransactionsUpdated = 0;
 map<COutPoint, CInPoint> mapNextTx;
--- a/bitcoin/src/main.h
+++ b/bitcoin/src/main.h
@@ -46,6 +46,7 @@

 extern CCriticalSection cs_main;
+extern std::map<uint256, CTransaction> mapTransactions;
 extern std::map<uint256, CBlockIndex*> mapBlockIndex;
 extern uint256 hashGenesisBlock;
 extern CBlockIndex* pindexGenesisBlock;

I tested that it builds, successfully fetches transactions from both mempool and database, and returns the expected errors for missing argument or transaction not found. It does accept invalid hex strings, perhaps a flaw in that SetHex method. I've been running the patch in production since around August 10th of this year.

  1. The original cause for my writing the patch was a stuck transaction that wasn't getting relayed to miners or block explorers for unknown reasons. Upon fishing the raw tx from the mempool and submitting it to one such site, a useful error was finally obtained identifying the problem as the S-value normalization mess; the -lows option provided a workaround, after double-spending to self for safety, which was a whole other pain. [^]


keksum, a Keccak implementation in C as standalone Unix utility: genesis

Filed under: Software — Jacob Welsh @ 17:36

I produced a Keccak implementation in May 2019, through about one week of intensive study and hacking. It builds on some techniques and routines I'd been developing for small, self-contained C programs on Unix, whereby the standard I/O library is thrown out in favor of a minimalistic interface fitting the needs of the program, requiring for portability only the system call wrappers, which generally have direct translations to assembly language. The approach ensures that system errors are detected without requiring effort by calling code and leaves no uncertainty regarding signal behavior, flushing of buffers or integer overflow. The resulting binary here (musl/amd64, static, unstripped) weighs in at 19K and contains not so much as a "malloc".

Regarding the permutation and sponge construction themselves, I found the Keccak reference quite comprehensible; the only topics I needed to brush up on were Linear Feedback Shift Registers and the polynomial algebra used to describe them.

Having previously written what ended up as possibly the world's slowest hash functions in an interpreted language, I was itching to make something fast for once: not pouring vast efforts into optimization but at least avoiding obvious slowdowns. I wish I could say I got it all right on the first try, or that I identified all mistakes through careful re-reading, but not quite:

jfw: well my keccak proggy hashes; now to look for some test vectors
jfw: sure enough I botched it:
jfw: mathematically, (x-1) mod y is equivalent to (x+y-1) mod y
jfw: (pretty much the definition of mod...)
jfw: but if x is an unsigned type -- which I made it, because it's used as an array index and those better be nonnegative -- the subtraction wraps *mod 2*
jfw: er, mod power-of-two
jfw: the arrays in question are indexed mod 5, which being coprime to any power of 2, gives a decidedly different result from if x were a signed or mathematical integer.
jfw: (in another part of the code I had in fact anticipated this.)

The arithmetic in question sure looked correct on the surface, so I tracked this down by adding fine-grained test probes to compare each step of the permutation against provided data.(i) Further contemplation lead me to see that the mod-5 operations were entirely invariant in the permutation's input and thus amenable to a lookup table without introducing a timing side-channel.

I did a full re-read a month after writing; the only mistake found was a comment with off-by-one range description.

Capacity and output length parameters are user-selectable; I chose a default of 512 bits for both, as explained in the code:

Sponge capacity is an upper bound on security level because the permutation is readily inverted if its full output is known. Beyond that, its relationship to actual security is not clear to me (or perhaps anyone); some margin of safety seems prudent. In the FIPS202 parameters, capacity under 512 is seen only in "SHAKE128" and none of the "SHA3" fixed-width hashes. The EuCrypt default is 256 (bitrate 1344); I have not found a discussion of this choice.

I should have piped up and asked about this at the time; though I was still a WoT non-entity living in the shadows, it happens that blog comments are one of the easier ways to get started in the grand conversation, and certainly if you have something interesting to say or ask.

The past being what it is, I will ask now: do any mathematical minds in the readership have input on what constitutes a "good" choice of capacity and why? If as I'd been thinking it's more than 256 to be at least as secure as SHA3, it would seem to suggest a need to regrind the existing Keccak vpatches or otherwise deal with a multiplicity of standards.

A compiler with 64-bit integer support is required; there is no dependence on machine byte order. The little-endian convention is used for interpreting bytes as the bit strings the permutation is defined on. I believe the code to be timing-invariant with respect to potentially secret bits, with the exception of output hex encoding. This would be good to fix. The other main deficiency I see is lack of a working "-c" option to verify provided hashes.

Finally, some basic performance numbers, from a modest Core 2 @2.26GHz, 3072K cache:

$ dd if=/dev/zero bs=1048576 count=100 | keksum
100+0 records in
100+0 records out
104857600 bytes (100.0MB) copied, 2.656227 seconds, 37.6MB/s

$ dd if=/dev/urandom bs=1048576 count=100 > rand
$ dd if=rand bs=1048576 count=100 | keksum
100+0 records in
100+0 records out
104857600 bytes (100.0MB) copied, 2.648968 seconds, 37.8MB/s

Download vpatches

  • genesis (seals: jfw)
  • keksum_genesis (seals: jfw) - redone to include project name as prefix in patch name and change default capacity to 256.
  • keksum_genesis_3 (seals: jfw diana_coman) - redone to include project name as prefix in patch name, change default capacity to 256, update self-test invocation including fixing on BSD-likes, and avoid false success in Makefile.
  1. The spiffy modular types in Ada would have prevented this mistake entirely, though not without some cost either in runtime performance or compiler complexity (and thus potential bugs...), I would imagine. [^]


Keccak background

Filed under: Bitcoin, Historia, Software — Jacob Welsh @ 18:52

"Keccak" is a cryptographic hash function, or rather, some primitives for constructing such functions in a desired size and shape, of relatively recent design as these things go. It was brought to the attention of the forum in early 2016 in the context of contemplating changes to the Bitcoin protocol,(i) (ii) (iii) and subsequently differentiated from SHA3.(iv)

Compared to the prevailing standards at the time - mostly variants on the MD4 concept, processing blocks of input through an iterated compression function - Keccak is based on a large pseudorandom permutation (1600 bits, though the spec also defines smaller variants). As this is readily invertible, the desired "one-way" property is provided by a "sponge construction", mixing in blocks of input and extracting output while iterating the permutation and keeping some number of its bits secret as internal state. This number is called the capacity (or by complement the rate, the two summing to the permutation bit width) and can be tuned for the desired balance of security and computational intensity. The construction can take unlimited input, or produce unlimited output as a kind of stream cipher.(v)

I started out in 2017 playing with a C implementation found in the wild, supposedly a "readable and compact" version written by the original team. With some cleanup I got it into a state that could be described as compact, but I couldn't get very far in reading it, at least without having first digested the spec. And it had the unfortunate limitation of requiring the full input and output to exist in memory, no streaming. My confidence as an applied cryptographer was growing and I soon implemented a number of classical hash functions, but set Keccak aside as not being an immediate necessity. Meanwhile, Diana Coman produced and incrementally published a very nice and documented reference implementation in Ada, which was adopted for use in V and soon became non-optional.

While I was well convinced by the Republican rationale for Ada, I was much less keen on introducing GNAT, the flagship implementation, into my environment. It was a million-plus-line-of-code beast that I wouldn't stand a chance to ever really understand; making matters worse, it was a "Thompsonism", a circular dependency requiring existing binaries in order to build from source and thus dubiously "open source" at all. While I already depended on one such thing - the C compiler - I was hoping to somehow keep this to ONE thing, or at least ensure a way to work with the crucial V on existing machines without pulling all this in.

Stay tuned for the result.

  1. mircea_popescu: actually i wouldn't go to war over keccak.
    mircea_popescu: letting bitfury & friends eat 100mn in unrecoupable engineering costs would provide exactly the correct lesson as to what it's a good idea to say and when it's a good idea to shut the fuck up and toe the line.


  2. The necessary prerequisite for any change to the Bitcoin protocol [^]

  3. mircea_popescu: << at least it wasn;t fucking developed by teh nsa.
    assbot: Logged on 01-02-2016 19:29:18; ascii_butugychag: ;;later tell mircea_popescu in what sense is adoptinc keccak a rejection of usg standards? it was actually adopted as sha3...
    mircea_popescu: as far as we know. whatevs. minor point.
    ascii_butugychag: btw between that thread and now i went and read the keccak spec
    ascii_butugychag: it is mighty spiffy.
    ascii_butugychag: accordionizes to size.
    mircea_popescu: :)
    mircea_popescu: i don't need to explain what i meant by not finite then ?
    ascii_butugychag: aha.
    ascii_butugychag: other hashes also accept infinite bits but they eat where they shit.
    mircea_popescu: quite.
    mircea_popescu: and mind that while in no means do i propose this is "Asic resistant", from a designer perspective you must appreciate i'm giving you a fun job to do.
    mircea_popescu: at least therer's that.
    mircea_popescu: always make sure everyone's having fun.
    ascii_butugychag: quite! nobody will be plagiarizing old verilog from fpga docs to bake this one.
    ascii_butugychag: very asian-resistant.
    ascii_butugychag: which is a mega-plus.


  4. asciilifeform: holyshit the original keccak www is gone
    asciilifeform: replaced with some horrorshow
    asciilifeform: ada code -- gone
    asciilifeform: fortunately still on my hdd
    asciilifeform: check this out, now forwards to buncha tards
    asciilifeform: < original
    shinohai: Notice that happened after declared their spec
    asciilifeform: shinohai: not immediately , iirc was still intact last yr
    asciilifeform: incidentally shinohai keccak != usg.sha3
    asciilifeform: they adopted ~particular params~ of keccak as the new national whatever
    asciilifeform: orig is ~family~ of functions.
    asciilifeform: see also << since 'unhappened' article
    asciilifeform: ' The SHA-3 version of Keccak being proposed appears to provide essentially the same level of security guarantees as SHA-2, its predecessor. If we are going to develop a next generation hash, there certainly should be standardized versions that provide a higher security level than the older hash functions! NIST, in the original call for submissions, specifically asked for four versions in each submission, with at least two that would
    asciilifeform: be stronger than what was currently available, so it's hard to understand this post-competition weakening.'
    asciilifeform: didjaknow.
    asciilifeform: notice how 'everyone' nao thinks 'oh, keccak? that's called sha3 nao' [^]
  5. Since state is still finite, output will of course repeat eventually; one would hope this cycle length approaches that order of 21600. [^]
Older Posts »

Powered by MP-WP. Copyright Jacob Welsh.