An organization may already have a data governance policy in place. It may already have a document for data inventory, its classification, its source systems, and its respective data owners. Yet the organization may still suffer from poor data governance, causing low trust in data. Why? Because those things may not be operationally connected.

The policy document may say sensitive data must be protected, but classification is not linked to access control to be seamlessly enforced. The governance committee may approve standards, but data assets get created faster than they are documented. The spreadsheet says a dataset exists, but analysts cannot find it, verify its lineage, or know whether it is safe to use. And data owners may be named in the spreadsheet, but incidents or any downstream concerns may not be routed back to the owners. Worst, there are no data owners.

The book Data Governance: The Definitive Guide argues that trust in data increases when (1) someone is accountable for the data, (2) the data are easily observable, and (3) the data are secure from unauthorized access, misuse, or corruption.

That’s Accountability, Observability, and Security. They encapsulates the capabilities the many tools and artifacts that is commonplace in data management and governance practices. Examples includes data contracts, metadata management, data catalog, data lineage, data quality tests, access controls, encryption, etc. Some contributes to more than one. Data lineage and data catalog, for example, not only improves observability by improving data discoverability; they also improve accountability by making data asset ownership more explicit and granular, closer to the logical data structure.

One way to model the interaction between these three capabilities is a self-reinforcing governance loop that continually builds trust in a systemic way:

That is to say, the relationship between these capabilities is cyclical: assigning ownership, with the proper incentives, establishes stronger accountability. It motivates data owners — through checks and balances such as data contracts and SLAs — to be accountable to the curation of metadata and improve documentation, making data easier to discover and observe, increasing observability. It also exposes a weak ownership structure if data issues are rampant. Better observability, in turn, enables stronger security controls and faster detection of misuse or quality issues. Those incidents can then be traced back to owners who should be held accountable — such as by institutional policy — reinforcing responsibilities and driving further improvements to observability and security. Thus, each cycle increases trust in the data.

In the world of digital data abundance, software plays a more important role in supporting this system, seamlessly connecting the three important capabilities in data governance, as I noted in DFCE 2025: Regulation, Innovation, and Friction. Yet paperwork may never be extinct. It’s still required for audits, legal accountability, or C-suite sign-offs, but it won’t be enough to operationalize data governance at scale. Beyond paperwork, however, software helps in making it easier to do the right thing with data, and difficult to do the wrong thing.

The Governance Loop on Software

Generally, data catalogs, used for metadata management, focuses on the observability capabilities (by ingesting technical metadata and curating them) and accountability (by assigning data owners to the ingested metadata). Security, meanwhile, is naturally integrated with the AI and data analytics platform, where data analytics or data engineering works — together with policy enforcements — happens.

To appreciate software’s role in data governance beyond bureaucratic paperwork, in this post I’ll use solutions such as OpenMetadata (as the data catalog) and Snowflake (as the AI and data analytics platform) as examples. Note, though, this isn’t a recommendation of the two solutions. Another powerful competitor to Snowflake is Databricks; and a good curated list of data catalog solutions can be found in the awesome-data-catalogs repository on GitHub.

Accountability

Documenting asset ownership usually means the following: Either an owner or an asset was identified first. An agreement then has to be made on who is accountable for the asset — an asset which should be valuable (otherwise why bother?) — to some stakeholders. Ownership usually lies where it is in his or her interest to keep the asset running, reliable, and sustainable (lest stakeholders can’t extract value and blame the owner).

In the corporate world, this is usually structured by some KPI to incentivize the asset owner to do the “right” thing with the asset. So far, this kind of incentive structure is straightforward and common. Examples abound, such as KPIs on incident resolution timeliness, data privacy compliance rating, external audit rating score, etc.

But generic KPIs are not enough. The data assets must also be measured against their intended use, so that dataset used for operational monitoring should have different definition of “good” data quality from dataset used in sales forecasts, for example. The owners are therefore not only accountable for IT-centric issues; equally important is accounting for data consumer needs — to ensure the data is fit-for-purpose and fit-for-use — which corresponds more strongly to the trust placed in data. Indeed, trust in data is not purely absolute (e.g., no duplicated records); it is also relative to the domain context and intended use (e.g., the records satisfies use case requirements).

In the world of data abundance, however, this effort of establishing the link between data owners (and their responsibilities) with their data assets has become cumbersome. The reason can be explained briefly in terms of the three V’s of big data: Volume, Velocity, and Variety1.

Increasing use of digital workflows increases the rate of growth of organization’s data footprint. The sheer amount of data volume being generated per day — or even minutes — may overwhelm unprepared organizations answerable to regulatory policies. At the same time, the rate — or velocity — at which data are being created, captured, and processed also increases, each of which proliferates new data assets that stakeholders rely on. Problems and confusion arise when this rate of data creation and usage is faster than the organization’s capability to document, manage, and control its data assets. Finally, different use cases necessitate the adoption of different variety of data storage formats and data processing technologies, from simple fixed-schedule batch-processing of tabular formats to more complicated event-driven unstructured data processing. Combined, the large volume, high velocity, and wide variety of data assets can compound accountability problems.

This brings us to another related problem: How do you define the data ownership boundaries? Where does the line begin that delineates one owner’s assets from the others, so that there’s no ownership overlap? And how do you even begin to draw that line2? The complexity of these questions is further compounded by the three V’s above. High velocity means data ownership boundaries change frequently. Ownership then needs to be frequently reviewed and updated. That means the inventory to keep track of the increasingly growing data assets with wide varieties needs to be kept up-to-date as frequently, too.

You can’t beat this data governance problem in a world of data abundance with more bureaucracy and paperwork. Since you can’t manage what you can’t observe — and what you want to manage scales out quickly in every dimension — we need to increase our capability to observe at scale, both organization-wise (as seen with data ownership concerns) and technology-wise (along the volume, velocity, and variety dimensions).

Observability

The oldest observability mechanism in the book involves writing paper documentation of your data assets and manually performing inventory bookkeeping. That usually translates to cumbersome yearly or bi-annual (because any faster will be exhausting) cross-team efforts of surveying and interviewing different teams to understand the data landscape: What are the databases that are currently running? Which database contains sales data? Which ones contains PII data? Which database gets its data from which other databases? If a specific table is corrupted, which downstream stakeholders are affected? As you can imagine, this approach to observability means such information gets stale quickly, and doesn’t scale well when data footprints grow larger and quicker over the years. Answering each question above may take days or weeks, depending on your three V’s.

Despite the increase in data footprints, it’s not uncommon to see digital things being inventoried manually in an Excel sheets or Word documents. That should change. With regulatory requirements, for example, data catalog plays an important role for data observability.

To remain compliant with PDPO in Brunei, for example, an artifact called Data Inventory Map was introduced. The fundamental reason for its introduction is, of course, to ensure organizations and regulatory bodies have the capacity to observe data assets and their use. The following is an illustration provided by AITI — the supervisory authority behind PDPO:

It captures important metadata about data assets containing personal data: the purpose of collection, the retention period, where they’re stored, the legal basis behind the collection, etc. The problem with the Data Inventory Map isn’t the artifact itself; it’s about keeping a live, granular, and accurate metadata of the data assets.

This is where data governance platforms like OpenMetadata or DataHub comes in handy. They can automatically discover, extract, and ingest the technical metadata —logical and structural information about your data assets, like table schema, view definitions, and lineage — in your data landscape3.

They generally work as follows. First, once they have read-only access to your data systems (e.g., Oracle and SQL Server RDBMS, or Snowflake data warehouse), they can extract the metadata information from the source systems. For example, on SQL Server, you can find this in the information schema views; on an Oracle database, the data catalog views; on Snowflake, the information schema (aka “data dictionary”) views. The good news is that instead of querying these views yourself — which usually isn’t meant for human consumption — platforms like OpenMetadata already have their own built-in technical metadata ingestion tools (called connectors) that reads from the catalog views and parses them further to enrich the metadata in the governance platforms. The caveat is that if there’s no built-in connector for a source system you want to ingest, you’d have to create a custom one.

Once the data governance platform ingests the technical metadata, the accountability-observability-security activities in the loop above become more streamlined and agile. Data assets inventory can be updated quickly in a matter of seconds or minutes, instead of weeks or months, greatly reducing the data inventory staleness problem. The bottleneck to governance is no longer on extracting information about data in systems, but extracting tacit knowledge in people.

The technical inventory of data assets is also more reliable and accurate because it’s inferred directly from source. Storing the inventory in the platform allows it to be indexed for search via fuzzy words matching or tags, which improves discoverability.

The technical metadata can further be curated with information useful for operationalizing data governance, such as data owners for each data assets, the business glossaries, classifications and tagging, and data quality status. While these tasks are traditionally done by human data stewards, future context-aware AI agents might help stewards speeds up tedious data curation exercises.

In short, the link between the governance contexts of the data — that is, the purpose of collection, legal basis, and intended recipients, etc. — and the technical data assets should be unambiguous and clear. Without data catalog, the link gets blurry, giving rise to ambiguity about data ownership and the actual scope of policy implementation.

With a more precise technical metadata structure, metadata management such as data labeling or classification can become less ambiguous, giving way to transparent traceability of data use and enabling precise policy enforcement, and thus — as the loop suggests — enabling better data security.

Security

If you can’t manage what you can’t observe, it goes without saying that you can’t secure what you can’t manage. In the context of data governance, securing data goes beyond the prevention of data exfiltration and data misuse. It also extends to include data corruption: the degradation of data integrity, such as inaccurate data updates, or simply technical data consistency failures (remember ACID?). Data corruption hence deserves a similar treatment as the former two, because violating any of them can cause considerable harm to the organization.

All these can be done by internal or external actors, whether it was done with malicious intent or otherwise. This is why observability is the key enabler for securing data. With the metadata ingested and properly curated, we know what data exists across our data landscape and how it is classified or labeled (e.g., PII, Confidential, Public, etc.). This observability capability allows organizations to better plan and be more proactive to establish the appropriate level of monitoring and controls to right data with the right security controls.

Let’s expand on how observability through metadata management enables security across the three security controls introduced above. The first is the most obvious one, which is to prevent data exfiltration by malicious actors. This means you want to safeguard access to your data systems, such as your enterprise data warehouse, to the right people with the right level of privilege. To ensure the control is in place, not only you should be able to know who can access your data systems, it is also important to monitor their usage to alert suspicious activities (a sudden overnight spike in database queries by a single user outside office hour should raise some suspicion); usage are, after all, metadata that can be ingested by data governance platforms. Naturally, the access control safeguard happens at the data systems level (e.g., SQL Server, Oracle, and Snowflake) — not the data governance platform such as OpenMetadata or DataHub. How data governance platform complements access control, however, is by making it easier for organizations to design controls commensurate with the classification of data attached to the ingested metadata.

The second control is relatively harder to manage, which is to control against data misuse. For example, you may have submitted an online survey form containing your personal email — with your consent in a checkbox — to allow the researchers to contact you for further inquiries about the research survey. However, privacy laws may prohibit the researchers from using your email for commercial purposes — as that would have been a misuse of your personal data (and a betrayal to your trust!)

Purpose-based access control is harder to manage because of low traceability of data usage from data access provision. Mostly, you’ll find the clues about the purpose the data access — and hence its usage — in some email trails, such as emails about data engineers requesting read-only access to a specific table. Indeed, in most cases, purposes are not a first-class citizen in data platforms. Often, they are only weakly inferred from the combination of usernames and roles, and the grants applied to them.

To tackle this issue, OpenMetadata allows us to represent purpose as metadata with the use of tags or classification attached to catalogued data assets. This can be made even tighter on the data systems level: On Snowflake, for example, access to data is restricted by roles defined not by job description (e.g., SALES_ANALYST), but also by purpose-specific roles (e.g., PURPOSE_REVENUE_REPORTING, PURPOSE_OPERATIONS_MONITORING). These purpose roles can then be combined with row-level access policies, masking policies, and other constraints so that the same data is exposed differently depending on the approved purpose of use.

The third control on data corruption, is one that’s often overlooked in the context of data security. Corrupted data may impair decision-making too, and, in the case of AI/ML models, makes them unreliable. This can be damaging to the organization. Again, observability helps. To tackle data corruption issues, organizations should catch data errors as early as possible, ideally within a defined SLA commensurate with the criticality of the data asset. This is usually done by running queries or scripts — often on schedule — that tests data assets for the expected data profile or quality. This can be as simple as rigid rule-based checks, such as looking for blank values, or as complex as regression analysis using machine learning models.

OpenMetadata, for example, has features that allow data stewards to define and run data quality assertions from its web UI. Test failures notify the stewards — or whoever is interested to subscribe to the notification — when quality assertion fail. If the data asset catalogued in OpenMetadata has an owner (as it should!) stakeholders would know who is accountable to move their team to investigate and fix the reported data issue. As discussed above, this is only possible because of accountability that drives the organization to this level of increased observability.

Remediation that follows the violation of the controls above feeds into accountability, as discussed earlier, in the form of a governance loop. We’ve seen how observability enables security across the three controls, thus increasing security capability. These controls force organizations to answer data ownership scrutiny. Who defines the quality metrics? How do you guarantee SLAs? Are your data consumer needs met? These questions, at least in theory, reinforce better accountability, and in turn, their incentive to do better on observability and security.

Conclusion

Data governance is ultimately a systems problem whose objective is to increase trust in data: Trust that the policy intents are applied where and when they should be applied; trust that data is discoverable, understandable, and fit for intended use; trust that sensitive data is protected from misuse, exfiltration, or corruption; and trust that someone is accountable when something breaks.

By modeling this problem using three underlying reinforcing capabilities —Accountability, Observability, and Security — we can reason and explain why their connectedness is important for operationalizing data governance to increase trust. Without closing the connection gap between them, ambiguity rises, accountability becomes blurry, and conformance checks become more costly to conduct. As we’ve seen, data catalog, by automatically ingesting technical metadata, reduces ambiguity that arises when the data asset inventory does not reliably capture (if at all) granular technical metadata information.

Indeed, data governance succeeds when the system — an amalgamation of people, process, and technology — makes doing the right thing with data easier than doing the wrong thing.

Subscribe now

All my new code will be closed-source from now on. I’ve contributed millions of lines of carefully written OSS code over the past decade, spent thousands of hours helping other people. If you want to use my libraries (1M+ downloads/month) in the future, you have to pay.

I made good money funneling people through my OSS and being recognized as expert in several fields. This was entirely based on HUMANS knowing and seeing me by USING and INTERACTING with my code. No humans will ever read my docs again when coding agents do it in seconds. Nobody will even know it’s me who built it.

Look at Tailwind: 75 million downloads/month, more popular than ever, revenue down 80%, docs traffic down 40%, 75% of engineering team laid off. Someone submitted a PR to add LLM-optimized docs and Wathan had to decline - optimizing for agents accelerates his business’s death. He’s being asked to build the infrastructure for his own obsolescence.

Two of the most common OSS business models:

• Open Core: Give away the library, sell premium once you reach critical mass (Tailwind UI, Prisma Accelerate, Supabase Cloud...)

• Expertise Moat: Be THE expert in your library - consulting gigs, speaking, higher salary

Tailwind just proved the first one is dying. Agents bypass the documentation funnel. They don’t see your premium tier. Every project relying on docs-to-premium conversion will face the same pressure: Prisma, Drizzle, MikroORM, Strapi, and many more.

The core insight: OSS monetization was always about attention. Human eyeballs on your docs, brand, expertise. That attention has literally moved into attention layers. Your docs trained the models that now make visiting you unnecessary. Human attention paid. Artificial attention doesn’t.

Some OSS will keep going - wealthy devs doing it for fun or education. That’s not a system, that’s charity. Most popular OSS runs on economic incentives. Destroy them, they stop playing. Why go closed-source? When the monetization funnel is broken, you move payment to the only point that still exists: access. OSS gave away access hoping to monetize attention downstream. Agents broke downstream. Closed-source gates access directly. The final irony: OSS trained the models now killing it. We built our own replacement.

My prediction: a new marketplace emerges, built for agents. Want your agent to use Tailwind? Prisma? Pay per access. Libraries become APIs with meters. The old model: free code -> human attention -> monetization. The new model: pay at the gate or your agent doesn’t get in.

I don’t agree with his implication that OSS is now left to “wealthy devs doing it for fun or education.” I do agree, though, that eventually there will have to be some kind of pay per access feature to libraries/content for this to be economically sustainable (see Agentic Web). Despite that, I think open-source won’t die out, even without money in the picture, because:

• Programmers will still want to code to contribute to open-source software (with or without the help of AI)

• We still benefit immensely from the compounding value of using and looking at the same code. This is because of something called Inverse Tragedy of the Commons. Open-source won’t die out; at least not the ones that the commons care about.

How the Open-Source Machine Works

To understand why open-source won’t die out, and will still be sustainable despite AI, let’s understand how open-source software survive and self-perpetuates across many industries.

I find Eric S. Raymond’s (aka ESR) essays very helpful in explaining why free and open-source works in great detail—which I think can be summarized as a virtuous cycle.

First, for individual programmers, there’s that itch. A programmer will work on the things that they care about the most; if they’re doing it for free, they’re doing it for themselves. Such was how Linux was conceived, and many others that preceded it or followed it. This is especially true when there’s no other (free) software that exists—or (free) software that exists but is deemed not good enough—that solves the programmer’s problem. To quote ESR:

Every good work of software starts by scratching a developer’s personal itch.

Now that a programmer has scratched the itch, either by forking an existing (dead) software, making incremental contributions, or by creating a new one from scratch (pun not intended), what keeps them going? Here lies the question of incentives. If they plan to sell the support or maintenance service (e.g., as with enterprise Red Hat Linux), then the incentive is obvious: money. But what about the software that exists without a business model? If we eliminate money from the equation, what’s the driving force for programmers to maintain and write code, besides the joy of it?

To answer this question, ESR first described two different ways humans commonly organize themselves to allocate scarce resources: command hierarchy and market economy. As the name suggests, command hierarchy follows a top-down approach of central planning that allocates resources for everyone in the organization. Perhaps the most popular example of command hierarchy in history is also a textbook example of why it doesn’t scale well and is bound to crumble: the communism in 20th century Soviet Union failed to deliver its promise—instead of a rich Utopia for the masses, it delivered chronic shortages, breadlines, and eventual economic collapse. The clear winning model for allocating scarce resources is that there should be little to no planning at all; instead, in a market economy, individual transactions are driven by supply and demand. This scales better, as prices become the signal for resource allocation, operated over the exchange of money for goods and services.

Yet neither of these economic models explains the behavior of programmers in the open-source ecosystem: While command hierarchy and market economy are economic models that can explain the allocation of scarce resources, it doesn’t fit well in a world of abundance! Unlike the resources managed in a command hierarchy model or market economy model, software is a resource that can be duplicated with virtually zero cost, at a very large scale and speed, given the abundance of compute, storage, and network bandwidths (thanks to the 90s dot-com bubbles). Abundance, ESR writes, makes “command relationships difficult to sustain and exchange relationships an almost pointless game.”

There’s a third model, however, that can help explain the allocation of resources in the open-source ecosystem characterized by abundance: the gift economy. In contrast to a market economy, where goods and services are exchanged for money, a gift economy operates differently. ESR explained the gift economy through an anthropological lens, using Lockean theory of property ownership as an analogy to explain the motivation behind the ownership and “homesteading” of open-source projects. Here’s how he explained it:

On a frontier, where land exists that has never had an owner, one can acquire ownership by homesteading, mixing one’s labor with the unowned land, fencing it, and defending one’s title. ... A piece of land that has become derelict in this way may be claimed by adverse possession—one moves in, improves it, and defends title as if homesteading. This theory, like hacker customs, evolved organically in a context where central authority was weak or nonexistent.

If we think of the open-source ecosystem as a frontier—ungoverned, with no central authority assigning ownership—it follows that hackers observe the same customs that Lockean theory describes: claim it through labor, maintain it, defend your title. Let a project go unmaintained, and someone else will fork it:

The Lockean logic of custom suggests strongly that open-source hackers observe the customs they do in order to defend some kind of expected return from their effort. The return must be more significant than the effort of homesteading projects, the cost of maintaining version histories that document “chain of title”, and the time cost of making public notifications and waiting before taking adverse possession of an orphaned project.

In other words, open-source software maintainers expect something in return, or a net benefit, for their efforts, even if it isn’t directly about money.

When the exchange of goods and services becomes pointless (everyone has access to all goods and services in the world), monetary exchange currency becomes worthless. ESR argues that the principal currency in a rich society blessed with abundance is reputation:

... prestige is a good way (and in a pure gift economy, the only way) to attract attention and cooperation from others. If one is well known for generosity, intelligence, fair dealing, leadership ability, or other good qualities, it becomes much easier to persuade other people that they will gain by association with you.

Obviously, if your gift is crap, the reciprocal is a crappy reputation. So there has to be some value associated with the open-source software that’s being maintained or built that gives the programmers their expected returns. The higher the value of the software, the better your prestige or status; the better you can make it when you enter the market economy or command hierarchy.

This is another property of open-source software that makes it different from traditional resources: Not only is it abundant (i.e., not scarce), but it also increases in value the more you use or consume it. The other thing I know that has this property is knowledge, which is befitting, since source code can be thought of as an encoding of someone’s or an organization’s knowledge.

ESR called this phenomena Inverse Tragedy of the Commons, after the popular concept called Tragedy of the Commons:

Over every attempt to explain cooperative behavior there looms the shadow of Garrett Hardin’s “Tragedy of the Commons”. Hardin famously asks us to imagine a green held in common by a village of peasants, who graze their cattle there. But grazing degrades the commons, tearing up grass and leaving muddy patches, which re-grow their cover only slowly. If there is no agreed-upon (and enforced!) policy to allocate grazing rights that prevents overgrazing, all parties’ incentives push them to run as many cattle as quickly as possible, trying to extract maximum value before the commons degrades into a sea of mud.

...

[W]idespread use of open-source software tends to increase its value, as users fold in their own fixes and features (code patches). In this inverse commons, the grass grows taller when it’s grazed upon.

The Inverse Tragedy of the Commons is what completes the open-source virtuous cycle loop. Somewhere down the line there’ll be some disgruntled programmer who uses the commons, creates a patch or adds new features because he has an itch. While doing that, he’s also doing it for himself, to earn reputation, and for self-satisfaction. And the open-source software (the commons) only grows taller.

Subscribe now

AI in the Open-Source Machine

If intelligence is one of the ingredients that fuels the open-source machinery and produces free software, then the programmers are vehicles of the intelligence that transforms ideas to runnable software. Yet while intelligence is scarce, intelligent programmers that are willing to write free software are even scarcer!

And then AI comes along, which sort of commoditize intelligence, and suddenly the scarce intelligence becomes abundant. I’ve tried Claude’s Opus 4.5, and so have many others including many veterans in the industry, and there’s no denying that AI will effectively be a huge part of the intelligence that creates software.

What this means to our open-source machinery cycle described above is the following: The cost of scratching an itch is cheaper, either because it takes less time to substantiate code, or because of lower skill ceiling. Whether this is a good thing or a bad thing for the overall quality of open-source source codes, the barrier to entry to scratch an itch is lower. Using the Lockean theory analogy above, one might think that the cost of reaping the benefits of homesteading an open-source project is now cheaper, and thus the return is higher (i.e., it is easier to gain reputation). Unfortunately, that’s not what happens.

A consequence of the lower barrier to entry, however, is that it devalues reputation. Or put in terms of currency, the value of reputation is being inflated. But not all reputation inflates equally: What AI commoditize is the execution—substantiation of code or working the CLI at breakneck speed. What remains scarce is the judgement: identifying which itch is worth scratching, making sound architectural decisions, and stewarding open-source projects over time (i.e., collaborating with some randoms on the Internet, establishing a hacking subculture, etc.)

This is like the barrel vs. the ammunition analogy. From my post Of Kind Chess and Wicked Programming:

This doesn’t mean I completely disregard the notion of AI replacing human programmers in the future. The key difference between the AI we’re using right now and the one that Jensen Huang portends is in their capacity (or lack thereof) to be a manager instead of the individual contributor; to be an architect instead of the builder; or, to borrow Ben Thompson’s analogy, to be an Artificial Super Intelligence (ASI)—the rifle barrel—instead of Artificial General Intelligence (AGI)—the ammunitions:

What o3 and inference-time scaling point to is something different: AI’s that can actually be given tasks and trusted to complete them. This, by extension, looks a lot more like an independent worker than an assistant — ammunition, rather than a rifle sight. That may seem an odd analogy, but it comes from a talk Keith Rabois gave at Stanford…My definition of AGI is that it can be ammunition, i.e. it can be given a task and trusted to complete it at a good-enough rate (my definition of Artificial Super Intelligence (ASI) is the ability to come up with the tasks in the first place).

In this perspective the future of programming is certainly bright: We may not be too far from a future where we can “hire” cheap individual contributors—the ammunitions—to help generate 100% of the code. But ammunitions are only useful when we—the rifle barrel—point them to the desired targets. It gets trickier in a wicked world, where the targets would often dance unpredictability. Often times, it is even unclear which targets we should point to. In other words, creativity is about knowing what and where to aim; simply being the projectile isn’t!

Back to Marc, the disgruntled open-source maintainer from the tweet above: Marc’s moat—and the foundation of his business model—was knowledge, or specifically, the kind of execution-layer expertise that AI now replicates at scale. His deep contextual knowledge about his own codebase was valuable when humans couldn’t easily acquire it. But now, with AI commoditizing intelligence that can not only easily acquire that knowledge, but also execute on it, his OSS income funnel threatens to collapse.

This is a classic example of how new technology decimates information asymmetry. To quote Freakonomics:

It is common for one party to a transaction to have better information than another party. In the parlance of economists, such a case is known as an information asymmetry. We accept as a verity of capitalism that someone (usually an expert) knows more than someone else (usually a consumer). But information asymmetries everywhere have in fact been gravely wounded by the Internet.

Then it was the Internet; now, AI collapses that information asymmetry even more by not only giving away the information, but also acting on it.

Of course, this is only saying that he’s losing his moat to monetize on his open-source project, but this does not mean that his open-source project is in any way less valuable. In fact, to the community as a whole, his open-source project may get more valuable, thanks to AI agents making it easier (and cheaper) to capture value.

AI doesn’t break the commons, but it does break the existing OSS monetization model. Yet programmers will still want to contribute to open-source projects: The broken monetization model is just one way of converting the currency of reputation to fiat currency in the market economy. AI may break that specific monetization model, but the mechanism of converting reputation to monetary currency hasn’t broken yet.

Pushing the Money Downstream

While AI doesn’t kill OSS, it doesn’t make it any easier for people to make money off it either. If not in the contribution to—or in the possession of knowledge moat of—OSS projects, where can monetary rewards be captured?

I believe OSS will still be alive due to the virtuous cycle I described earlier: Programmers will still get the itch, they will still benefit from the reputation reaped from their OSS contribution in the gift economy, and this in turn will make the open-source ecosystem much better. But while there’s no money to be made in this cycle, there’s money to be captured downstream from the products of this cycle, as illustrated below:

The key difference between the commons and the uncommons lies in the nature of the problems they solve. The commons layer addresses problems shared across a domain—problems whose boundaries often map to industry lines (oil and gas, finance, pharmacy, and so on). Contributors and users benefit from network effects: the more people use and improve the same codebase, the more valuable it becomes for everyone. This isn’t charity either. Every contribution reflects a cost-benefit calculation, whether explicit or intuitive: is it cheaper to privately fork and bear the full maintenance burden, or to contribute upstream and let the community carry the weight?

The uncommons, by contrast, solves problems specific to individuals or organizations—the differentiated logic, the proprietary integrations, the things you don’t want your competitors to have. But because the uncommons depends on the commons, programmers working in the uncommons layer inevitably encounter friction in the foundations they build on. When they do, they face a choice: fork and maintain privately, or fix it upstream. For non-differentiating fixes, contributing back is almost always cheaper. The uncommons doesn’t just extract from the commons—it feeds the commons whenever doing so costs less than going it alone.

It is also interesting to note the dynamic between agentic AI and the commons and the uncommons. Due to AI slops, thanks to unguided code generation, the maintainers of the commons may react negatively to AI generated contributions. More slops means it can take more time, energy, and resources for the maintainers to do code review, which may lead to stricter governance on how code contributions are made, slowing down progress. Downstream in the uncommons, it’s a different story: AI helps individuals or organizations with specific problems in the uncommons, problems that otherwise wouldn’t have been addressed in the commons, and thus are more welcomed. In fact, using AI to aid software development is often encouraged in the name of productivity, as we’ve seen in many enterprises in the past few years. This is because the closed-source projects in the uncommons enjoy the luxury that open-source projects in the commons don’t have: a handful of familiar contributors.

This isn’t really a happy answer for OSS maintainers like Marc, who depends on the moat of deep source knowledge and expertise to fund his projects. With AI easily collapsing that moat, his choice to close-source his future projects behind a paywall is entirely rational, and perhaps necessary for surviving this paradigm shift. Perhaps the only consolation is that his prestige and personal brand—the judgement, taste, and stewardship that AI cannot replicate—remains valuable. The monetization path might have changed, but the currency of reputation hasn’t disappeared.

Share

Heads up: I’ll be posting new content at amenji.io first, then republishing it here on Abstract Engineering. The subscriber lists are separate, so if you’d like to keep following, please subscribe at amenji.io/subscribe.

Writing on my own site gives me more creative freedom—Substack, for instance, still doesn’t support tables.

Thanks for reading, and I hope you’ll find this post interesting!

Magnus Carlsen announced in a podcast that he would no longer defend the title he’d won in 2022 after consecutively remaining undefeated in the World Chess Championship since 2013. Many critics speculated he had lost his passion or was too afraid to lose to the next generation of players. About two years later, in March 2024, he announced the inaugural tournament of the Freestyle Chess Grand Slam Tour—a series of five major Chess960 tournaments held across different continents throughout 2025.

Chess960, also known as Freestyle Chess or Fischer Random, radically increases the possible combinations of the starting positions in a game by randomizing the positions of the back-rank pieces, leaving the front-rank pawns untouched. As the name suggests, the number of possible starting positions is increased to 960 possible combinations. The Freestyle Chess Grand Slam Tour, though, bans the traditional Chess starting position, including swapping the King and Queen.

Clearly, the motivation behind the Freestyle Chess Grand Slam Tour is to make chess more interesting and entertaining in the hope of drawing in a larger audience. This is reflected by the fact the tournament offers a bigger prize pool than traditional chess tournaments and features an innovative player-focused gameplay approach, such as the “confession booth.” Players are also equipped with heart rate monitors to provide “a novel layer of drama,” as Carlsen puts it.

I’m no chess fan, but this seems like a good revitalization of the ancient game, sparking enthusiasm and debate amongst the sport’s fans and grandmasters alike (though the first tournament of Chess960 dates back to 1996). It is definitely pertinent now with how AI has crept into the sport over the last three decades, starting with a brute-force approach of Deep Blue by IBM in 1997, to the use of artificial neural network and reinforcement learning pioneered by Google DeepMind with AlphaZero in 2017. It is no surprise then of AI’s increasing influence on how the game is played. Magnus Carlsen wrote in The Economists for the By Invitation section:

Why freestyle? Following my fifth consecutive victory in the World Chess Championship in 2022, I announced that I would no longer defend the title. Many speculated that I was exhausted, or that I was scared of the next generation of players. On the contrary, my passion for chess remains as strong as ever, and I am as ambitious as I’ve always been. What changed was my perspective on the format of the classical world championship itself.

⁠⁠The challenge wasn’t the games, which often stretch for hours. I enjoy the length of time allowed under the rules. My title defence in 2018 against Fabiano Caruana, for instance, pushed us, over 12 drawn games, to our mental and physical limits, before I emerged victorious in the tiebreak. The issue lay elsewhere, in the months of grinding preparation leading up to the event. Modern World Chess Championships demand endless memorisation of computer-generated opening lines, reducing the sport’s artistry to rote learning. As someone who treasures the creativity of chess, I wanted to focus more on this aspect of the game. Also, life beyond chess deserved my attention too.

Subscribe

Saving Creativity

In a way, I share Carlsen’s sentiment. Chess960 was originally introduced to reduce the emphasis on opening preparation and encourage creativity. By moving to freestyle, player decision-making in chess may be free from the need to internalize patterns over years of grinding to gain a competitive edge. Going freestyle is how the grandmasters strike back against AI with a renewed hope for creativity in the sport.

Should the same thought be entertained about programming? Programming, after all, is an art. It is a skill that’s improved with creativity rather than mere rote memorization of patterns of codes (though having some repertoire of knowledge on software design patterns does help). Many prominent programmers have expressed similar sentiments on the importance of taste, elegance, and beauty with regard to programming. Here’s how Linus Torvalds, the original hacker behind the Linux kernel and Git, explained good taste in 2007:

To me… the sign of people I really want to work with is that they have good taste… Good taste is about really seeing the big patterns and kind of instinctively knowing what’s the right way to do things.

And what are instincts but the use of implicit patterns so deeply familiar that we no longer remember learning them after years of repetition and experience—embedded deep in the back of our mind? If this is true, then AI is just a monstrous instinct machine that has learned to recognize the big patterns of our collective knowledge that we’ve put out on the Internet. It doesn’t truly understand the right way to do things, but it can infer them from the big patterns—and it will only get better.

To that point, AI has already internalized a wealth of creative output of perhaps some of the best programmers in the world and makes them readily available as generative code assistants. They help us scaffold code from a blank page, suggest code improvements for refactoring opportunities, and even fix our code plagued with cryptic error messages. If this effectively takes away the creative aspect of programming from the programmers, then will the programmers too, strike back?

The problem with this line of thinking is that chess is an adversarial zero-sum activity (one must lose for another to win), while programming—like many productive economic activities—is a positive-sum activity (everyone wins something). So while AI takes away the creative value of chess from the players (by eating away their creative decision-making opportunities), AI adds creative value to the programmers (by adding more options for creative decision-making opportunities). It makes no sense to strike back. We’re living the dream!

Exploiting the Kind and Capitalizing on the Wicked

There’s also another interesting angle of AI’s influence on chess, and it is related to learning and decision-making. Carlsen’s issue with the overreliance on rote learning computer-generated patterns and his desire to revive the creative aspect of chess reminds me of two thought-provoking ideas introduced by Robin Hogarth about how we learn and make decisions: the kind learning environment and its antithesis, the wicked learning environment.

To frame the two antithetical learning environments into questions: Does our decision-making improve with more experience? Or does having more experience make us myopic to the broader worldview, making us more susceptible to being biased in our judgment?

If your belief leans more to the former, then you’re in the camp of expert intuition who believes that good decision-making comes with years of repeated practice and experience. Otherwise, if you’re skeptical that having more experience is a good predictor for excellent decision-making, then you’re in the camp of heuristics and bias, who believe that experience leads to overconfidence and results in more decision errors.

A seminal paper titled Conditions for Intuitive Expertise: A Failure to Disagree reconciles the opposing views of these two camps. The two authors, Daniel Kahneman (camp heuristic and bias) and Gary Klein (camp expert intuition) touched on their differing perspectives and their reconciliation:

In this article we report on an effort to compare our views on the issues of intuition and expertise and to discuss the evidence for our respective positions. When we launched this project, we expected to disagree on many issues, and with good reason: One of us (GK) has spent much of his career thinking about ways to promote reliance on expert intuition in executive decision making and identifies himself as a member of the intellectual community of scholars and practitioners who study naturalistic decision making (NDM). The other (DK) has spent much of his career running experiments in which intuitive judgment was commonly found to be flawed; he is identified with the “heuristics and biases” (HB) approach to the field.

A surprise awaited us when we got together to consider our joint field of interest. We found ourselves agreeing most of the time. Where we initially disagreed, we were usually able to converge upon a common position. Our shared beliefs are much more specific than the commonplace that expert intuition is sometimes remarkably accurate and sometimes off the mark. We accept the common-place, of course, but we also have similar opinions about more specific questions: What are the activities in which skilled intuitive judgment develops with experience? What are the activities in which experience is more likely to produce overconfidence than genuine skill?

And so the answer to the above questions is that it depends. Ask a chess grandmaster, and you may get a nod of agreement. Ask a senior programmer, and he may disagree vehemently (simply because programmers are an angry bunch). What sets them apart is the domain in which their decision-making is applied. One lives in a world where the rules seldom change, and the other has to adjust and adapt frequently to meet the demands of the rapidly shifting world (and gets angry every time a new feature is requested “at the last minute,” or a change introduced an unexpected behavior).

Chess is considered the gold standard of a kind learning environment. It has simple and predictable rules and provides immediate feedback, and yet it is quite complex thanks to the astronomical size of the possible positional configuration of the pieces on a 8x8 board. (For traditional chess, a recent estimate is in the order of 10^44 reachable legal positions.) On the contrary, a wicked learning environment is the opposite: the rules are unclear and complex and may change overnight. Feedback in the environment is often delayed. And what you learn today may not be applicable tomorrow. These are the defining features of problems faced by any white-collar workers—and sure is true for programmers!

In a kind, zero-sum environment like chess, the incentive is to use AI’s super computational capability to exploit the solution space—commonly through “prepping”—to devise optimal opening strategies and gain competitive leverage over competitors. This effectively shrinks the space for human creativity, turning expertise into a matter of rote memorization rather than tactical ingenuity. In contrast, in a wicked, positive-sum environment like programming, the incentive is to capitalize on existing knowledge and tools using AI to accelerate discovery and expanding our horizons. Here, AI doesn’t limit our creativity; rather, it enables us to explore and traverse more complex solution spaces that were previously beyond our reach. In other words, AI’s interaction with the kind environment reduces the creativity space for humans; but it expands the creativity space for humans in a wicked environment.

Though in this post I limit myself to only use chess vs. programming to drive my musing, I’m sure this observation rings true to many other domains as well. As I’m writing this post, another example that came to mind is that of F1 (a kind, zero-sum environment) vs. forecasting long-term stock prices (a wicked, positive-sum environment).

In the case of F1, teams uses AI to exploit the useful patterns from real-time data to help in making fast pit strategy decisions, a job which traditionally relies on a team of engineers to make the call. This narrows down human choices to those already exploited by AI. As for traders interested in the long-term future price of stocks, the logical thing to do is capitalize on the wealth of financial and economical knowledge—a job better suited for AI than human—to help them make better decisions.

Well, does this mean AI’s involvement in chess—or any other kind environment—is bad? Not really. While it is generally true that in a kind environment AI reduces the number of available choices to go for optimizations, it also shift the creative frontier downstream into regions of small pockets of wicked environment within the domain. This is apparently Magnus Carlsen’s play style: by going for offbeat opening choices, he embraces ambiguity and face it with ingenuity. By steering away from exploited openings, he’d purposely land himself and his opponents into small pockets of wicked environments in chess—in the middlegames and the endgames—where creativity triumphs over rote memorization.

Creativity is About Knowing Where to Aim

With the advent of AI assistant tools like GitHub CoPilot and Cursor, I’ve come across many claims about the impending irrelevancy of programmers in the future where AI reigns supreme. In the mean time, the programming culture spawn yet another subculture, called vibe coding—where the main language for software development is English, and codes generated by AI are not meant to be fully understood (if you do review, understand, and test the code, you’re not vibing).

Perhaps one of most vocal leader who shared this thought is Jensen Huang, the CEO of NVIDIA. In 2024 last year at World Government Summit, he mentioned something about the future of programming to the chagrin of many programmers (myself included):

I’m going to say something, and it’s going to sound completely opposite of what people feel.

You probably recall, over the course of the last 10 to 15 years, almost everybody who sits on a stage like this would tell you it is vital that your children learn computer science. Everybody should learn how to program. In fact, it’s almost exactly the opposite: it is our job to create computing technology such that nobody has to program, and that the programming language is human.

Everybody in the world is now a programmer.

This is the miracle; this is the miracle of artificial intelligence. For the very first time, we have closed the gap. The technology divide has been completely closed, and this is the reason why so many people can engage with artificial intelligence. It is the reason why every single government, every single industry conference, and every single company is talking about artificial intelligence today.

His speech would ring true, if only programming is only constrained to the problem of writing code to make the interpreter or compiler happy. But programming is more than just about generating code. Software needs to be useful in order for it to continue to exist, and much of the input to the activities that generate real values in the software product requires humans. While writing code is an important part of the business, it is not complete in itself to deliver real software value. To give some examples, think software architecture and domain-driven design—both of which require very hard-to-master skills (especially as programmers), like developing business acumen, making architectural decisions, and making complex topics stick.

We should also be aware of the fallacy of thinking that the programming domain is a zero-sum game: That for one party to win, another must lose; that if AI dominates programming, then the programmers are pushed out of job. This worldview disregards the fact that programmers and software developers operate in a collaborative positive-sum world. In fact, as AI joins us in this positive-sum world, it is just so that AI needs to capitalize on human programmers’ inputs for its outputs to be useful, and in turn, human programmers need to capitalize on AI to remain relevant.

This doesn’t mean I completely disregard the notion of AI replacing human programmers in the future. The key difference between the AI we’re using right now and the one that Jensen Huang portends is in their capacity (or lack thereof) to be a manager instead of the individual contributor; to be an architect instead of the builder; or, to borrow Ben Thompson’s analogy, to be an Artificial Super Intelligence (ASI)—the rifle barrel—instead of Artificial General Intelligence (AGI)—the ammunitions:

What o3 and inference-time scaling point to is something different: AI’s that can actually be given tasks and trusted to complete them. This, by extension, looks a lot more like an independent worker than an assistant — ammunition, rather than a rifle sight. That may seem an odd analogy, but it comes from a talk Keith Rabois gave at Stanford…My definition of AGI is that it can be ammunition, i.e. it can be given a task and trusted to complete it at a good-enough rate (my definition of Artificial Super Intelligence (ASI) is the ability to come up with the tasks in the first place).

In this perspective the future of programming is certainly bright: We may not be too far from a future where we can “hire” cheap individual contributors—the ammunitions—to help generate 100% of the code. But ammunitions are only useful when we—the rifle barrel—point them to the desired targets. It gets trickier in a wicked world, where the targets would often dance unpredictability. Often times, it is even unclear which targets we should point to. In other words, creativity is about knowing what and where to aim; simply being the projectile isn’t!

In an era where AI reigns supreme, the game we must learn to play will be that of unpredictability, chaos, and ambiguity. It is no longer about perfecting the established art, but mastering the art of exploration and experimentation. And if history is any guide, humanity has always been remarkably good at just that.

Thanks for reading! Subscribe to my main newsletter at amenji.io for free to receive new posts.

Subscribe to amenji.io

The first few months of my professional programming experience were mostly spent on old codebases. As many programmers can attest, it’s not unusual to find old source codes that reeks of code smells and littered with anti-patterns, making them frustrating to test and maintain. Sure, I could have refactored and cleaned up the code. But that’s not how my manager—nor the software users—measures the value of my work.

In the race against deadlines, I trudged through the code smells and anti-patterns before finally delivering the new features and bug fixes on time. The lingering anti-patterns, however, remain—a decision I know would vex future maintainers as it had vexed me.

Stopping this vicious cycle seems deceptively simple: just refactor your code! Yet, why aren’t we doing it as often?

Subscribe now

Three Reasons Why

While most programmers are eager to write code, few are eager to refactor code—a sin I’m also guilty of. As discussed below, I think there are a few salient reasons for programmers’ reluctance to refactor codes (and it's not because we're lazy).

Broken Windows

Imagine that a group of delinquents have recently started throwing rocks at the windows of a nice and cozy apartment.

The first few broken windows may initially raise voices of concern amongst the tenants. If the police or the people who live in the apartment weren’t too keen on stopping the petty crime, the delinquents might continue to break more windows—after all, it seems that “no one cares.”

Frustrated, some people may start to move out. Rents may dwindle, the price of the apartment may drop, and the upkeeping costs may incessantly increase. Unwelcome idlers, loiters, and vagrants may eventually take their place. Those who stay are becoming disaffected and are normalized by the growing disorder, which, in effect, lowers the average “accepted” civility level of the community.

Over time, the voices of concern slowly turned into silence and indifference. As the apartment grew more dilapidated, abandonment soon exponentially followed. If only we’ve nipped the delinquents in the butts—as the old wisdom goes.

This is the core idea of Broken Windows Theory in criminology, and I was first introduced to it in The Pragmatic Programmer when the authors drew a parallel of the theory with refactoring.

According to The Atlantic Monthly article titled Broken Windows which first introduces the theory1, the “no one cares” behavior of the community—in response to the vandalism (or lack thereof)—gives off signals that can lead to the breaking down of the mutual sense of regard and civil obligations. The authors wrote:

[V]andalism can occur anywhere once communal barriers—the sense of mutual regard and the obligations of civility—are lowered by actions that seem to signal that "no one cares."

The lowered “communal barriers” lead to a downward spiral of a vicious cycle where less serious crimes begets the more serious ones, as described in the same article:

[…] A stable neighborhood of families who care for their homes, mind each other's children, and confidently frown on unwanted intruders can change, in a few years or even a few months, to an inhospitable and frightening jungle. A piece of property is abandoned, weeds grow up, a window is smashed. Adults stop scolding rowdy children; the children, emboldened, become more rowdy. Families move out, unattached adults move in. Teenagers gather in front of the corner store. The merchant asks them to move; they refuse. Fights occur. Litter accumulates. People start drinking in front of the grocery; in time, an inebriate slumps to the sidewalk and is allowed to sleep it off. Pedestrians are approached by panhandlers.

Back to our topic of refactoring: The temptation to ignore the problem at its onset is similar to how a programmer would ignore the signal of code smells during the early phase of its development. It is easier to say that we’ll fix the problem tomorrow or after delivering some important feature than to fix it on the spot when you’re racing to the deadlines.

In the programming world, we sugar-coat this bad habit with a term called “technical debt.” But we all know too well that as more of these debts are incurred (often at an exponential rate), we grow more reluctant to pay them. In reality, technical debt is a proxy for lowering the “accepted” level of code quality, not unlike how the neighborhood described in the Broken Windows article tends to become normalized with the lowered level of communal barriers. Indeed, each technical debt that goes unpaid over time amplifies the signal that says, “No one cares.”

The Price of Engineering

Even if we do manage to overcome the temptation and react early to the onset of code smells, we may face some difficulty in nipping the buds. How we make decisions to restructure and refactor our code—to fix the code defect as signaled by anti-patterns and code smells—can be positively influenced by our knowledge and expertise in the broader field of software engineering.

One tool that makes design decisions easier in software engineering is borne out of the realization that many problems—when we look at them at a high enough level—share some similarities that can be solved by similar design patterns. From refactoring.guru:

Design patterns are typical solutions to common problems in software design. Each pattern is like a blueprint that you can customize to solve a particular design problem in your code.

Since design patterns are established as typical solutions to common problems programmers face, most programmers in the industry should at least consider them when building software, right? After all, as the Gang of Four puts it: “Once you know the pattern, a lot of design decisions follow automatically.”

But not every programmer is willing to invest in learning design patterns when, realistically speaking, they can write away codes—while being oblivious to the “blueprints”—and still deliver working features. Why take the time to painstakingly learn the biblical Design Patterns by the Gang of Four when you can write working code now2?

One superficial answer is this: In every engineering profession, having the appropriate degree of engineering knowledge is critical for an engineer to observe the fine details that are easy to miss—to know that something’s not right—and, consequently, to have the know-how to make things right.

That may sound like a truism.

But when you observe the rise of a phenomenon such as resume-driven development—where narrow expertise on specific tools is given merit for hiring over “good taste” in software design—it seems that the superficial answer above may not be as evident.

The recently hired “Python API developer” may have done their job by the job description; after all, they have developed a web service with an API using Python. But does it scale to infinity3? Does the software design conform to the SOLID design principle? Do they design the application so that it is easily testable4? Clearly, there are many more abstract engineering factors that need to be considered when building any software—not just the language or tools with which the software is built. Design patterns are just one of them.

For software engineers, negligence to anti-patterns and the lack of necessary application of design patterns to combat them means that their source code—without clever refactoring to drive the positive evolution of the software—would eventually fall prey to software rot. In other words, being more knowledgeable in design patterns can help you articulate and reason about the anti-patterns and code smells plaguing your code, enabling you to develop a well-informed refactoring decision.

Of course, all these are necessarily shadowed by extra costs. The extra costs are the time spent not working on writing codes to learn about software engineering concepts and money spent to pay for the learning materials or training. Are these extra costs worth it? With a pair of myopic eyes, it may not be perceived to be so.

A politician may make a decision and enforce a policy that may sound good in the short term, just enough to rally voters to have a higher chance of winning the upcoming election. A programmer may also choose to divest from software engineering concepts and deliberations to quickly write away codes to meet the upcoming deadline, code quality be damned.

On both accounts, their choices can lead to consequences with terrible costs that only reveal themselves months or years in the future, perhaps after they’re no longer working on the project and have since moved on (perhaps to a higher position in the career ladder.)

As Thomas Sowell wrote in Basic Economics:

The fact that economic consequences take time to unfold has enabled government officials in many countries to have successful political careers by creating current benefits at future costs.

In other words, the fact that short-term benefits are easier to measure and observe than long-term benefits can be a boon to many politicians (and also—cough—to many programmers).

One of the examples the author used to illustrate this point concerns the problem of maintaining and replacing municipal buses, where it’s getting more expensive and costly. To keep the business afloat, the logical response for the bus service operator is to increase the fare price to cover the increasing cost of replacing the buses as they wear out. Yet the act of opposing such an “unjustified” increase in bus fares (as the politicians put it) by putting a price cap on the fare may be well received by oblivious bus riders. While this can, in the short term, reduce the cost of travel for many bus riders, it can increase the costs for travelers in the long term as the buses wear out throughout the years.

On the long-term consequences of force-capping the price fare, Thomas Sowell wrote:

It may be some years before enough buses start breaking down and wearing out, without adequate replacements, for the bus riders to notice that there now seem to be longer waits between buses and buses do not arrive on schedule as often as they used to.

When you also take into account the compounding effect of the Broken Windows theory, the overall economic penalty for the affected community can be quite high.

Using the municipal bus analogy, refactoring is the “price” increase the business has to pay to the engineers to cover the increasing maintenance “costs” thanks to anti-patterns and code smells. Due to this seemingly higher price of writing software than what would be otherwise with code that isn’t being regularly refactored—and compounded by how hard it is to measure the long-term benefits of such undertaking versus the allure of short-term benefits—it is easy to dismiss refactoring as unjustified extra work, as politicians do with the “unjustified” bus fare increase to cover the increasing replacement costs.

What can make refactoring expensive is the fact that good refactoring decisions require a solid grasp of software engineering concepts and knowledge—both of which require more time and effort for the programmer to learn and implement. This is simply the price to pay if you want well-engineered source code that will be easier to maintain, integrate, and test in the long run than you would otherwise with poorly engineered source code. At the managerial level, this translates to more expensive labor.

In short, if you want to write code that can stand the test of time, pay the price of engineering now or regret needing to pay for more later. Unfortunately, in reality, the person who chose not to pay the price of engineering and the person who does have to pay more some years later isn’t always the same person.

Incentives and Information Asymmetry

Perhaps programmers (or their managers) perceive little to no value in refactoring. As previously discussed, it may be viewed as extra work that can get in the way of delivering users the features or changes they want. Some, perhaps the clients for whom the software was developed, may even remark: Why bother fixing codes that aren’t broken?

Steven Levitt and Stephen Dubner, in their book Freakonomics, wrote that humans are driven by incentives:

Doctors, lawyers, contractors, stockbrokers, auto mechanics, mortgage brokers, financial planners: they all enjoy a gigantic information advantage. And they use that advantage to help you, the person who hired them, get exactly what you want for the best price.

Right?

It would be lovely to think so. But experts are human, and humans respond to incentives. How any given experts treat you, therefore, will depend on how that expert’s incentives are set up.

These incentives that influence the experts can work in your favor or against you. Car manufacturers, for example, are incentivized by designing better aesthetics, safety controls, comfortability, and durability of their cars to attract prospective buyers or risk losing their customers to their competitors (in a free market economy, of course). On the other hand, real estate agents may sell your house cheaper than they would their own house to sell your house quicker, and in doing so, they would lose only a meager couple of hundred dollars in commission compared to you, who potentially could lose a couple of thousands. (You will probably sell your house faster, though.)

Programmers—as their clients are often led to believe—are experts in writing software that delivers features deemed valuable to them. If their clients want some fancy features, the programmers are incentivized to do just that: Write codes to deliver the fancy features, often sacrificing best practices and code quality to meet deadlines and secure project milestones.

As the project evolves (or perhaps even later after “go-live”), however, the source code tends to grow unwieldy: Integrating new features becomes increasingly complex, and bugs become harder to catch and fix. But the programmers were effectively inculpable—for they had lived up to their promise of delivering the requested features in the initial allotted time frame, and the clients were still no less oblivious about their unfortunate situation.

Here, we can see an example of information asymmetry: a typical situation where one party of a transaction has better information than the other. Naturally, the party with more information would be incentivized to withhold some information so as to maximize their reward or minimize penalty. Thus, while neglecting refactoring works may save the programmers some effort, unbeknownst to their trustful clients, they will have to pay the hefty consequences of hard-to-maintain software in the future (e.g., paying more for support and more costly maintenance). In other words, the programmers are rewarded with more work (and pay) to fix the growing problems thanks to poor design decisions early in the development cycle.

Interestingly, the example above may not hold true in the Software-as-a-Service (SaaS) model, where the client would make recurring payments—as opposed to the traditional model of paying for the software to be developed (usually by outsourcing) and run in the client’s own infrastructure operated by the client’s own team. In this case, the programmers are incentivized to maintain high-quality source code, motivating them to make changes, add new features, and catch bugs more efficiently, thus ensuring better customer retention and attracting prospective new subscribers. If they were to do otherwise, they'd risk losing their customers’ subscriptions, who would rather move to better SaaS providers. Like what Gregor Hohpe (an enterprise architect at AWS) had advised, don’t run what you didn’t build!

Moral of The Story

A good piece of advice is to refactor whenever you see the signals—the smell of code stinks. That’s when you know there’s some refactoring to do. At the very least, if you cannot refactor immediately, make a note of it—raise an issue or a backlog to highlight the code that needs refactoring. Other programmers in your team may not be aware that the code suffers from anti-patterns and can be further cleaned and improved. Don’t let that one broken window instigate more code smells being left unattended. It is far easier to refactor small pieces of code than to refactor a giant, big ball of mud.

Apart from the deterioration of code quality over time, the continuing shifts in the knowledge and available technology that went into the design and implementation of the software are another reason that refactoring is inherently a never-ending task. For example, gaining more clarity into the business domain may guide you to better design decisions5, as does learning about design patterns appropriate for your use cases. In addition, adopting new technology that wasn’t available even a few years ago might make your code a hundred times easier to maintain.

Alas, you can never really end the cycle. The vicious cycle will always be alive, and with each cycle, your source code can grow insidiously. It is thus crucial for you and your team to understand that refactoring is an essential, perpetual task; negligence of this fact would ensue further disarray and anti-patterns proliferating the source code—plunging you, your team, or future maintainers into maintenance hell.

The Real Moral of the Story

Perhaps it’s too easy to give such advice that puts the burden of change on the individual programmer. Instead, let us ask ourselves: Are we working in an environment where it’s easier to do the right thing than the wrong thing? Can we be incentivized to refactor code—an unglamorous and underappreciated work—so that we’re not only myopically focused on short-term rewards (i.e., features and bug fixes) but also on long-term values (i.e., maintainability, testability, and scalability)?

In Atomic Habits, the author James Clear wrote:

Environment is the invisible hand that shapes human behavior.

If we as programmers can’t change our behavior—to fix broken windows early, to pay the price of engineering, or to be more vigilant on code quality—by simply trying to be more proactive and disciplined, then perhaps it’s time to scrutinize the environment that shapes how we work.

The Masquerade

Digitalization, or digital transformation, has long since become a key factor responsible for the progress and development of the world's economies. "Every company is a technology company," or variations of the phrase, is an oft-spoken mantra for highlighting the importance of digitalization to every organization, no matter the industry. Recently, the push for digital transformation is usually induced by the furor surrounding emerging technologies, such as artificial intelligence (AI), cloud computing, big data, and blockchain.

With the hype around the emerging tech, many organizations, perhaps due to fear of missing out, are racing to transform their business to be "digital". But what does that mean? If it's about adopting digital technology, we've had a track record of that going back to a few decades ago. (The first digital alarm clock was first patented in 1956.)

From experience, when people talk about going digital, they usually mean adopting emerging technologies to realize its lofty promises: that big data and data analytics enable data-driven business decisions, and in some way or another, AI is a silver bullet that can solve an organization's myriad problems.

Of course, the promises of big data and AI are real. However, they have led some organizations to develop a lackadaisical strategy for digital transformation. Decisions for digital transformation are mostly pivoted around the procurement of emerging technologies, rather than by reimagining the operating system of the business and changing the ways of working; that is, transforming the business operating model.

This is unfortunate. Digitalization should be an enabler and offer opportunities for lifestyle change instead of a mere leap in the technology being used. To do otherwise is to only assume the appearance of a smart organization. A masquerade.

Subscribe now

Masquerading Digitalization with Digitization

Simply carrying out a mere transition from paper to its digital forms (e.g., PDF documents, image files, etc.) does not imply digitalization. Here's an example.

There are a few local restaurants that I've personally experienced where their effort to digitalize backfires miserably, to the point where it would have been much easier to order the menu without the digital technology.

There's no longer any physical menu placed on the counter or the dining table. QR code stickers, however, are conspicuously glued onto each table. When scanned, it would lead to a shared Google Photos link containing the images of the restaurant menu. With a bad cellular connection, it would take a while for the thumbnails of the images to load at their full quality. There was quite a number of images; one image for each page. After a few swipes left and right or scrolling up and down the list of images, it can take a while to read most of the menus to find what I'm looking for before I can finalize my order.

In the end, replacing physical menus with digital copies does not make discovering (and re-discovering) menus easier. The overall experience turned out to be frustrating. To make things worse, everyone can see who has visited the shared Google Photos link.

Clearly, whoever decided to replace the physical menu with digital pictures muddled digitization for digitalization. Lifting and shifting your physical copies of data into a digital landscape does not make your organization digital. In fact, the meaning of digital transformation is evolving, with more than 72% of industry leaders viewing digitalization as much more than lift-and-shift.

To give them the benefit of the doubt, for all I know, the restaurants were just trying to save some paper, albeit at the cost of a somewhat disappointing customer experience. But some of the worst offenders yet are those who lauded and coveted digitalization the most: Enterprise IT.

Masquerading Digitalization with Emerging Technology

Migrating to the cloud is a no-brainer when organizations decide to "go digital". The cloud had risen in favor over on-premise infrastructure for more than a decade since Amazon first revealed a unique business model at the time, stemming from an idea -- a multi-billion dollar idea1 -- that their existing compute and storage infrastructure can be offered as a service, with great agility and scalability.

According to a 2023 cloud adoption statistics by Zippia, 61% of businesses migrated their workloads to the cloud in 2020, although the share of workflows on the cloud is quite low, at only 25%, according to The Economist.

In Brunei, a 2019 report by the Authority for Info-communications Technology (AITI), a statutory body that focuses on the development of IT in Brunei, showed that most IT services infrastructure (e.g., data analytic tools, business process automation, HR management system, etc.) are on-premise2, with less than 5% cloud adoption for each sector. That was five years ago. (Unfortunately, I couldn't find recent reports on cloud adoption in Brunei.)

Nevertheless, over the past few years, signs for cloud adoptions by local businesses are increasingly popping up, such as the adoption of AWS Managed Services by Datastream Digital Sdn Bhd (DST) and Progresif's partnership with Alibaba Cloud. Brunei's Digital Economy Masterplan 2025 by the Digital Economy Council (DEC) even mentioned "Public Cloud Adoption" as part of its key projects in 2021, among other digital transformation initiatives.

It is all the more important, then, to understand what it really takes for an organization, especially those already operating for decades with the support of Enterprise IT to undergo digital transformation.

Many Enterprise IT are prone to hold on to their long-held traditions. Most are unable to forgo IT processes and operational culture that are already well-established and deeply entrenched in their organization. The processes are followed almost down to a dogmatic level, in the name of minimizing risks and ensuring stability and reliability.

The typical tradition includes loathsome bureaucratic IT processes. They are the manual and tedious checks and balances organically injected at almost every step of the value streams throughout the organization's lifetime, thanks to the learnings from past incidents and failures. In other words, they're the organization's counter-responses to past disappointments, which, unfortunately, end up introducing waste in the organization's value streams. Fittingly, in the DevOps community, they are referred to as "scar tissues".

Traditional change management processes are one example of scar tissue, where each "normal" change requires a lengthy process of approvals, careful planning and coordination, and manual testing and deployment methods. To make things worse, these are usually exacerbated by the lack of cooperation between teams in organizations that suffers the symptoms of organizational silos.

Organizational silos are the defining feature of traditional IT, where teams are locally optimized, but end-to-end cross-team value delivery are not. Thus, works that require cross-team collaborations are impeded due to friction. The existence of the friction is easy to explain from economic standpoint: the builders (e.g., software engineers, data engineers, data scientists) are incentivized by making more changes or adding more features in response to the changing business needs, whereas the infrastructure team and the support team are disincentivized because they want to keep the software reliable and easy to support (by the virtue of less frequent changes and fewer features to support). As economists are wont to say, our actions are driven by incentives. When the incentives are not aligned, you get friction.

Without fixing the organization's processes and organizational structure, whatever fancy technology the organization is trying to adopt will go to waste. Suppose you have set up Continuous Integration and Continuous Delivery pipelines (CI/CD) which automatically run unit tests and integration tests against your application each time a change is made to your code in your version control repository. Suppose also that you've painstakingly written some custom scripts to allow an application custodian to pick a snapshot version of the application to be automatically deployed to the production environment. If your organization still fanatically follows a traditional change management process -- the likes that require approvals across multiple stages and require you to fill in numerous forms -- the value proposition of the continuous property made available by the CI/CD pipelines will be diminished. In short, the inherent culture of dogmatism and fanaticism for traditional ways of working inhibits digital transformation from squeezing out its maximal value outcome.

Understanding the implication of this masquerade, as described above, is paramount for successful digital transformation. In the effort to transform an organization to be digital, it would be neglectful to be obsessed with the procurement and implementation of emerging technologies without expending more energy and resources on transforming their people and the operating model they're clinging to.

Seeing Through the Masquerade

Mixing emerging technology with traditional IT culture is a recipe for disaster. As Gregor Hohpe, an Enterprise Strategist at Amazon Web Services wrote in his book Cloud Strategy:

If you don't know how to drive, buying a faster car is the worst thing you can do.

Viral videos online showing amateur drivers driving recklessly and crashing their cars are all too common. Its dour consequences are obvious; and, grim as they may be, they serve as effective reminders and warnings to other drivers.

Organizations driving the digitalization vehicle do not normally have this luxury. Instead, digitalization mishaps are difficult to notice: We're using the cloud now, we're digital!, or We're using Docker to do the deployment! are just fanciful masquerades if symptoms of organizational silos are still rampant across the organization, or deliverables take months before they can be deployed and realized.

For digitalization strategy and execution to be successful, it is paramount that everyone across all levels in the organization can see through the masquerades and understand the desired outcome of digitalization for the organization that maximizes its business value.