From Tomorrow Back to Yesterday: A Tale of Two Web Architectures - Yang

SUMMARY

David Yang, co-founder of Lightweight Labs, presents a case study migrating from local-first "Web After Tomorrow" architecture to server-centric "Web From Yesterday" using DataStar, highlighting trade-offs in performance, complexity, and developer experience with Clojure examples.

STATEMENTS

• David Yang introduces himself as a co-founder of Lightweight Labs and shares his background in web development, including founding Fullstack Academy.
• He reflects on the evolution of web apps starting from Gmail and Google Maps in 2004-2005, which transformed the browser into a desktop-like runtime.
• The browser's openness led developers to tolerate its shortcomings to build rich applications, providing users with magical experiences like client-side routing and incremental DOM updates.
• Over the last 20 years, massive investments in web technology aimed to make the browser an instant-download desktop client, improving speed, APIs, and data management.
• React popularized the idea that UI is a function of state but left state management as an exercise for developers, leading to added complexity through various tools.
• Early React assumptions included slow DOM and network versus fast JavaScript and scalable client compute, influencing designs like virtual DOM manipulation on the client.
• React introduced many developers to functional programming concepts, paving the way for Clojure's adoption in web development with tools like Reagent and Re-frame.
• Clojure innovations like Datomic, Datascript, and event sourcing emphasized immutability and single source of truth for client-side graph databases.
• Nikita Tonsky's "Web After Tomorrow" essay envisioned client-side data streaming, local storage in Datascript, and self-contained environments synced by data.
• At Lightweight Labs, they built a V1 app syncing QuickBooks data into a fast spreadsheet interface using Web After Tomorrow architecture with Datascript for optimistic updates.
• Initial user feedback praised the speed, but pain points emerged like slow initial loads for large datasets, differences between Datascript and Datomic, and non-commutative operations.
• The architecture became overly complex for their small team, leading to high development overhead and difficulty adding features without reconciliation issues.
• They considered alternatives like InstantDB, Electric, and Replicache but opted against them due to needs for server control and readiness concerns.
• Linear's talks revealed scaling challenges in their sync architecture, including adding queries, caches, and lazy collections over time.
• A quote from video game rendering emphasized that duplicating state leads to lifelong bugs in synchronization.
• David Nolan recommended DataStar, which uses HTML morphing and server-sent events for server-side rendering.
• Anders Murphy's blog posts on DataStar inspired them, especially for rendering large datasets efficiently.
• Modern networks enable immediate mode rendering, treating the browser as a frame buffer like in video games.
• They developed "Feather," a framework using SSE for a 60 FPS render loop, sending full HTML that DataStar morphs into the DOM.
• All state is managed server-side in tab connections, including session and UI state, with the browser handling minimal tasks.
• Frontend bundle size dropped from 5.5 MB to 30 KB, simplifying development and improving load times dramatically.
• Demos showed real-time updates, Git browser via server shell commands, and QuickBooks ledger with responsive interactions, all without client JavaScript.
• Trade-offs include no offline mode, retained browser state for inputs and scrolling, and the need to unlearn SPA habits.
• The approach reduces client-server sync complexity, echoing simpler web development of the past but with modern real-time features.
• In the AI era, developers should seek simpler abstractions to build desired user experiences without piling on complexity.

IDEAS

• The browser's transformation from a simple delivery vessel to a full application runtime began with apps like Gmail, challenging proprietary plugins like Flash.
• Decades of desktop tooling like persistence and sockets were absent in early browsers, driving innovation to bridge that gap.
• React's core philosophy treats UI as a pure function of state, but outsourcing state management created a proliferation of complex libraries.
• Assumptions from React's era—slow networks favoring client compute—no longer hold with today's fast networks and powerful servers.
• Clojure's functional paradigms, such as immutability and reactive atoms, were early adapters to React-like thinking in web development.
• Tonsky's vision of local-first apps with Datascript enables self-contained clients synced minimally, ideal for offline-capable tools.
• Syncing QuickBooks data into Datascript allowed optimistic updates for spreadsheet-like speed but struggled with large initial payloads.
• Non-commutative mutations in optimistic updates complicate reconciliation, especially with external APIs like QuickBooks.
• Building a custom sync engine solves many distributed systems problems but often becomes a maintenance burden outweighing benefits.
• Immediate mode rendering, redrawing everything per frame, becomes feasible with HTML morphing over fast networks, simplifying state handling.
• Treating the browser as a mere frame buffer shifts complexity to the server, leveraging cloud scalability for UI state.
• Server-side state storage per tab and session enables full observability, like jacking into user sessions for debugging.
• Dropping client JavaScript bundles to 30 KB drastically reduces load times and eliminates frontend reconciliation bugs.
• Real-time collaboration emerges naturally from event-sourced server renders, without additional sync logic.
• Unlearning SPA heuristics, like preferring partial updates, can lead to counterintuitive wins like full HTML compression via Brotli.
• Beefier servers are cheaper now, making server-centric models more economical than client-side complexity.
• The PHP meme highlights shipping speed over philosophical purity, applicable to web architectures favoring simplicity.
• Automation efforts, like sync engines, often create new problems consuming more time than they save.
• In AI's rise, reducing complexity through simpler abstractions allows focus on user features rather than infrastructure battles.
• Web components can encapsulate JavaScript libraries, allowing server morphing around them without interference.
• Server continuations simplify callbacks by maintaining state without rehydration or security checks.
• Performance becomes device-independent, equalizing experiences from old PCs to modern devices.
• Event streams enable replayability for auditing user actions, akin to event sourcing benefits.

INSIGHTS

• The web's evolution prioritized client-side richness, but diminishing returns in browser capabilities now favor server-centric simplicity for reduced developer burden.
• State duplication across client and server inevitably leads to synchronization bugs that compound over time, making single-server-state models preferable for maintainability.
• Fast modern networks render client-optimization heuristics obsolete, allowing full re-renders to outperform incremental updates in perceived responsiveness.
• Functional programming's emphasis on immutability scales better in server environments, avoiding the pitfalls of mutable client state.
• Local-first architectures excel for offline needs but falter in complexity for teams without deep sync expertise, especially with large datasets.
• Shifting UI state to the server unlocks observability and collaboration but requires rethinking browser roles as passive renderers.
• Trade-offs in architecture should prioritize developer velocity in startups, where execution speed trumps theoretical elegance.
• Immediate mode rendering revives CGI-like simplicity while delivering SPA fluidity, bridging past and future web paradigms.
• AI tools may automate complexity, but fundamental reductions in state management yield more sustainable productivity gains.
• Web components provide a pragmatic bridge for integrating legacy JavaScript libraries into server-rendered flows.
• Event-sourced server models inherently support replayability and auditing, enhancing reliability without extra effort.
• Nostalgia for early web's upload-and-refresh simplicity masks its modern potential when augmented with streaming technologies.

QUOTES

• "I was there Gandalf when the power of Flash failed."
• "UI is a function of state, right? But the state itself that was really left, you know, as the meme goes as an exercise to the reader."
• "Jesus Christ guys, this is so fast. I don't know how you're doing. This is too fast."
• "You give someone state they'll have bug for a day but you teach them how to represent state in two places they have to be kept in sync and they'll get bugs for a lifetime."
• "The network is so fast that my wife and my kids are eating up three 4K streams in my network. And I'm sitting here trying to optimize like JSON messages, right?"
• "Worse is better, right? That our natural heuristic sometimes fails us that, you know, sending all the HTML compressed via Brotli might be better than partial updates."
• "Other languages are philosophers, but PHP developers drive Lamborghinis because they just ship, right?"
• "I spent a lot of time on this task. Maybe I should write a program automating it."

HABITS

• Regularly reflect on personal journey in web development to contextualize current architectural choices.
• Evaluate alternatives like InstantDB or Electric by consulting founders directly for fit with specific needs.
• Prototype architectures quickly in startups to test execution speed against feature complexity.
• Hire experts like Anders Murphy when discovering aligned explorations in community blog posts.
• Maintain all state server-side for full observability, allowing session jacking for debugging.
• Cap render loops at 60 FPS to mimic game-like responsiveness in web apps.
• Use macros in Hiccup for arbitrary Clojure code execution during renders to simplify actions.

FACTS

• Gmail launched in 2004 and Google Maps in 2005, marking the shift to browser-based desktop-like apps.
• Browser technology investments over 20 years enabled tools from Backbone.js to React Query and GraphQL for data management.
• Lightweight Labs' V1 frontend bundle was 5.5 MB, reduced to 30 KB in V2 using DataStar.
• Loading a QuickBooks instance with 100,000 transactions took 20 seconds to first paint in V1, but only 100 milliseconds in V2.
• Nvidia's GeForce Now streams 4K games from cloud GPUs, demonstrating scalable server compute over client hardware.
• Linear's sync architecture evolved over three years to include queries, caches, partial bootstraps, and lazy collections.
• DataStar enables rendering a billion cells in the browser via server-side morphing.

REFERENCES

• Gmail (2004 launch).
• Google Maps (2005 launch).
• Yahoo Maps team (David Yang's experience).
• Flash and Java applets.
• Backbone.js.
• React.
• React Query.
• GraphQL.
• Reagent.
• Re-frame.
• Datomic.
• Datascript.
• "Web After Tomorrow" essay by Nikita Tonsky.
• QuickBooks API.
• InstantDB.
• Electric (V2 and V3).
• Replicache.
• Linear app talks (V1 and follow-up).
• Video game frame rendering quote.
• DataStar project (HTML morphing and SSE).
• Anders Murphy's blog posts and Hyperfission framework.
• GeForce Now (Nvidia cloud gaming).
• Rum (previous V1 library).
• Hiccup macros.
• Web components.
• LiveView Native.
• PHP community meme.
• XKCD automation comic.

HOW TO APPLY

• Assess your app's data size and permissions; if datasets fit client-side with simple sync needs, start with local-first like Datascript for optimistic updates.
• Prototype initial loads by serializing large Datascript instances and measure against user tolerance, aiming for under 5 seconds to avoid discomfort.
• Identify non-commutative operations early and implement delta management to reconcile client mutations with server responses.
• When complexity rises, evaluate sync alternatives by discussing requirements with tool founders to ensure alignment with server control needs.
• Integrate DataStar by setting up SSE connections and a render loop that generates full HTML frames at 60 FPS for morphing.
• Store all state in server tab connections, including UI details like hover states, and hook data sources like Datomic during renders.
• Develop actions using Hiccup macros to execute Clojure code, encoding them as UUIDs for browser event dispatch back to the server.

ONE-SENTENCE TAKEAWAY

Embrace server-centric rendering to slash web app complexity while delivering responsive experiences via modern fast networks.

RECOMMENDATIONS

• Prioritize developer velocity by choosing architectures that eliminate client-server sync, freeing time for user features.
• Experiment with DataStar for server-side morphing to simplify frontend code and reduce bundle sizes dramatically.
• Unlearn SPA assumptions favoring partial updates; test full HTML re-renders for potential latency wins.
• Use web components to encapsulate interactive JavaScript libraries, maintaining server control over morphing.
• Maintain single-server state for observability, enabling easy session debugging and real-time collaboration.
• Cap renders at 60 FPS in prototypes to achieve game-like fluidity without client-side reactivity.
• Consult community explorations like Anders Murphy's posts before investing in custom sync solutions.
• Shift to event-sourced models for inherent replayability, aiding auditing and reliability.
• Accept trade-offs like no offline mode if network reliability is high, focusing on shipping speed.

MEMO

David Yang, co-founder of Lightweight Labs, stood before a Clojure/Conj audience in Charlotte, North Carolina, recounting his odyssey through two decades of web architecture. Once a proponent of full-stack JavaScript via his bootcamp Fullstack Academy, Yang lamented the loss of the web's "innocent days"—viewing source code, FTP deploys—eclipsed by the browser's ascent as a desktop rival. Pivotal moments like Gmail's 2004 launch and Google Maps in 2005 shattered illusions of proprietary plugins like Flash, ushering in client-side magic: no more jarring page reloads, just fluid routing and DOM tweaks. Yet, this dream demanded bridging the browser's gaps in persistence and sockets, fueling an industry-wide push toward instant, rich clients.

React crystallized the era's ethos: UI as state function, born when networks lagged and JavaScript zipped. Virtual DOM diffs and client-templated layouts made sense then, duplicating business logic for scalability. Yang credits React with awakening him to functional elegance, echoing Clojure's early wins—immutable rendering via Reagent, reactive atoms in Re-frame, Datascript's client graph parity with Datomic. These fed Nikita Tonsky's "Web After Tomorrow": local data streams, Datascript storage, minimal sync between self-contained realms. At Lightweight Labs, Yang and Aaron Iba chased this for a QuickBooks-syncing spreadsheet app. V1 ingested data into Datomic, serialized Datascript snapshots to clients for optimistic mutations—blissfully fast once loaded, earning awed customer gasps. But reality bit: 20-30 second initial loads for big files clashed with their "fastest QuickBooks" pitch.

Pain mounted. Datascript-Datomic mismatches (like long IDs) snarled deltas; non-commutative updates erred on reconciliation; query rerenders puzzled. Worst, complexity ossified their duo's workflow—features felt like "open heart surgery." Alternatives like InstantDB, Electric, or Replicache tempted, but server control demands and maturity gaps deterred. Linear's saga, from elegant IndexedDB objects to scaled caches and lazy loads, warned of sync's siren call. A game-rendering quip crystallized it: duplicate state, invite eternal bugs. Yang pondered his home: wife's JP Morgan remote desktops, son's GeForce Now Fortnite—4K streams gobbling bandwidth while he fretted JSON optimization. Enter DataStar, via David Nolan's nudge and Anders Murphy's blogs: HTML morphing plus server-sent events revived "immediate mode" rendering, redrawing worlds per frame like games, feasible on swift nets.

Enthralled, they hired Murphy, birthing "Feather"—a lightweight SSE-tethered loop pumping 60 FPS HTML frames. Browsers morph via DataStar, events bounce UUIDs back; state lives server-side in tab connections: shared sessions, per-tab UI (hovers, inputs). Datascript queries hook seamlessly. Frontend shrank from 5.5 MB React behemoth to 30 KB, no Ajax, pure server HTML. Demos dazzled: a crowd-sourced counter hitting 67 (nod to his kids' meme), real-time hovers tracked server-side; a Git browser piping shell commands for CGI nostalgia with SPA seamlessness; QuickBooks ledger sorting columns, popping menus—all responsive, no client JS.

Performance soared: 100,000-transaction loads hit first paint in 100 ms, versus V1's taxing 20 seconds—"I feel bad for the browser," Yang quipped, its flame graphs churning. Implications abound: server continuations nix rehydration woes; security via render-time permissions; full observability lets devs hijack sessions. Device-agnostic, event-replayable, collaboration-native. Trade-offs? No offline, latent inputs/scrolls, SPA muscle memory purge. Beefier servers? Cheap now. PHP's "philosophers vs. Lamborghinis" meme resonated—ship over theorize. Echoing XKCD's automation trap, Yang decried sync as time sink. V2 evokes youth's SQL-to-HTML uploads, but turbocharged: real-time, collaborative, desktop-rich.

In AI's shadow, Yang urges ditching complexity piles for abstractions delivering user joy. Not open-sourcing Feather yet—sharp edges in library-app boundaries, state ownership linger—but DataStar and Murphy's Hyperfission beckon explorers. Q&A probed JS interop via web components, mobile via LiveView Native analogs, open-source paths. Applause sealed it: a nostalgic pivot forward, simplifying the web's tangled weave.