Distributed, high-performance blob storage on SQLite

(zerodha.tech)

47 points | by ghostfoxgod 2 hours ago

6 comments

  • bob1029 19 minutes ago
    > JSON blobs

    An alternative to faster blob storage is to represent the blob's schema in SQL directly.

    If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.

    One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.

  • CyrusGrissom 27 minutes ago
    I couldn't find the word "replication" in the post or on the GitHub page. So, what about fault tolerance?
    • tomasol 18 minutes ago
      I'm curious too. The only backup I found on GitHub uses cp; they may use filesystem snapshots or Litestream in production. If it’s Litestream, litestream sync -wait can force a sync after a write and wait for remote replication to complete.
  • feverzsj 1 hour ago
    Sounds like you should use a database-per-user design.
    • mkesper 1 hour ago
      Right, then even sqlite should be enough?
  • zodwick 11 minutes ago
    all roads lead to sqlite
  • vrox 1 hour ago
    [flagged]
  • mesm7903 1 hour ago
    [flagged]