HNNewShowAskJobs
Built with Tanstack Start
Hosting SQLite Databases on GitHub Pages (2021)(phiresky.github.io)
84 points by WA9ACE 4 days ago | 20 comments
  • misthop4 days ago

    It would be a it more compelling if every example, at least for me, didn't return `[error: NetworkError: A network error occurred.]`

    • cwmma4 days ago |parent

      I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file.

      1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

      • Groxx4 days ago |parent

        huh. is this due to ambiguity in whether you want gzipped content vs gzipped transport (of arbitrary content), and/or which range the bytes are requesting? I can see both being useful, but I don't know what headers are available for these intentions...

        • cwmma4 days ago |parent

          my understanding is that technically only gzipped content is supported, not gzipped transport of arbitrary content. Due to ambiguity around the word 'append' [1] in the spec, firefox adds 'identity' (aka don't compress) to the end of the list of compressions supported while most other browsers replace the list with 'identity'. Also it should be noted that this is not a user configurable header so you can't actually try to override it.

          There is a second layer to the bug in that github pages should almost certainly not be sending back slices of compressed files even if gzip is listed before identity and some change to something in the github stack probably exposed the bug that was there in firefox all along.

          I literally just stumbled on this last week while doing a side project that involved browser range requests so this is fresh in my head.

          1. it comes down to whether it was meant that 'Accept-Encoding:identity' should be appended to the list of header values possibly overwriting the one that already was there or if 'identity' should be appended to the list of values already in the 'Accept-Encoding' header. Firefox does the latter, everyone else does the former.

      • smithcoin4 days ago |parent

        It doesn't work for me on safari either.

        • cwmma4 days ago |parent

          works fine on safari desktop for me

      • ngc66774 days ago |parent

        this exact same example used to work in firefox a few years back, i guess some change introduced this bug in between

        • seligman994 days ago |parent

          Indeed, I lost the history in a shuffle, but a similar use case broke in some Firefox update, and it's the exact reason behind this comment:

          https://github.com/seligman/podcast_to_text/blob/master/sear...

          In my case, loading the entire file is loading a tiny bit more data, so this fallback doesn't hurt, but it's still annoying, and broke any hope I had of doing something more clever with the dataset.

        • cwmma4 days ago |parent

          The actual regression might be with githup pages where firefox was sending the same ambiguous headers the whole time but something in github's stack started interpreting them differently.

    • graemep4 days ago |parent

      I get "Error connecting to database: RuntimeError: abort(Error: server uses gzip or doesn't have length). Build with -s ASSERTIONS=1 for more info" from the sposorship stats site lined toand the same as you for the demo.

      its interesting, but it is not robust and I very much doubt it will have the claimed benefit for lasting without further work. At the very least its more work and requires cross browser testing.

      It seems a lot of effort just to avoid paying a small amount a month for a VPS. Its free and " I forget about the backend and stop paying for whatever VPS it was on" stops being a problem (although running all the little things you have on one VPS would also simplify that).

  • pspeter34 days ago

    I wonder if the author would use DuckDB WASM now?

    • mrtimo4 days ago |parent

      I'm using DuckDB WASM on github pages. This will take about 10 seconds to load [1] and shows business trends in my county (Spokane County). This site is built using data-explorer [2] which uses many other open-source projects including malloy and malloy-explorer. One cool thing... if you use the UI to make a query on the data - you can share the URL with someone and they will see the same result / query (it's all embedded in the URL).

      [1] - https://mrtimo.github.io/spokane-co-biz/#/model/businesses/e... [2] - https://github.com/aszenz/data-explorer

    • westurner4 days ago |parent

      From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 :

      >> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs

      >> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http

      >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution

      duckdb/duckdb-wasm: https://github.com/duckdb/duckdb-wasm

      • westurner4 days ago |parent

        "PSA: SQLite WAL checksums fail silently and may lose data" so that's probably not how to sync sqlite; https://news.ycombinator.com/item?id=44672902

        electric-sql/electric: https://github.com/electric-sql/electric :

        > Specifically, Electric is a read-path sync engine for Postgres. It syncs data out of Postgres into ... anything you like. The core sync protocol is based on a low-level HTTP API. This integrates with CDNs for highly-scalable data delivery.

        electric-sql/pglite: https://github.com/electric-sql/pglite :

        > Embeddable Postgres with real-time, reactive bindings.

        "Using Postgres for Everything" https://news.ycombinator.com/item?id=40420474

    • ngc66774 days ago |parent

      is it doing HTTP range request, to not download an entire db?

  • zX41ZdbW4 days ago

    It's old news, but you can even host ClickHouse databases on GitHub pages: https://github.com/ClickHouse/web-tables-demo

  • bomewish4 days ago

    Where does the 600mb SQLite file live though? GitHub won’t host that. Lfs server?

  • crtasm4 days ago

    (2021)

    >(or IPFS or any static file hoster)

  • amighol4 days ago

    Is it just me or It does not run on Firefox. I ran on edge it works. In Firefox I get: [error: NetworkError: A network error occurred.]

  • 4 days ago
    [deleted]