Skip to content

Notes

Jump to bottom
Max Marrone edited this page Mar 7, 2026 · 7 revisions

Probable access patterns

Sort by most-recent or least-recent

Filter by syslog ID or unit

Filter by date ("hide all before this log" etc.)

Filter by severity

Search message text (basic or regex)

Search all fields by text (basic or regex)

Marking, commenting, and re-saving (if we can save files that big)

Not probable access patterns

Sort by anything other than wall clock time or entry number

To do

  • Investigate RxDB, IndexedDB, PouchDB
  • Optimize index by making it an int32 array--a log will probably not have more than 4 billion records.
  • Figure out displaying many records, virtualization/bigtables
  • Figure out indexing, filtering, sorting

Simplify list virtualization by precomputing each message's line count? Then each list item's size can be defined up-front and passed to our virtualization library.

Rough priorities

  • Re-exporting

  • Reboot markers

  • Fix horizontal scrolling somehow

Concurrency notes

Questions like "what services are there" can and should be kept in main thread memory and answered basically instantly.

  • Building the index takes a long while. It should be truly cancellable and it should not block the event loop.
    • Manually yield to event loop?
      • Feels hacky
      • Overhead of a few percent because of 4ms minimum delay.
      • Will feel stuttery on especially large records, if we go too long between yields.
    • Offload to WebWorker?
      • Send index back to main thread by copy?
        • Copy MAY add significant overhead. Need to test.
        • Copy MAY block the main thread. Need to test.
      • Transfer index back to main thread?
        • Need to represent index as ArrayBuffer or similar. Not necessarily trivial or fun--may require doing our own memory management for resizing arrays, etc. May involve overhead just as bad as copying the full index across the worker boundary.
      • Keep index in original worker?
        • Can only serve one query at a time
        • Can't cancel queries
  • Querying the index to do a full text search takes a few seconds. It should ideally be cancellable, and it should not block the event loop.
  • The index can be several GB big.

Table layout crap

https://adamlynch.com/flexible-data-tables-with-css-grid/

Clone this wiki locally