[docs] some more updates to docs

Also, add support for overstruck/backspace-underscore for
doing bold/underline.

docs/source/hotkeys.rst

@@ -10,6 +10,8 @@ a more detailed explanation of each key.
 Spatial Navigation
 ------------------
 
+The majority of these hotkeys should be available in all views.
+
 .. list-table::
    :header-rows: 1
    :widths: 6 6 6 20
@@ -27,8 +29,8 @@ Spatial Navigation
      - :kbd:`PgUp`
      - Up a page
    * - :kbd:`j`
-     - :kbd:`Return`
      - :kbd:`↓`
+     -
      - Down a line
    * - :kbd:`k`
      - :kbd:`↑`
@@ -100,6 +102,9 @@ Spatial Navigation
 Chronological Navigation
 ------------------------
 
+These hotkeys are only functional on views that are time-based, like the log
+view or the histogram view.
+
 .. list-table::
    :header-rows: 1
    :widths: 5 5 20
@@ -123,6 +128,44 @@ Chronological Navigation
      - :kbd:`Shift` + :kbd:`r`
      - Forward/backward by the relative time that was last used with the goto command.
 
+Breadcrumb Navigation
+---------------------
+
+The following hotkeys are related to the breadcrumb bar that is below the top
+status bar.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 5 20
+
+   * - Keypress
+     - Description
+   * - :kbd:`ENTER`
+     - Focus on the breadcrumb bar.  Or, if the bar is currently focused,
+       accept the selected value and drop focus.
+   * - :kbd:`Escape`
+     - Drop focus on the breadcrumb bar.
+   * - :kbd:`←`
+     - Select the crumb to the left. If the first crumb is selected, the
+       selection will wrap around to the last crumb.
+   * - :kbd:`→`
+     - Accept the current value, which might mean navigating to the value in
+       the view, then selecting the crumb to the right.
+   * - :kbd:`Ctrl` + :kbd:`a`
+     - Select the first crumb.
+   * - :kbd:`Ctrl` + :kbd:`e`
+     - Select the last crumb.
+   * - :kbd:`↓`
+     - Select the next value in the crumb dropdown.
+   * - :kbd:`↑`
+     - Select the previous value in the crumb dropdown.
+   * - :kbd:`Home`
+     - Select the first value in the crumb dropdown.
+   * - :kbd:`End`
+     - Select the last value in the crumb dropdown.
+
+While a crumb is selected, you can perform a fuzzy search on the possible
+values by typing in the value you are interested in.
 
 .. _hotkeys_bookmarks:
 

docs/source/index.rst

@@ -1,8 +1,3 @@
-.. lnav documentation master file, created by
-   sphinx-quickstart on Fri Jul 12 21:09:39 2013.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
 Welcome to lnav's documentation!
 ================================
 
@@ -15,13 +10,12 @@ Contents:
    :maxdepth: 2
 
    intro
-   usage
-   cookbook
-   howitworks
-   config
-   cli
    ui
    hotkeys
+   cli
+   usage
+   cookbook
+   config
    formats
    sessions
    commands
@@ -29,6 +23,7 @@ Contents:
    sqltab
    events
    data
+   howitworks
    faq
 
 

docs/source/ui.rst

@@ -20,7 +20,9 @@ hotkeys described in the :ref:`hotkeys_display` section or by pressing
 :kbd:`ENTER` to activate the breadcrumb bar, moving to the first crumb, and
 then selecting the desired view.  You can switch back to the previous view by
 pressing :kbd:`q`.  You can switch forward to the new view by pressing
-:kbd:`a`.
+:kbd:`a`.  If the views are time-based (e.g. log and histogram), pressing
+:kbd:`Shift` + :kbd:`q` and :kbd:`Shift` + :kbd:`a` will synchronize the top
+times in the views.
 
 The right side of the display has a proportionally sized 'scrollbar' that
 shows:
@@ -31,14 +33,62 @@ shows:
 * the locations of search hits by using a tick-mark pointing to the left;
 * the locations of bookmarks by using a tick-mark pointing to the right.
 
-Above and below the main body are status lines that display:
+Top Status Bar
+--------------
 
-* the current time;
-* the name of the file the top line was pulled from;
-* the log format for the top line;
-* the current view;
-* the line number for the top line in the display;
-* the current search hit, the total number of hits, and the search term;
+The top status bar shows the current time and messages stored in the
+:code:`lnav_user_notifications` table.
+
+Below the top status bar is the breadcrumb bar that displays the semantic
+location of the top line in the main view.  For example, within a
+pretty-printed JSON document, it will show the path to property at the top
+of the view.  The actual content of the bar depends on the current view and
+will be updated as you navigate around the main view.  The bar can also be
+used to navigate around the document by focusing on it.
+
+Breadcrumb Bar
+--------------
+
+.. figure:: lnav-breadcrumbs-help.png
+   :align: center
+   :figwidth: 90%
+
+   Screenshot of the breadcrumb bar focused and navigating the help text
+
+To focus on the breadcrumb bar, press :kbd:`ENTER`.  The :kbd:`←`/:kbd:`→`
+cursor keys can be used to select a crumb and the :kbd:`↑`/:kbd:`↓` keys can
+be used select a value of that crumb.  To accept a value and drop focus on the
+bar, press :kbd:`ENTER`.  To accept a value and move to the next crumb, press
+:kbd:`→`.  Using :kbd:`→` makes it quicker to drill down into a document
+without having to constantly switch focus.  To drop focus on the bar without
+accepting anything, press :kbd:`Escape`.
+
+There are three types of crumbs:
+
+* a dropdown where one of a limited set of values can be selected;
+* a combobox where a value can be entered directly or selected;
+* a numeric input for entering array indexes.
+
+When a dropdown or combobox is selected, you can type part of the desired value
+to filter the list of values.  For example, the first crumb is always the
+current view, typing in "hi" will filter the list down to the "HIST" value.
+
+Configuration Panels
+--------------------
+
+After the main view content, there is a header bar for two configuration
+panels: Files and Filters.  These panels provide visual access to parts of
+lnav's configuration.
+
+The Files panel is open initially to display progress in loading files.
+The following information can be displayed for each file:
+
+* the "unique" portion of the path relative to the other files;
+* the amount of data that has been indexed;
+* the date range of log messages contained in the file;
+* the errors that were encountered while trying to index the file;
+* the notes recorded for files where some automatic action was taken,
+  like hiding the file if it was seen as a duplicate of another file.
 
 If the view supports filtering, there will be a status line showing the
 following:
@@ -94,6 +144,14 @@ On color displays, the log messages will be highlighted as follows:
 
     :config /ui/theme grayscale
 
+The breadcrumb bar will show:
+
+* the timestamp
+* the log format for the top line;
+* the name of the file the top line was pulled from;
+* the line number for the top line in the display;
+* the current search hit, the total number of hits, and the search term;
+
 DB
 ^^
 

lnav.1

@@ -19,7 +19,7 @@
 .ft R
 .fi
 ..
-.TH LNAV "1" "August 2013"
+.TH LNAV "1" "August 2022"
 .SH NAME
 lnav \- ncurses-based log file viewer
 .SH SYNOPSIS
@@ -49,13 +49,31 @@ Print help and exit
 \fB\-H\fR
 Display the internal help text.
 .TP
+\fB\-n\fR
+Run without the curses UI.  (headless mode)
+.TP
+\fB\-c\fR cmd
+Execute a command after the files have been loaded.
+.TP
+\fB\-f\fR path
+Execute the commands in the given file.
+.TP
 \fB\-I\fR path
 Add the given configuration directory to the search path.
 .TP
+\fB\-n\fR
+Do not open the default syslog file if no files are given.
+.TP
+\fB\-q\fR
+Quiet mode.  Do not print the log messages after executing all of the commands.
+.TP
 \fB\-i\fR
 Install the given format files in the $HOME/.lnav/formats/installed directory
 and exit.
 .TP
+\fB\-u\fR
+Update formats installed from git repositories.
+.TP
 \fB\-C\fR
 Check the configuration and exit.  The log format files will be loaded and
 checked.  Any files given on the command-line will be loaded checked to make
@@ -67,9 +85,6 @@ Write debug messages to the given file.
 \fB\-V\fR
 Print version information.
 .TP
-\fB\-a\fR
-Load all of the most recent log file types.
-.TP
 \fB\-r\fR
 Recursively load files from the given directories.
 .TP

src/base/ansi_scrubber.cc

@@ -42,21 +42,55 @@
 static pcrepp&
 ansi_regex()
 {
-    static pcrepp retval("\x1b\\[([\\d=;\\?]*)([a-zA-Z])");
+    static pcrepp retval("\x1b\\[([\\d=;\\?]*)([a-zA-Z])|(.)\b\\3|.\b_|_\b.");
 
     return retval;
 }
 
 void
-scrub_ansi_string(std::string& str, string_attrs_t& sa)
+scrub_ansi_string(std::string& str, string_attrs_t* sa)
 {
     pcre_context_static<60> context;
     auto& regex = ansi_regex();
     pcre_input pi(str);
+    int64_t origin_offset = 0;
+    int last_origin_offset_end = 0;
 
     replace(str.begin(), str.end(), '\0', ' ');
     while (regex.match(context, pi)) {
         auto* caps = context.all();
+        const auto sf = pi.get_string_fragment(caps);
+
+        if (sf.length() == 3 && sf[1] == '\b') {
+            if (sa != nullptr) {
+                shift_string_attrs(*sa, caps->c_begin + 1, -2);
+                if (sf[0] == '_' || sf[2] == '_') {
+                    sa->emplace_back(
+                        line_range{caps->c_begin, caps->c_begin + 1},
+                        VC_STYLE.value(text_attrs{A_UNDERLINE}));
+                } else {
+                    sa->emplace_back(
+                        line_range{caps->c_begin, caps->c_begin + 1},
+                        VC_STYLE.value(text_attrs{A_BOLD}));
+                }
+                sa->emplace_back(
+                    line_range{last_origin_offset_end, caps->c_begin + 1},
+                    SA_ORIGIN_OFFSET.value(origin_offset));
+            }
+            if (sf[0] == '_') {
+                str.erase(str.begin() + caps->c_begin,
+                          str.begin() + caps->c_end - 1);
+                last_origin_offset_end = caps->c_begin + 1;
+            } else {
+                str.erase(str.begin() + caps->c_begin + 1,
+                          str.begin() + caps->c_end);
+                last_origin_offset_end = caps->c_begin + 1;
+            }
+            origin_offset += 2;
+            pi.reset(str);
+            continue;
+        }
+
         struct line_range lr;
         bool has_attrs = false;
         text_attrs attrs;
@@ -154,26 +188,37 @@ scrub_ansi_string(std::string& str, string_attrs_t& sa)
             }
         }
         str.erase(str.begin() + caps[0].c_begin, str.begin() + caps[0].c_end);
-        shift_string_attrs(sa, caps[0].c_begin, -caps[0].length());
+        if (sa != nullptr) {
+            shift_string_attrs(*sa, caps[0].c_begin, -caps[0].length());
 
-        if (has_attrs) {
-            for (auto rit = sa.rbegin(); rit != sa.rend(); rit++) {
-                if (rit->sa_range.lr_end != -1) {
-                    break;
+            if (has_attrs) {
+                for (auto rit = sa->rbegin(); rit != sa->rend(); rit++) {
+                    if (rit->sa_range.lr_end != -1) {
+                        continue;
+                    }
+                    rit->sa_range.lr_end = caps[0].c_begin;
                 }
-                rit->sa_range.lr_end = caps[0].c_begin;
+                lr.lr_start = caps[0].c_begin;
+                lr.lr_end = -1;
+                if (attrs.ta_attrs || attrs.ta_fg_color || attrs.ta_bg_color) {
+                    sa->emplace_back(lr, VC_STYLE.value(attrs));
+                }
+                role | [&lr, &sa](role_t r) {
+                    sa->emplace_back(lr, VC_ROLE.value(r));
+                };
             }
-            lr.lr_start = caps[0].c_begin;
-            lr.lr_end = -1;
-            if (attrs.ta_attrs || attrs.ta_fg_color || attrs.ta_bg_color) {
-                sa.emplace_back(lr, VC_STYLE.value(attrs));
-            }
-            role |
-                [&lr, &sa](role_t r) { sa.emplace_back(lr, VC_ROLE.value(r)); };
+            sa->emplace_back(line_range{last_origin_offset_end, caps->c_begin},
+                             SA_ORIGIN_OFFSET.value(origin_offset));
+            origin_offset += caps->length();
         }
 
         pi.reset(str);
     }
+
+    if (sa != nullptr && last_origin_offset_end > 0) {
+        sa->emplace_back(line_range{last_origin_offset_end, (int) str.size()},
+                         SA_ORIGIN_OFFSET.value(origin_offset));
+    }
 }
 
 void

src/base/ansi_scrubber.hh

@@ -62,7 +62,7 @@
  * @param str The string to check for ANSI escape sequences.
  * @param sa  The container for any style attributes.
  */
-void scrub_ansi_string(std::string& str, string_attrs_t& sa);
+void scrub_ansi_string(std::string& str, string_attrs_t* sa);
 
 /**
  * Populate a variable map with strings that contain escape sequences that

src/base/attr_line.cc

@@ -51,7 +51,7 @@ attr_line_t::with_ansi_string(const char* str, ...)
 
     if (ret >= 0 && formatted_str != nullptr) {
         this->al_string = formatted_str;
-        scrub_ansi_string(this->al_string, this->al_attrs);
+        scrub_ansi_string(this->al_string, &this->al_attrs);
     }
 
     return *this;
@@ -61,7 +61,7 @@ attr_line_t&
 attr_line_t::with_ansi_string(const std::string& str)
 {
     this->al_string = str;
-    scrub_ansi_string(this->al_string, this->al_attrs);
+    scrub_ansi_string(this->al_string, &this->al_attrs);
 
     return *this;
 }

src/base/lnav.console.cc

@@ -303,6 +303,9 @@ println(FILE* file, const attr_line_t& al)
                     if (style.ta_attrs & A_BOLD) {
                         line_style |= fmt::emphasis::bold;
                     }
+                    if (style.ta_attrs & A_UNDERLINE) {
+                        line_style |= fmt::emphasis::underline;
+                    }
                     if (style.ta_fg_color) {
                         auto color_opt = curses_color_to_terminal_color(
                             style.ta_fg_color.value());

src/base/string_attr_type.cc

@@ -40,6 +40,8 @@ string_attr_type<void> SA_PREFORMATTED("preformatted");
 string_attr_type<std::string> SA_INVALID("invalid");
 string_attr_type<std::string> SA_ERROR("error");
 string_attr_type<int64_t> SA_LEVEL("level");
+string_attr_type<string_fragment> SA_ORIGIN("origin");
+string_attr_type<int64_t> SA_ORIGIN_OFFSET("origin-offset");
 
 string_attr_type<role_t> VC_ROLE("role");
 string_attr_type<role_t> VC_ROLE_FG("role-fg");

src/base/string_attr_type.hh

@@ -155,7 +155,8 @@ using string_attr_value = mapbox::util::variant<int64_t,
                                                 std::string,
                                                 std::shared_ptr<logfile>,
                                                 bookmark_metadata*,
-                                                timespec>;
+                                                timespec,
+                                                string_fragment>;
 
 class string_attr_type_base {
 public:
@@ -202,6 +203,8 @@ extern string_attr_type<void> SA_PREFORMATTED;
 extern string_attr_type<std::string> SA_INVALID;
 extern string_attr_type<std::string> SA_ERROR;
 extern string_attr_type<int64_t> SA_LEVEL;
+extern string_attr_type<string_fragment> SA_ORIGIN;
+extern string_attr_type<int64_t> SA_ORIGIN_OFFSET;
 
 extern string_attr_type<role_t> VC_ROLE;
 extern string_attr_type<role_t> VC_ROLE_FG;

src/data_scanner.cc

@@ -105,6 +105,19 @@ static struct {
         pcrepp("\\A(</[\\w:]+\\s*>)"),
     },
 
+    {
+        "h1",
+        pcrepp("\\A([A-Z \\-])"),
+    },
+    {
+        "h2",
+        pcrepp("\\A([A-Z \\-])"),
+    },
+    {
+        "h3",
+        pcrepp("\\A([A-Z \\-])"),
+    },
+
     {
         "coln",
         pcrepp("\\A(:)"),

src/data_scanner.hh

@@ -50,6 +50,11 @@ enum data_token_t {
     DT_XML_EMPTY_TAG,
     DT_XML_OPEN_TAG,
     DT_XML_CLOSE_TAG,
+
+    DT_H1,
+    DT_H2,
+    DT_H3,
+
     /* DT_QUALIFIED_NAME, */
 
     DT_COLON,

src/data_scanner_re.re

@@ -21,18 +21,17 @@
  * DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY
  * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
  * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
- * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
- * ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
  * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
-#include "config.h"
-
 #include <arpa/inet.h>
 #include <netinet/in.h>
 #include <sys/socket.h>
 
+#include "config.h"
 #include "data_scanner.hh"
 
 bool data_scanner::tokenize2(pcre_context &pc, data_token_t &token_out)
@@ -196,6 +195,10 @@ bool data_scanner::tokenize2(pcre_context &pc, data_token_t &token_out)
            RET(DT_XML_CLOSE_TAG);
        }
 
+       "\n"[A-Z][A-Z _\-0-9]+"\n" {
+           RET(DT_H1);
+       }
+
        ":" { RET(DT_COLON); }
        "=" { RET(DT_EQUALS); }
        "," { RET(DT_COMMA); }

src/document.sections.cc

@@ -81,8 +81,21 @@ hier_node::lookup_path(const hier_node* root,
     return retval;
 }
 
-metadata
-discover_metadata(const attr_line_t& al)
+struct metadata_builder {
+    std::vector<section_interval_t> mb_intervals;
+    std::unique_ptr<hier_node> mb_root_node;
+
+    metadata to_metadata() &&
+    {
+        return {
+            std::move(this->mb_intervals),
+            std::move(this->mb_root_node),
+        };
+    }
+};
+
+static void
+discover_metadata_int(const attr_line_t& al, metadata_builder& mb)
 {
     const auto& orig_attrs = al.get_attrs();
     auto headers = orig_attrs
@@ -115,7 +128,7 @@ discover_metadata(const attr_line_t& al)
         }
     }
 
-    std::vector<section_interval_t> intervals;
+    auto& intervals = mb.mb_intervals;
 
     struct open_interval_t {
         open_interval_t(uint32_t level, file_off_t start, section_key_t id)
@@ -185,15 +198,51 @@ discover_metadata(const attr_line_t& al)
         }
     }
 
-    return {
-        sections_tree_t{std::move(intervals)},
-        std::move(root_node),
-    };
+    for (auto& interval : intervals) {
+        auto start_off_opt
+            = get_string_attr(orig_attrs, &SA_ORIGIN_OFFSET, interval.start);
+        if (start_off_opt) {
+            interval.start += start_off_opt.value()->sa_value.get<int64_t>();
+        }
+        auto stop_off_opt
+            = get_string_attr(orig_attrs, &SA_ORIGIN_OFFSET, interval.stop - 1);
+        if (stop_off_opt) {
+            interval.stop += stop_off_opt.value()->sa_value.get<int64_t>();
+        }
+    }
+
+    hier_node::depth_first(root_node.get(), [&orig_attrs](hier_node* node) {
+        auto off_opt
+            = get_string_attr(orig_attrs, &SA_ORIGIN_OFFSET, node->hn_start);
+
+        if (off_opt) {
+            node->hn_start += off_opt.value()->sa_value.get<int64_t>();
+        }
+    });
+
+    if (!root_node->hn_children.empty()
+        || !root_node->hn_named_children.empty())
+    {
+        mb.mb_root_node = std::move(root_node);
+    }
+}
+
+metadata
+discover_metadata(const attr_line_t& al)
+{
+    metadata_builder mb;
+
+    discover_metadata_int(al, mb);
+
+    return std::move(mb).to_metadata();
 }
 
 class structure_walker {
 public:
-    explicit structure_walker(string_fragment content) : sw_scanner(content)
+    explicit structure_walker(attr_line_t& al, line_range lr)
+        : sw_line(al), sw_range(lr),
+          sw_scanner(string_fragment::from_str_range(
+              al.get_string(), lr.lr_start, lr.lr_end))
     {
         this->sw_interval_state.resize(1);
         this->sw_hier_nodes.push_back(std::make_unique<hier_node>());
@@ -201,9 +250,9 @@ public:
 
     metadata walk()
     {
+        metadata_builder mb;
         pcre_context_static<30> pc;
         data_token_t dt = DT_INVALID;
-        lnav::document::metadata retval;
         auto& pi = this->sw_scanner.get_input();
 
         while (this->sw_scanner.tokenize2(pc, dt)) {
@@ -241,6 +290,16 @@ public:
                     this->flush_values();
                     break;
                 }
+                case DT_H1: {
+                    this->sw_line.get_attrs().emplace_back(
+                        line_range{
+                            this->sw_range.lr_start + el.e_capture.c_begin + 1,
+                            this->sw_range.lr_start + el.e_capture.c_end - 1,
+                        },
+                        VC_ROLE.value(role_t::VCR_H1));
+                    this->sw_line_number += 2;
+                    break;
+                }
                 case DT_LCURLY:
                 case DT_LSQUARE:
                 case DT_LPAREN: {
@@ -323,10 +382,12 @@ public:
             this->sw_hier_stage->hn_parent = nullptr;
         }
 
-        retval.m_sections_root = std::move(this->sw_hier_stage);
-        retval.m_sections_tree = sections_tree_t(std::move(this->sw_intervals));
+        mb.mb_root_node = std::move(this->sw_hier_stage);
+        mb.mb_intervals = std::move(this->sw_intervals);
 
-        return retval;
+        discover_metadata_int(this->sw_line, mb);
+
+        return std::move(mb).to_metadata();
     }
 
 private:
@@ -435,6 +496,8 @@ private:
         ivstate.is_name.clear();
     }
 
+    attr_line_t& sw_line;
+    line_range sw_range;
     data_scanner sw_scanner;
     int sw_depth{0};
     size_t sw_line_number{0};
@@ -446,9 +509,9 @@ private:
 };
 
 metadata
-discover_structure(string_fragment content)
+discover_structure(attr_line_t& al, struct line_range lr)
 {
-    return structure_walker(content).walk();
+    return structure_walker(al, lr).walk();
 }
 
 std::vector<breadcrumb::possibility>

src/document.sections.hh

@@ -104,7 +104,7 @@ struct metadata {
 
 metadata discover_metadata(const attr_line_t& al);
 
-metadata discover_structure(string_fragment sf);
+metadata discover_structure(attr_line_t& al, struct line_range lr);
 
 }  // namespace document
 }  // namespace lnav

src/lnav_util.cc

@@ -149,7 +149,8 @@ to_json(yajlpp_gen& gen, const attr_line_t& al)
                         elem_map.gen("");
                     },
                     [&](const bookmark_metadata* bm) { elem_map.gen(""); },
-                    [&](const timespec& ts) { elem_map.gen(""); });
+                    [&](const timespec& ts) { elem_map.gen(""); },
+                    [&](const string_fragment& sf) { elem_map.gen(sf); });
             }
         }
     }

src/logfile_sub_source.cc

@@ -2178,14 +2178,14 @@ logfile_sub_source::text_crumbs_for_line(int line,
                                uniq_path.template get<std::string>()));
         });
 
-    string_attrs_t sa;
     logline_value_vector values;
     auto& sbr = values.lvv_sbr;
+    attr_line_t al(to_string(sbr));
 
     lf->read_full_message(msg_start_iter, sbr);
-    format->annotate(file_line_number, sa, values);
+    format->annotate(file_line_number, al.get_attrs(), values);
 
-    auto opid_opt = get_string_attr(sa, logline::L_OPID);
+    auto opid_opt = get_string_attr(al.get_attrs(), logline::L_OPID);
     if (opid_opt && !opid_opt.value().saw_string_attr->sa_range.empty()) {
         const auto& opid_range = opid_opt.value().saw_string_attr->sa_range;
         const auto opid_str
@@ -2224,14 +2224,16 @@ logfile_sub_source::text_crumbs_for_line(int line,
     }
 
     auto sf = sbr.to_string_fragment();
+    auto body_opt = get_string_attr(al.get_attrs(), SA_BODY);
     auto sf_lines = sf.split_lines();
     auto msg_line_number = std::distance(msg_start_iter, line_pair.second);
     auto line_from_top = line - msg_line_number;
-    if (sf_lines.size() > 1) {
+    if (sf_lines.size() > 1 && body_opt) {
         if (this->lss_token_meta_line != file_line_number
             || this->lss_token_meta_size != sf.length())
         {
-            this->lss_token_meta = lnav::document::discover_structure(sf);
+            this->lss_token_meta = lnav::document::discover_structure(
+                al, body_opt.value().saw_string_attr->sa_range);
             this->lss_token_meta_line = file_line_number;
             this->lss_token_meta_size = sf.length();
         }

src/readline_curses.cc

@@ -1412,7 +1412,7 @@ readline_curses::do_update()
 
         if (!this->rc_alt_value.empty()) {
             alt_al.get_string() = this->rc_alt_value;
-            scrub_ansi_string(alt_al.get_string(), alt_al.get_attrs());
+            scrub_ansi_string(alt_al.get_string(), &alt_al.get_attrs());
 
             alt_start = getmaxx(this->vc_window) - alt_al.get_string().size();
         }

src/spectro_source.cc

@@ -301,7 +301,7 @@ spectrogram_source::list_value_for_overlay(const listview_curses& lv,
         line.append(" ");
     }
     line.append(buf);
-    scrub_ansi_string(line, value_out.get_attrs());
+    scrub_ansi_string(line, &value_out.get_attrs());
 
     snprintf(buf, sizeof(buf), "Max: %'.10lg", sb.sb_max_value_out);
     buflen = strlen(buf);

src/statusview_curses.cc

@@ -44,7 +44,7 @@ status_field::set_value(std::string value)
 
     sa.clear();
 
-    scrub_ansi_string(value, sa);
+    scrub_ansi_string(value, &sa);
     this->sf_value.with_string(value);
 }
 

src/textfile_sub_source.cc

@@ -29,6 +29,7 @@
 
 #include "textfile_sub_source.hh"
 
+#include "base/ansi_scrubber.hh"
 #include "base/fs_util.hh"
 #include "base/itertools.hh"
 #include "config.h"
@@ -323,8 +324,8 @@ textfile_sub_source::text_crumbs_for_line(
         auto ll_iter = lf->begin() + lfo->lfo_filter_state.tfs_index[line];
         auto ll_next_iter = ll_iter + 1;
         auto end_offset = (ll_next_iter == lf->end())
-            ? lf->get_index_size()
-            : ll_next_iter->get_offset();
+            ? lf->get_index_size() - 1
+            : ll_next_iter->get_offset() - 1;
         const auto initial_size = crumbs.size();
 
         meta_iter->second.ms_metadata.m_sections_tree.visit_overlapping(
@@ -560,13 +561,16 @@ textfile_sub_source::rescan_files(
                     auto read_res = lf->read_file();
 
                     if (read_res.isOk()) {
-                        auto content = read_res.unwrap();
+                        auto content = attr_line_t(read_res.unwrap());
 
+                        scrub_ansi_string(content.get_string(),
+                                          &content.get_attrs());
                         this->tss_doc_metadata[lf->get_filename()]
                             = metadata_state{
                                 st.st_mtime,
                                 static_cast<file_size_t>(st.st_size),
-                                lnav::document::discover_structure(content),
+                                lnav::document::discover_structure(
+                                    content, line_range{0, -1}),
                             };
                     }
                 }

src/textview_curses.cc

@@ -438,7 +438,7 @@ textview_curses::textview_value_for_row(vis_line_t row, attr_line_t& value_out)
     this->tc_sub_source->text_value_for_line(*this, row, str);
     this->tc_sub_source->text_attrs_for_line(*this, row, sa);
 
-    scrub_ansi_string(str, sa);
+    scrub_ansi_string(str, &sa);
 
     struct line_range body, orig_line;
 
@@ -790,6 +790,23 @@ textview_curses::set_sub_source(text_sub_source* src)
     return *this;
 }
 
+bool
+textview_curses::grep_value_for_line(vis_line_t line, std::string& value_out)
+{
+    bool retval = false;
+
+    if (this->tc_sub_source
+        && line < (int) this->tc_sub_source->text_line_count())
+    {
+        this->tc_sub_source->text_value_for_line(
+            *this, line, value_out, text_sub_source::RF_RAW);
+        scrub_ansi_string(value_out, nullptr);
+        retval = true;
+    }
+
+    return retval;
+}
+
 void
 text_time_translator::scroll_invoked(textview_curses* tc)
 {

src/textview_curses.hh

@@ -563,21 +563,10 @@ public:
             : this->tc_sub_source->text_source_name(*this);
     }
 
-    bool grep_value_for_line(vis_line_t line, std::string& value_out)
+    bool grep_value_for_line(vis_line_t line, std::string& value_out);
+
+    void grep_quiesce()
     {
-        bool retval = false;
-
-        if (this->tc_sub_source
-            && line < (int) this->tc_sub_source->text_line_count()) {
-            this->tc_sub_source->text_value_for_line(
-                *this, line, value_out, text_sub_source::RF_RAW);
-            retval = true;
-        }
-
-        return retval;
-    }
-
-    void grep_quiesce() {
         if (this->tc_sub_source != nullptr) {
             this->tc_sub_source->quiesce();
         }
@@ -649,10 +638,7 @@ public:
         return retval;
     }
 
-    bool get_hide_fields() const
-    {
-        return this->tc_hide_fields;
-    }
+    bool get_hide_fields() const { return this->tc_hide_fields; }
 
     void execute_search(const std::string& regex_orig);
 

src/view_helpers.cc

@@ -326,7 +326,7 @@ open_pretty_view()
                 al.get_string(),
                 text_sub_source::RF_FULL | text_sub_source::RF_REWRITE);
             lss.text_attrs_for_line(*log_tc, vl, al.get_attrs());
-            scrub_ansi_string(al.get_string(), al.get_attrs());
+            scrub_ansi_string(al.get_string(), &al.get_attrs());
             if (log_tc->get_hide_fields()) {
                 al.apply_hide();
             }

test/document.sections.tests.cc

@@ -35,7 +35,7 @@
 
 TEST_CASE("lnav::document::sections::basics")
 {
-    static const std::string INPUT = R"(
+    attr_line_t INPUT = R"(
 {
    "msg": "Hello, World!",
    "obj": {
@@ -66,7 +66,7 @@ TEST_CASE("lnav::document::sections::basics")
 }
 )";
 
-    auto meta = lnav::document::discover_structure(INPUT);
+    auto meta = lnav::document::discover_structure(INPUT, line_range{0, -1});
 
     meta.m_sections_tree.visit_all([](const auto& intv) {
         auto ser = intv.value.match(
@@ -85,10 +85,42 @@ TEST_CASE("lnav::document::sections::basics")
 
 TEST_CASE("lnav::document::sections::empty")
 {
-    static const std::string INPUT
+    attr_line_t INPUT
         = R"(SOCKET 1 (10) creating new listening socket on port -1)";
 
-    auto meta = lnav::document::discover_structure(INPUT);
+    auto meta = lnav::document::discover_structure(INPUT, line_range{0, -1});
+
+    meta.m_sections_tree.visit_all([](const auto& intv) {
+        auto ser = intv.value.match(
+            [](const std::string& name) { return name; },
+            [](const size_t index) { return fmt::format("{}", index); });
+        printf("interval %d:%d %s\n", intv.start, intv.stop, ser.c_str());
+    });
+    lnav::document::hier_node::depth_first(
+        meta.m_sections_root.get(), [](const auto* node) {
+            printf("node %p %d\n", node, node->hn_start);
+            for (const auto& pair : node->hn_named_children) {
+                printf("  child: %p %s\n", pair.second, pair.first.c_str());
+            }
+        });
+}
+
+TEST_CASE("lnav::document::sections::doc")
+{
+    attr_line_t INPUT = R"(
+
+NAME
+    foo -- bar
+
+SYNOPSIS
+    foo -o -b
+
+DESCRIPTION
+    Lorem ipsum
+
+)";
+
+    auto meta = lnav::document::discover_structure(INPUT, line_range{0, -1});
 
     meta.m_sections_tree.visit_all([](const auto& intv) {
         auto ser = intv.value.match(
@@ -107,7 +139,7 @@ TEST_CASE("lnav::document::sections::empty")
 
 TEST_CASE("lnav::document::sections::sql")
 {
-    static const std::string INPUT
+    attr_line_t INPUT
         = R"(2022-06-03T22:05:58.186Z verbose -[35642] [Originator@6876 sub=Default] [VdbStatement]Executing SQL:
 -->       INSERT INTO PM_CLUSTER_DRAFT_VALIDATION_STATE
 -->         (draft_id, errors, hosts) VALUES (?::integer, ?::jsonb, ARRAY[]::text[])
@@ -116,7 +148,7 @@ TEST_CASE("lnav::document::sections::sql")
 -->
 )";
 
-    auto meta = lnav::document::discover_structure(INPUT);
+    auto meta = lnav::document::discover_structure(INPUT, line_range{0, -1});
 
     meta.m_sections_tree.visit_all([](const auto& intv) {
         auto ser = intv.value.match(

test/expected/expected.am

@@ -2,6 +2,8 @@
 EXPECTED_FILES = \
     $(srcdir)/%reldir%/test_cli.sh_17a68b798354f9a6cdfab372006caeb74038d15c.err \
     $(srcdir)/%reldir%/test_cli.sh_17a68b798354f9a6cdfab372006caeb74038d15c.out \
+    $(srcdir)/%reldir%/test_cli.sh_5524542b1a6954ff9741155101497270a2f0c557.err \
+    $(srcdir)/%reldir%/test_cli.sh_5524542b1a6954ff9741155101497270a2f0c557.out \
     $(srcdir)/%reldir%/test_cli.sh_97e19b9ff3775d84074455a2e8993a0611b1c269.err \
     $(srcdir)/%reldir%/test_cli.sh_97e19b9ff3775d84074455a2e8993a0611b1c269.out \
     $(srcdir)/%reldir%/test_cli.sh_a1a09f890f4604309d0a81bbbec8e50fb7d5e887.err \

test/expected/test_cli.sh_5524542b1a6954ff9741155101497270a2f0c557.out

@@ -0,0 +1 @@
+a a a

test/expected/test_cmds.sh_b6a3bb78e9d60e5e1f5ce5b18e40d2f1662707ab.out

@@ -111,7 +111,7 @@ On color displays, the lines will be highlighted as follows:
 
  • Errors will be colored in red;
  • warnings will be yellow;
- • boundaries between days will be underlined; and
+ • boundaries between days will be underlined; and
  • various color highlights will be applied to: IP
    addresses, SQL keywords, XML tags, file and line numbers
    in Java backtraces, and quoted strings.
@@ -425,7 +425,7 @@ can always use  q  to pop the top view off of the stack.
  CTRL+], ESCAPE       Abort command-line entry started with  / ,  : ,  ;
                       , or  | .
 
- ▌Note: The regular expression format used by lnav is PCRE[1]  
+ ▌Note: The regular expression format used by lnav is PCRE[1]  
  ▌(Perl-Compatible Regular Expressions).                       
  ▌                                                             
  ▌ ▌[1] - http://perldoc.perl.org/perlre.html                  
@@ -664,13 +664,13 @@ commands, which is especially useful when scripting lnav.
 
 For more information, visit the lnav website at:
 
-http://lnav.org[1]
+http://lnav.org[1]
 
  ▌[1] - http://lnav.org 
 
 For support questions, email:
 
-lnav@googlegroups.com[1] support@lnav.org[2]
+lnav@googlegroups.com[1] support@lnav.org[2]
 
  ▌[1] - mailto:lnav@googlegroups.com 
  ▌[2] - mailto:support@lnav.org      

test/expected/test_cmds.sh_b9f8bf53ec2736432eb048d94a391175eb4dc5bf.out

@@ -1,2 +1,2 @@
-Feb  3 09:23:38 veridian automount[7998]: lookup(file): lookup for foobar failed
+Feb  3 09:23:38 veridian automount[7998]: lookup(file): lookup for foobar failed
 Jan  3 09:23:38 veridian automount[16442]: attempting to mount entry /auto/opt

test/expected/test_meta.sh_41f643bb4f720130625b042563e9591bee4ae588.out

@@ -1,2 +1,2 @@
-log_tags 
+log_tags 
 ["#foo"] 

test/expected/test_meta.sh_c75128169049bd88d5eaf8b84a7f617e5ae5d936.out

@@ -1,4 +1,4 @@
-log_line  log_comment  log_tags 
+log_line  log_comment  log_tags 
        0 Hello, World! ["#foo"] 
        1 <NULL>        <NULL>   
        2 <NULL>        <NULL>   

test/expected/test_sql.sh_13429aed81d7edfd47b57e9cdb8a25c43aff35c4.out

@@ -1,2 +1,2 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters   c_ip    cs_bytes cs_method cs_uri_query   cs_uri_stem   cs_username cs_vars cs_version s_app s_core s_pid s_req s_runtime s_switches s_worker_reqs sc_bytes sc_header_bytes sc_headers sc_status 
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters   c_ip    cs_bytes cs_method cs_uri_query   cs_uri_stem   cs_username cs_vars cs_version s_app s_core s_pid s_req s_runtime s_switches s_worker_reqs sc_bytes sc_header_bytes sc_headers sc_status 
        0   <NULL> 2016-03-13 22:49:12.000              0 info             0      <NULL>   <NULL>      <NULL> 127.0.0.1      696 POST            <NULL> /update_metrics                  38 HTTP/1.1   0     3      88185     1     0.129          1             1       47             378          9       200 

test/expected/test_sql.sh_1cbb81cfe40ee16332c5c775a74d06b945aa65c2.out

@@ -1,3 +1,3 @@
-id first_name last_name age 
+id first_name last_name age 
  0 Phil       Myman      30 
  1 Lem        Hewitt     35 

test/expected/test_sql.sh_2f15b8a38673ac4db45dc6ed2eafe609c332575b.out

@@ -1,3 +1,3 @@
-id first_name last_name age 
+id first_name last_name age 
  0 Phil       Myman      30 
  1 Lem        Hewitt     35 

test/expected/test_sql.sh_31df37f254255115611fc321b63374a2fa4a1cd5.out

@@ -1,2 +1,2 @@
-                                                replicate('foobar', 120)                                                 
+                                                replicate('foobar', 120)                                                 
 foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar⋯oobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar 

test/expected/test_sql.sh_56047c9470e515bc3e3709354c01e5d50462cde7.out

@@ -1,2 +1,2 @@
-log_top_line() 
+log_top_line() 
         <NULL> 

test/expected/test_sql.sh_62eb85c9569e71a630d72065238559528a16114c.out

@@ -1,2 +1,2 @@
-filepath 
+filepath 
 foo      

test/expected/test_sql.sh_753c343a256d1286750314957d1b4e155464e03e.out

@@ -1,2 +1,2 @@
-log_top_datetime() 
+log_top_datetime() 
             <NULL> 

test/expected/test_sql.sh_764306f0e5f610ba71f521ba3d19fe158ece0ba5.out

@@ -1,2 +1,2 @@
-  col_0   
+  col_0   
 eth0.IPv4 

test/expected/test_sql.sh_85fe3b9803254ea54b864d4865d7bd4d7a7f86c6.out

@@ -1,4 +1,4 @@
 192.168.202.254 - - [20/Jul/2009:23:00:26 +0000] "GET /vmw/cgi/tramp HTTP/1.0" 200 134 "-" "gPXE/0.9.7"
 192.168.202.254 - - [20/Jul/2009:23:00:29 +0000] "GET /vmw/vSphere/default/vmkboot.gz HTTP/1.0" 404 46210 "-" "gPXE/0.9.7"
-192.168.202.254 - - [20/Jul/2009:23:00:29 +0000] "GET /vmw/vSphere/default/vmkernel.gz HTTP/1.0" 200 78929 "-" "gPXE/0.9.7"
+192.168.202.254 - - [20/Jul/2009:23:00:29 +0000] "GET /vmw/vSphere/default/vmkernel.gz HTTP/1.0" 200 78929 "-" "gPXE/0.9.7"
 10.112.81.15 - - [15/Feb/2013:06:01:31 +0000] "-" 400 0 "-" "-"

test/expected/test_sql.sh_8ee288f1508eaab0367e465e9f382e848f3282aa.out

@@ -1,4 +1,4 @@
-       log_time         
+       log_time         
 2009-07-20 22:59:26.000 
 2009-07-20 22:59:29.000 
 2009-07-20 22:59:29.000 

test/expected/test_sql.sh_9b03e9f7a1bc35e408b3a17ee90cfdadea164df6.out

@@ -1,4 +1,4 @@
-Min: 0                    1-23    24-48    49+                   Max: 291690
+Min: 0                    1-23    24-48    49+                   Max: 291690
  Thu Nov 03 00:15:00                                                          
 ▲ 70 values in the range 0.00-3788.18 
  Thu Nov 03 00:20:00                                                          

test/expected/test_sql.sh_ae7b1f1684e14bf9c16e0d789257b6ef57cfb2b1.out

@@ -1,2 +1,2 @@
-       log_time         
+       log_time         
 2009-07-20 22:59:26.000 

test/expected/test_sql.sh_afe9cdc4898df5c4e112c13dfe3db6dc089c0d7c.out

@@ -1,4 +1,4 @@
-               log_body                
+               log_body                
 lookup(file): lookup for foobar failed 
 attempting to mount entry /auto/opt    
 lookup(file): lookup for opt failed    

test/expected/test_sql.sh_b085d26043f9661d70f82cb90ecb3c5245d25eac.out

@@ -1,4 +1,4 @@
-       log_time         
+       log_time         
 2009-07-20 22:59:26.000 
 2009-07-20 22:59:29.000 
 2009-07-20 22:59:29.000 

test/expected/test_sql.sh_b5aa0561a65de7e8e22085db184c72a94b1a89a9.out

@@ -1,2 +1,2 @@
-                                                     log_body                                                      
+                                                     log_body                                                      
 timstack : TTY=pts/6 ; PWD=/auto/wstimstack/rpms/lbuild/test ; USER=root ; COMMAND=/usr/bin/tail /var/log/messages 

test/expected/test_sql.sh_c353ef036c505b75996252138fbd4c8d22e8149c.out

@@ -1,4 +1,4 @@
-       log_time         
+       log_time         
 2009-07-20 22:59:26.000 
 2009-07-20 22:59:29.000 
 2009-07-20 22:59:29.000 

test/expected/test_sql.sh_c5b8da04734fadf3b9eea80e0af997e38e0fb811.out

@@ -1,3 +1,3 @@
-log_line   col_0   
+log_line   col_0   
        0 eth0.IPv4 
        7 eth0.IPv4 

test/expected/test_sql.sh_c7e1dbf4605914720b55787785abfafdf2c4178a.out

@@ -1,2 +1,2 @@
-  log_top_datetime()    
+  log_top_datetime()    
 2016-03-13 22:49:15.000 

test/expected/test_sql.sh_cc77a633a66d1778705a34e3657737547b3fb08d.out

@@ -1,2 +1,2 @@
-log_top_line() 
+log_top_line() 
              2 

test/expected/test_sql_indexes.sh_1614ebb5e2e83bab11023354dea8a0885ddf64b4.out

@@ -1,3 +1,3 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format 
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format 
        1   <NULL> 2009-07-20 22:59:29.000           3000 error            0      <NULL>   <NULL>      <NULL>                
        3   <NULL> 2013-02-15 06:00:31.000   112777262000 error            0      <NULL>   <NULL>      <NULL>                

test/expected/test_sql_indexes.sh_541a8e35f34a206e340a3880128b6ce137847872.out

@@ -1,4 +1,4 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters      c_ip       cs_method cs_referer cs_uri_query           cs_uri_stem            cs_user_agent cs_username cs_version sc_bytes sc_status cs_host   log_unique_path    
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters      c_ip       cs_method cs_referer cs_uri_query           cs_uri_stem            cs_user_agent cs_username cs_version sc_bytes sc_status cs_host   log_unique_path    
        0   <NULL> 2009-07-20 22:59:26.000              0 info             0      <NULL>   <NULL>      <NULL> 192.168.202.254 GET       -                <NULL> /vmw/cgi/tramp                   gPXE/0.9.7    -           HTTP/1.0        134       200  <NULL> logfile_access_log.0 
        1   <NULL> 2009-07-20 22:59:29.000           3000 error            0      <NULL>   <NULL>      <NULL> 192.168.202.254 GET       -                <NULL> /vmw/vSphere/default/vmkboot.gz  gPXE/0.9.7    -           HTTP/1.0      46210       404  <NULL> logfile_access_log.0 
        2   <NULL> 2009-07-20 22:59:29.000              0 info             0      <NULL>   <NULL>      <NULL> 192.168.202.254 GET       -                <NULL> /vmw/vSphere/default/vmkernel.gz gPXE/0.9.7    -           HTTP/1.0      78929       200  <NULL> logfile_access_log.0 

test/expected/test_sql_indexes.sh_59a1497c13a5e09bc8f95ef02552b2835ebea6e5.out

@@ -1,2 +1,2 @@
-$id $parent $notused                 replace($detail, 'SCAN TABLE', 'SCAN')                  
+$id $parent $notused                 replace($detail, 'SCAN TABLE', 'SCAN')                  
   2       0        0 SCAN all_logs VIRTUAL TABLE INDEX 1:SEARCH all_logs USING log_level < ? 

test/expected/test_sql_indexes.sh_69fd19d56a8cd1fc9c7eb9351270eabb491f8233.out

@@ -1,4 +1,4 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format log_format 
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format log_format 
        0   <NULL> 2009-07-20 22:59:26.000              0 info             0      <NULL>   <NULL>      <NULL>                access_log 
        1   <NULL> 2009-07-20 22:59:29.000           3000 error            0      <NULL>   <NULL>      <NULL>                access_log 
        2   <NULL> 2009-07-20 22:59:29.000              0 info             0      <NULL>   <NULL>      <NULL>                access_log 

test/expected/test_sql_indexes.sh_6f707b6e856dbaab6f95e7e89b98dc3652021f85.out

@@ -1,3 +1,3 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format 
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format 
        0   <NULL> 2009-07-20 22:59:26.000              0 info             0      <NULL>   <NULL>      <NULL>                
        2   <NULL> 2009-07-20 22:59:29.000              0 info             0      <NULL>   <NULL>      <NULL>                

test/expected/test_sql_indexes.sh_b615b6737b1e0d383c8ce4a1db56332f11dbc158.out

@@ -1,2 +1,2 @@
-$id $parent $notused                  replace($detail, 'SCAN TABLE', 'SCAN')                  
+$id $parent $notused                  replace($detail, 'SCAN TABLE', 'SCAN')                  
   2       0        0 SCAN all_logs VIRTUAL TABLE INDEX 1:SEARCH all_logs USING log_format = ? 

test/expected/test_sql_indexes.sh_dab07d8de7728752ae938a174468d75e85f3ae7e.out

@@ -1,2 +1,2 @@
-$id $parent $notused                    replace($detail, 'SCAN TABLE', 'SCAN')                     
+$id $parent $notused                    replace($detail, 'SCAN TABLE', 'SCAN')                     
   2       0        0 SCAN access_log VIRTUAL TABLE INDEX 1:SEARCH access_log USING log_path GLOB ? 

test/expected/test_sql_indexes.sh_f7681c234d4f60df16c997a05163aeb058c52870.out

@@ -1,4 +1,4 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format 
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters log_msg_format 
        0   <NULL> 2009-07-20 22:59:26.000              0 info             0      <NULL>   <NULL>      <NULL>                
        1   <NULL> 2009-07-20 22:59:29.000           3000 error            0      <NULL>   <NULL>      <NULL>                
        2   <NULL> 2009-07-20 22:59:29.000              0 info             0      <NULL>   <NULL>      <NULL>                

test/expected/test_sql_search_table.sh_1a0d872ebc492fcecb2e79a0993170d5fc771a5b.out

@@ -1,2 +1,2 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters  comp   opid    tid   user   item  prc  reason  req    sid         src        sub   vpxa_update  line   file  match_index   lro_id       entity                 operation                           SessionId                           SessionSubId             
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters  comp   opid    tid   user   item  prc  reason  req    sid         src        sub   vpxa_update  line   file  match_index   lro_id       entity                 operation                           SessionId                           SessionSubId             
        2   <NULL> 2022-06-02 11:58:12.376            182 info             0      <NULL>   <NULL>      <NULL> <NULL> e3979f6 45709 <NULL> <NULL> vpxd <NULL> <NULL> <NULL> Originator@6876 vpxLro      <NULL> <NULL> <NULL>           0 lro-846064 SessionManager vim.SessionManager.sessionIsActive 52626140-422b-6287-b4e4-344192c6a01d 523e0a4b-6e83-6bcd-9342-22502dd89866 

test/expected/test_sql_search_table.sh_5aaae556ecb1661602f176215e28f661d3404032.out

@@ -1,4 +1,4 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters match_index user pid cpu_pct mem_pct vsz rss tty stat start_time cpu_time    cmd      cmd_name  cmd_args 
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters match_index user pid cpu_pct mem_pct vsz rss tty stat start_time cpu_time    cmd      cmd_name  cmd_args 
        0   <NULL> 2022-06-02 00:01:01.000              0 info             0      <NULL>   <NULL>      <NULL>           1 root   2       0       0   0   0 ?   S    Jun01      0:00     [kthreadd] [kthreadd]   <NULL> 
       12   <NULL> 2022-06-02 00:02:01.000          60000 info             0      <NULL>   <NULL>      <NULL>           1 root   2       0       0   0   0 ?   S    Jun01      0:00     [kthreadd] [kthreadd]   <NULL> 
       30   <NULL> 2022-06-02 00:03:01.000          60000 info             0      <NULL>   <NULL>      <NULL>           1 root   2       0       0   0   0 ?   S    Jun01      0:00     [kthreadd] [kthreadd]   <NULL> 

test/expected/test_sql_search_table.sh_df0fd242f57a96d40f466493938cda0789a094fa.out

@@ -1,4 +1,4 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters match_index user pid cpu_pct mem_pct  vsz   rss  tty stat start_time cpu_time                              cmd                                      cmd_name                        cmd_args                  
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters match_index user pid cpu_pct mem_pct  vsz   rss  tty stat start_time cpu_time                              cmd                                      cmd_name                        cmd_args                  
        0   <NULL> 2022-06-02 00:01:01.000              0 info             0      <NULL>   <NULL>      <NULL>           0 root   1       0       0 158392 7792 ?   Ss   Jun01      0:14     /lib/systemd/systemd --switched-root --system --deserialize 16 /lib/systemd/systemd   --switched-root --system --deserialize 16 
        0   <NULL> 2022-06-02 00:01:01.000              0 info             0      <NULL>   <NULL>      <NULL>           1 root   2       0       0      0    0 ?   S    Jun01      0:00     [kthreadd]                                                     [kthreadd]             <NULL>                                    
        0   <NULL> 2022-06-02 00:01:01.000              0 info             0      <NULL>   <NULL>      <NULL>           2 root   3       0       0      0    0 ?   I<   Jun01      0:00     [rcu_gp]                                                       [rcu_gp]               <NULL>                                    

test/expected/test_sql_search_table.sh_ef9373a76853f345d06234f6e0fe11b5d40da27b.out

@@ -1,4 +1,4 @@
-log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters  comp             opid              tid   user   item  prc  reason  req    sid         src        sub   vpxa_update  line   file  match_index   lro_id         entity                       operation                              SessionId                           SessionSubId                                                                     log_body                                                         
+log_line log_part        log_time         log_idle_msecs log_level log_mark log_comment log_tags log_filters  comp             opid              tid   user   item  prc  reason  req    sid         src        sub   vpxa_update  line   file  match_index   lro_id         entity                       operation                              SessionId                           SessionSubId                                                                     log_body                                                         
        0   <NULL> 2022-06-02 11:58:12.193              0 info             0      <NULL>   <NULL>      <NULL> <NULL> 7e1280cf                    45715 <NULL> <NULL> vpxd <NULL> <NULL> <NULL> Originator@6876 vpxLro      <NULL> <NULL> <NULL>           0 lro-846063 SessionManager     vim.SessionManager.sessionIsActive        528e6e0c-246d-58b5-3234-278c6e0c5d0d 52c289ac-2563-48d5-8a8e-f178da022c0d [VpxLRO] -- BEGIN lro-846063 -- SessionManager -- vim.Sessio⋯8b5-3234-278c6e0c5d0d(52c289ac-2563-48d5-8a8e-f178da022c0d) 
        2   <NULL> 2022-06-02 11:58:12.376            182 info             0      <NULL>   <NULL>      <NULL> <NULL> e3979f6                     45709 <NULL> <NULL> vpxd <NULL> <NULL> <NULL> Originator@6876 vpxLro      <NULL> <NULL> <NULL>           0 lro-846064 SessionManager     vim.SessionManager.sessionIsActive        52626140-422b-6287-b4e4-344192c6a01d 523e0a4b-6e83-6bcd-9342-22502dd89866 [VpxLRO] -- BEGIN lro-846064 -- SessionManager -- vim.Sessio⋯287-b4e4-344192c6a01d(523e0a4b-6e83-6bcd-9342-22502dd89866) 
        4   <NULL> 2022-06-02 11:58:12.623            246 info             0      <NULL>   <NULL>      <NULL> <NULL> l3wrhr4o-cbf-h5:70001034-60 47524 <NULL> <NULL> vpxd <NULL> <NULL> <NULL> Originator@6876 vpxLro      <NULL> <NULL> <NULL>           0 lro-846066 ChangeLogCollector vim.cdc.ChangeLogCollector.waitForChanges 526861fc-0c28-1930-ae5e-d8c2772bf8c2 52a7a308-9646-c054-f1e7-16131c1a7db6 [VpxLRO] -- BEGIN lro-846066 -- ChangeLogCollector -- vim.c⋯1930-ae5e-d8c2772bf8c2(52a7a308-9646-c054-f1e7-16131c1a7db6) 

test/expected/test_sql_views_vtab.sh_62d15cb9d5a9259f198aa01ca8ed200d6da38d68.out

@@ -1,3 +1,3 @@
-view_name filter_id enabled type language pattern 
+view_name filter_id enabled type language pattern 
 log               1       1 in   regex    vmk     
 log               2       1 in   regex    vmk1    

test/expected/test_sql_views_vtab.sh_6ffd89498b9a7758ded6717148fc2ce77a12621b.out

@@ -1,2 +1,2 @@
-search 
+search 
 warn   

test/expected/test_sql_views_vtab.sh_81dc3eb51ec4dc3066a2365524001242c423a9cf.out

@@ -1,2 +1,2 @@
-view_name filter_id enabled type language pattern 
+view_name filter_id enabled type language pattern 
 log               0       1 out  sql      1       

test/expected/test_sql_views_vtab.sh_a2c0f0e51b3f85ea2a05ecdcacaad962b4fe5d4f.out

@@ -1,2 +1,2 @@
-view_name filter_id enabled type language pattern 
+view_name filter_id enabled type language pattern 
 log               1       1 in   regex    vmk     

test/expected/test_text_file.sh_5b51b55dff7332c5bee2c9b797c401c5614d574a.out

@@ -1,4 +1,4 @@
-Build[1][2] Docs[3][4] Coverage Status[5][6] lnav[7][8]
+Build[1][2] Docs[3][4] Coverage Status[5][6] lnav[7][8]
 
  ▌[1] - https://github.com/tstack/lnav/workflows/ci-build/badge.svg               
  ▌[2] - https://github.com/tstack/lnav/actions?query=workflow%3Aci-build          
@@ -9,7 +9,7 @@ Build[1][2] Docs[3][4] Coverage Status[5][6] lnav[7][8]
  ▌[7] - https://snapcraft.io//lnav/badge.svg                                      
  ▌[8] - https://snapcraft.io/lnav                                                 
 
-This is the source repository for lnav, visit https://lnav.org[1] for
+This is the source repository for lnav, visit https://lnav.org[1] for
 a high level overview.
 
  ▌[1] - https://lnav.org 
@@ -26,7 +26,7 @@ to no setup.
 The following screenshot shows a syslog file. Log lines are displayed
 with highlights. Errors are red and warnings are yellow.
 
-Screenshot[1][2]
+Screenshot[1][2]
 
  ▌[1] - file://{top_srcdir}/docs/assets/images/lnav-syslog-thumb.png 
  ▌[2] - file://{top_srcdir}/docs/assets/images/lnav-syslog.png       
@@ -43,8 +43,8 @@ Screenshot[1][2]
 
 Installation
 
-Download a statically-linked binary for Linux/MacOS from the release
-page[1]
+Download a statically-linked binary for Linux/MacOS from the release
+page[1]
 
  ▌[1] - https://github.com/tstack/lnav/releases/latest#release-artifacts 
 
@@ -102,9 +102,9 @@ log lines fed into  lnav  via  journalctl 's [
 
 Links
 
- • Main Site[1]
- • Documentation[2] on Read the Docs
- • Internal Architecture[3]
+ • Main Site[1]
+ • Documentation[2] on Read the Docs
+ • Internal Architecture[3]
 
  ▌[1] - https://lnav.org                                 
  ▌[2] - https://docs.lnav.org                            
@@ -112,7 +112,7 @@ log lines fed into  lnav  via  journalctl 's [
 
 Contributing
 
- • Become a Sponsor on GitHub[1]
+ • Become a Sponsor on GitHub[1]
 
  ▌[1] - https://github.com/sponsors/tstack 
 
@@ -151,7 +151,7 @@ Run  ./autogen.sh  if compiling from a cloned repository.
 
 See Also
 
-Angle-grinder[1] is a tool to slice and dice log files on the
+Angle-grinder[1] is a tool to slice and dice log files on the
 command-line. If you're familiar with the SumoLogic query language,
 you might find this tool more comfortable to work with.
 

test/test_ansi_scrubber.cc

@@ -45,27 +45,34 @@ using namespace std;
 int
 main(int argc, char* argv[])
 {
+    {
+        std::string boldish = "a\ba _\ba a\b_ b";
+        string_attrs_t sa;
+
+        sa.clear();
+        scrub_ansi_string(boldish, &sa);
+        assert(boldish == "a a a b");
+        for (const auto& attr : sa) {
+            printf("attr %d:%d %s\n",
+                   attr.sa_range.lr_start,
+                   attr.sa_range.lr_end,
+                   attr.sa_type->sat_name);
+            if (attr.sa_type == &SA_ORIGIN_OFFSET) {
+                printf("  value: %d\n", attr.sa_value.get<int64_t>());
+            }
+        }
+    }
+
     string_attrs_t sa;
     string str_cp;
 
     str_cp = "Hello, World!";
-    scrub_ansi_string(str_cp, sa);
+    scrub_ansi_string(str_cp, &sa);
 
     assert(str_cp == "Hello, World!");
     assert(sa.empty());
 
     str_cp = "Hello\x1b[44;m, \x1b[33;mWorld\x1b[0;m!";
-    scrub_ansi_string(str_cp, sa);
+    scrub_ansi_string(str_cp, &sa);
     assert(str_cp == "Hello, World!");
-
-    assert(sa[0].sa_range.lr_start == 5);
-    assert(sa[0].sa_range.lr_end == 7);
-    assert(sa[0].sa_type == &VC_STYLE);
-    assert(sa[0].sa_value.get<text_attrs>().ta_bg_color.value() == COLOR_BLUE);
-
-    assert(sa[1].sa_range.lr_start == 7);
-    assert(sa[1].sa_range.lr_end == 12);
-    assert(sa[1].sa_type == &VC_STYLE);
-    assert(sa[1].sa_value.get<text_attrs>().ta_fg_color.value()
-           == COLOR_YELLOW);
 }

test/test_cli.sh

@@ -14,3 +14,6 @@ mkdir -p nested/sub1/sub2
 echo "2021-07-03T21:49:29 Test" > nested/sub1/sub2/test.log
 
 run_cap_test ${lnav_test} -nr nested
+
+printf "a\ba _\ba a\b_" | run_cap_test env TEST_COMMENT="overstrike bold" \
+    ${lnav_test} -n