mirror of https://github.com/tstack/lnav.git
[docs] more docs and some simple bug fixes
This commit is contained in:
parent
ba5b89ffe9
commit
94ce0ffd83
|
@ -5,16 +5,18 @@ Extracting Data
|
||||||
**Note**: This feature is still in **BETA**, you should expect bugs and
|
**Note**: This feature is still in **BETA**, you should expect bugs and
|
||||||
incompatible changes in the future.
|
incompatible changes in the future.
|
||||||
|
|
||||||
Log messages contain a good deal of useful data, but it's not always easy to
|
Log messages contain a good deal of useful data, but it's not always easy to get
|
||||||
get at. The log parser built into **lnav** is able to extract data as
|
at. The log parser built into **lnav** is able to extract data as described by
|
||||||
described by log formats as well as discovering data in plain text messages.
|
:ref:`log-formats` as well as discovering data in plain text messages. This data
|
||||||
This data can then be queried and processed using the SQLite front-end that is
|
can then be queried and processed using the SQLite front-end that is also
|
||||||
also incorporated into **lnav**. As an example, the following Syslog message
|
incorporated into **lnav**. As an example, the following Syslog message from
|
||||||
from :cmd:`sudo` will be parsed and several
|
:cmd:`sudo` can be processed to extract several key/value pairs::
|
||||||
|
|
||||||
Jul 31 11:42:26 Example-MacBook-Pro.local sudo[87024]: testuser : TTY=ttys004 ; PWD=/Users/testuser/github/lbuild ; USER=root ; COMMAND=/usr/bin/make install
|
Jul 31 11:42:26 Example-MacBook-Pro.local sudo[87024]: testuser : TTY=ttys004 ; PWD=/Users/testuser/github/lbuild ; USER=root ; COMMAND=/usr/bin/make install
|
||||||
|
|
||||||
|
The data that can be extracted by the parser is viewable directly in **lnav**
|
||||||
|
by pressing the 'p' key. The results will be shown in an overlay like the
|
||||||
|
following::
|
||||||
|
|
||||||
Current Time: 2013-07-31T11:42:26.000 Original Time: 2013-07-31T11:42:26.000 Offset: +0.000
|
Current Time: 2013-07-31T11:42:26.000 Original Time: 2013-07-31T11:42:26.000 Offset: +0.000
|
||||||
Known message fields:
|
Known message fields:
|
||||||
|
@ -27,3 +29,159 @@ from :cmd:`sudo` will be parsed and several
|
||||||
├ PWD = /Users/testuser/github/lbuild
|
├ PWD = /Users/testuser/github/lbuild
|
||||||
├ USER = root
|
├ USER = root
|
||||||
└ COMMAND = /usr/bin/make install
|
└ COMMAND = /usr/bin/make install
|
||||||
|
|
||||||
|
Notice that the parser has detected pairs of the form '<key>=<value>'. The data
|
||||||
|
parser will also look for pairs separated by a colon. If there are no clearly
|
||||||
|
demarcated pairs, then the parser will extract anything that looks like data
|
||||||
|
values and assign them keys of the form 'col_N'. For example, two data values,
|
||||||
|
an IPv4 address and a symbol, will be extracted from the following log
|
||||||
|
messsage::
|
||||||
|
|
||||||
|
Apr 29 08:13:43 sample-centos5 avahi-daemon[2467]: Registering new address record for 10.1.10.62 on eth0.
|
||||||
|
|
||||||
|
Since there are no keys for the values in the message, the parser will assign
|
||||||
|
'col_0' for the IP address and 'col_1' for the symbol, as seen here::
|
||||||
|
|
||||||
|
Current Time: 2013-04-29T08:13:43.000 Original Time: 2013-04-29T08:13:43.000 Offset: +0.000
|
||||||
|
Known message fields:
|
||||||
|
├ log_hostname = sample-centos5
|
||||||
|
├ log_procname = avahi-daemon
|
||||||
|
├ log_pid = 2467
|
||||||
|
Discovered message fields:
|
||||||
|
├ col_0 = 10.1.10.62
|
||||||
|
└ col_1 = eth0
|
||||||
|
|
||||||
|
Now that you have an idea of how the parser works, you can begin to perform
|
||||||
|
queries on the data that is being extracted. The SQLite database engine is
|
||||||
|
embedded into **lnav** and its `Virtual Table
|
||||||
|
<http://www.sqlite.org/vtab.html>`_ mechanism is used to provide a means to
|
||||||
|
process this log data. Each log format has its own table that can be used to
|
||||||
|
access all of the loaded messages that are in that format. For accessing log
|
||||||
|
message content that is more free-form, like the examples given here, the
|
||||||
|
**logline** table can be used. The **logline** table is recreated for each
|
||||||
|
query and is based on the format and pairs discovered in the log message at
|
||||||
|
the top of the display.
|
||||||
|
|
||||||
|
Queries can be performed by pressing the semi-colon (;) key in **lnav**. After
|
||||||
|
pressing the key, the overlay showing any known or discovered fields will be
|
||||||
|
displayed to give you an idea of what data is available. The query can be any
|
||||||
|
`SQL query <http://sqlite.org/lang.html>`_ supported by SQLite. To make
|
||||||
|
analysis easier, **lnav** includes many extra functions for processing strings,
|
||||||
|
paths, and IP addresses. See :ref:`sql-ext` for more information.
|
||||||
|
|
||||||
|
As an example, the simplest query to perform initially would be a "select all",
|
||||||
|
like so::
|
||||||
|
|
||||||
|
select * from logline
|
||||||
|
|
||||||
|
When this query is run against the second example log message given above, the
|
||||||
|
following results are received::
|
||||||
|
|
||||||
|
log_line log_part log_time log_idle_msecs log_level log_hostname log_procname log_pid col_0 col_1
|
||||||
|
|
||||||
|
292 p.0 2013-04-11T16:42:51.000 0 info localhost avahi-daemon 2480 fe80::a00:27ff:fe98:7f6e eth0
|
||||||
|
293 p.0 2013-04-11T16:42:51.000 0 info localhost avahi-daemon 2480 10.0.2.15 eth0
|
||||||
|
330 p.0 2013-04-11T16:47:02.000 0 info localhost avahi-daemon 2480 fe80::a00:27ff:fe98:7f6e eth0
|
||||||
|
336 p.0 2013-04-11T16:47:02.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0
|
||||||
|
343 p.0 2013-04-11T16:47:02.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0
|
||||||
|
370 p.0 2013-04-11T16:59:39.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0
|
||||||
|
377 p.0 2013-04-11T16:59:39.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0
|
||||||
|
382 p.0 2013-04-11T16:59:41.000 0 info localhost avahi-daemon 2480 fe80::a00:27ff:fe98:7f6e eth0
|
||||||
|
401 p.0 2013-04-11T17:20:45.000 0 info localhost avahi-daemon 4247 fe80::a00:27ff:fe98:7f6e eth0
|
||||||
|
402 p.0 2013-04-11T17:20:45.000 0 info localhost avahi-daemon 4247 10.1.10.75 eth0
|
||||||
|
|
||||||
|
735 p.0 2013-04-11T17:41:46.000 0 info sample-centos5 avahi-daemon 2465 fe80::a00:27ff:fe98:7f6e eth0
|
||||||
|
736 p.0 2013-04-11T17:41:46.000 0 info sample-centos5 avahi-daemon 2465 10.1.10.75 eth0
|
||||||
|
781 p.0 2013-04-12T03:32:30.000 0 info sample-centos5 avahi-daemon 2465 10.1.10.64 eth0
|
||||||
|
788 p.0 2013-04-12T03:32:30.000 0 info sample-centos5 avahi-daemon 2465 10.1.10.64 eth0
|
||||||
|
1166 p.0 2013-04-25T10:56:00.000 0 info sample-centos5 avahi-daemon 2467 fe80::a00:27ff:fe98:7f6e eth0
|
||||||
|
1167 p.0 2013-04-25T10:56:00.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.111 eth0
|
||||||
|
1246 p.0 2013-04-26T06:06:25.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.49 eth0
|
||||||
|
1253 p.0 2013-04-26T06:06:25.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.49 eth0
|
||||||
|
1454 p.0 2013-04-28T06:53:55.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.103 eth0
|
||||||
|
1461 p.0 2013-04-28T06:53:55.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.103 eth0
|
||||||
|
|
||||||
|
1497 p.0 2013-04-29T08:13:43.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.62 eth0
|
||||||
|
1504 p.0 2013-04-29T08:13:43.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.62 eth0
|
||||||
|
|
||||||
|
Note that **lnav** is not returning results for all messages that are in this
|
||||||
|
syslog file. Rather, it searches for messages that match the format for the
|
||||||
|
given line and returns only those messages in results. In this case, that
|
||||||
|
format is "Registering new address record for <IP> on <symbol>", which
|
||||||
|
corresponds to the parts of the message that were not recognized as data.
|
||||||
|
|
||||||
|
More sophisticated queries can be done, of course. For example, to find out the
|
||||||
|
frequency of IP addresses mentioned in these messages, you can run::
|
||||||
|
|
||||||
|
SELECT col_0,count(*) FROM logline GROUP BY col_0
|
||||||
|
|
||||||
|
The results for this query are::
|
||||||
|
|
||||||
|
col_0 count(*)
|
||||||
|
|
||||||
|
10.0.2.15 1
|
||||||
|
10.1.10.49 2
|
||||||
|
10.1.10.62 2
|
||||||
|
10.1.10.64 2
|
||||||
|
10.1.10.75 6
|
||||||
|
10.1.10.103 2
|
||||||
|
10.1.10.111 1
|
||||||
|
fe80::a00:27ff:fe98:7f6e 6
|
||||||
|
|
||||||
|
Since this type of query is fairly common, **lnav** includes a "summarize"
|
||||||
|
command that will compute the frequencies of identifiers as well as min, max,
|
||||||
|
average, median, and standard deviation for number columns. In this case, you
|
||||||
|
can run the following to compute the frequencies and return an ordered set of
|
||||||
|
results.
|
||||||
|
|
||||||
|
:summarize col_0
|
||||||
|
|
||||||
|
|
||||||
|
Recognized Data
|
||||||
|
---------------
|
||||||
|
|
||||||
|
When searching for data to extract from log messages, **lnav** looks for the
|
||||||
|
following set of patterns:
|
||||||
|
|
||||||
|
|
||||||
|
Strings
|
||||||
|
Single and double-quoted strings. Example: "The quick brown fox."
|
||||||
|
|
||||||
|
URLs
|
||||||
|
URLs that contain the '://' separator. Example: http://example.com
|
||||||
|
|
||||||
|
Paths
|
||||||
|
File system paths. Examples: /path/to/file, ./relative/path
|
||||||
|
|
||||||
|
MAC Address
|
||||||
|
Ethernet MAC addresses. Example: c4:2c:03:0e:e4:4a
|
||||||
|
|
||||||
|
Hex Dumps
|
||||||
|
A colon-separated string of hex numbers. Example: e8:06:88:ff
|
||||||
|
|
||||||
|
Date/Time
|
||||||
|
Date and time stamps of the form "YYYY-mm-DD" and "HH:MM:SS".
|
||||||
|
|
||||||
|
IP Addresses
|
||||||
|
IPv4 and IPv6 addresses. Examples: 127.0.0.1, fe80::c62c:3ff:fe0e:e44a%en0
|
||||||
|
|
||||||
|
UUID
|
||||||
|
The common formatting for 128-bit UUIDs. Example:
|
||||||
|
0E305E39-F1E9-4DE4-B10B-5829E5DF54D0
|
||||||
|
|
||||||
|
Version Numbers
|
||||||
|
Dot-separated version numbers. Example: 3.7.17
|
||||||
|
|
||||||
|
Numbers
|
||||||
|
Numbers in base ten, hex, and octal formats. Examples: 1234, 0xbeef, 0777
|
||||||
|
|
||||||
|
E-Mail Address
|
||||||
|
Strings that look close to an e-mail address. Example: gary@example.com
|
||||||
|
|
||||||
|
Constants
|
||||||
|
Common constants in languages, like: true, false, null, None.
|
||||||
|
|
||||||
|
Symbols
|
||||||
|
Words that follow the common conventions for symbols in programming
|
||||||
|
languages. For example, containing all capital letters, or separated
|
||||||
|
by colons. Example: SOME_CONSTANT_VALUE, namespace::value
|
||||||
|
|
|
@ -39,7 +39,12 @@ fields:
|
||||||
:description: A longer description of the format.
|
:description: A longer description of the format.
|
||||||
:url: A URL to the definition of the format.
|
:url: A URL to the definition of the format.
|
||||||
|
|
||||||
:regex:
|
:regex: This object contains sub-objects that describe the message formats
|
||||||
|
to match in a log file.
|
||||||
|
|
||||||
|
:pattern: The regular expression that should be used to match log messages.
|
||||||
|
The `PCRE <http://www.pcre.org>`_ library is used by **lnav** to do all
|
||||||
|
regular expression matching.
|
||||||
|
|
||||||
:level-field: The name of the regex capture group that contains the log
|
:level-field: The name of the regex capture group that contains the log
|
||||||
message level.
|
message level.
|
||||||
|
|
|
@ -1,10 +1,12 @@
|
||||||
|
|
||||||
|
.. _sql-ext:
|
||||||
|
|
||||||
SQLite Extensions Reference
|
SQLite Extensions Reference
|
||||||
===========================
|
===========================
|
||||||
|
|
||||||
To make it easier to analyze log data from within **lnav**, there are several
|
To make it easier to analyze log data from within **lnav**, there are several
|
||||||
built-in extensions that provide extra functions and collators beyond those
|
built-in extensions that provide extra functions and collators beyond those
|
||||||
`provided by sqlite <http://www.sqlite.org/lang_corefunc.html>`_. The majority
|
`provided by SQLite <http://www.sqlite.org/lang_corefunc.html>`_. The majority
|
||||||
of the functions are from the
|
of the functions are from the
|
||||||
`extensions-functions.c <http://www.sqlite.org/contrib>`_ file available from
|
`extensions-functions.c <http://www.sqlite.org/contrib>`_ file available from
|
||||||
the `sqlite.org <http://sqlite.org>`_ web site.
|
the `sqlite.org <http://sqlite.org>`_ web site.
|
||||||
|
|
|
@ -57,7 +57,7 @@ static struct {
|
||||||
{ "time", pcrepp(
|
{ "time", pcrepp(
|
||||||
"\\A(\\d?\\d:\\d\\d(:\\d\\d)?(:\\d\\d)?([,.]\\d{3,6})?)\\b"), }, /* XXX be more specific */
|
"\\A(\\d?\\d:\\d\\d(:\\d\\d)?(:\\d\\d)?([,.]\\d{3,6})?)\\b"), }, /* XXX be more specific */
|
||||||
/* { "qual", pcrepp("\\A([^\\s:=]+:[^\\s:=,]+(?!,)(?::[^\\s:=,]+)*)"), }, */
|
/* { "qual", pcrepp("\\A([^\\s:=]+:[^\\s:=,]+(?!,)(?::[^\\s:=,]+)*)"), }, */
|
||||||
{ "ipv6", pcrepp("\\A(::|[:\\da-fA-f\\.]+[a-fA-f\\d])"),
|
{ "ipv6", pcrepp("\\A(::|[:\\da-fA-f\\.]+[a-fA-f\\d](?:%\\w+)?)"),
|
||||||
},
|
},
|
||||||
|
|
||||||
{ "coln", pcrepp("\\A(:)"),
|
{ "coln", pcrepp("\\A(:)"),
|
||||||
|
@ -94,7 +94,7 @@ static struct {
|
||||||
{ "uuid", pcrepp(
|
{ "uuid", pcrepp(
|
||||||
"\\A([0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12})"), },
|
"\\A([0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12})"), },
|
||||||
|
|
||||||
{ "vers", pcrepp("\\A([0-9]+(?:\\.[0-9]+){2,}\\b)"),
|
{ "vers", pcrepp("\\A([0-9]+(?:\\.[0-9]+\\w+?){2,}\\b)"),
|
||||||
},
|
},
|
||||||
{ "oct", pcrepp("\\A(-?0[0-7]+\\b)"),
|
{ "oct", pcrepp("\\A(-?0[0-7]+\\b)"),
|
||||||
},
|
},
|
||||||
|
@ -116,7 +116,7 @@ static struct {
|
||||||
"\\A([^\";\\s:=,\\(\\)\\{\\}\\[\\]\\+#!@%\\^&\\*'\\?<>\\~`\\|\\\\]+"
|
"\\A([^\";\\s:=,\\(\\)\\{\\}\\[\\]\\+#!@%\\^&\\*'\\?<>\\~`\\|\\\\]+"
|
||||||
"(?:::[^\";\\s:=,\\(\\)\\{\\}\\[\\]\\+#!@%\\^&\\*'\\?<>\\~`\\|\\\\]+)*)"),
|
"(?:::[^\";\\s:=,\\(\\)\\{\\}\\[\\]\\+#!@%\\^&\\*'\\?<>\\~`\\|\\\\]+)*)"),
|
||||||
},
|
},
|
||||||
{ "line", pcrepp("\\A(\r?sdfadsf\n|\r|;)"),
|
{ "line", pcrepp("\\A(\r?\n|\r|;)"),
|
||||||
},
|
},
|
||||||
{ "wspc", pcrepp("\\A([ \\r\\t\\n]+)"),
|
{ "wspc", pcrepp("\\A([ \\r\\t\\n]+)"),
|
||||||
},
|
},
|
||||||
|
|
|
@ -1055,10 +1055,12 @@ static void handle_paging_key(int ch)
|
||||||
case 'C':
|
case 'C':
|
||||||
if (lss) {
|
if (lss) {
|
||||||
lss->get_user_bookmarks()[&textview_curses::BM_USER].clear();
|
lss->get_user_bookmarks()[&textview_curses::BM_USER].clear();
|
||||||
tc->reload_data();
|
|
||||||
|
|
||||||
lnav_data.ld_rl_view->set_value("Cleared bookmarks");
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
tc->get_bookmarks()[&textview_curses::BM_USER].clear();
|
||||||
|
tc->reload_data();
|
||||||
|
|
||||||
|
lnav_data.ld_rl_view->set_value("Cleared bookmarks");
|
||||||
break;
|
break;
|
||||||
|
|
||||||
case 'e':
|
case 'e':
|
||||||
|
|
|
@ -201,10 +201,14 @@ static void cleanup_session_data(void)
|
||||||
|
|
||||||
session_info_list.sort();
|
session_info_list.sort();
|
||||||
|
|
||||||
|
int session_loops = 0;
|
||||||
|
|
||||||
while (session_info_list.size() > MAX_SESSION_FILE_COUNT) {
|
while (session_info_list.size() > MAX_SESSION_FILE_COUNT) {
|
||||||
const session_file_info &front = session_info_list.front();
|
const session_file_info &front = session_info_list.front();
|
||||||
|
|
||||||
if (session_count[front.sfi_id] == 1) {
|
session_loops += 1;
|
||||||
|
if (session_loops < MAX_SESSION_FILE_COUNT &&
|
||||||
|
session_count[front.sfi_id] == 1) {
|
||||||
session_info_list.splice(session_info_list.end(),
|
session_info_list.splice(session_info_list.end(),
|
||||||
session_info_list,
|
session_info_list,
|
||||||
session_info_list.begin());
|
session_info_list.begin());
|
||||||
|
|
|
@ -0,0 +1,13 @@
|
||||||
|
IPv4=192.168.1.2 IPv6_LL=fe80::c62c:3ff:fe0e:e44a%en0 IPV6=fdca:a98b:b1bf:e05d:7c49:35b3:5949:ec6f
|
||||||
|
key 0:4 ^--^ IPv4
|
||||||
|
ipv4 5:16 ^---------^ 192.168.1.2
|
||||||
|
val 5:16 ^---------^ 192.168.1.2
|
||||||
|
pair 0:16 ^--------------^ IPv4=192.168.1.2
|
||||||
|
key 17:24 ^-----^ IPv6_LL
|
||||||
|
ipv6 25:53 ^--------------------------^ fe80::c62c:3ff:fe0e:e44a%en0
|
||||||
|
val 25:53 ^--------------------------^ fe80::c62c:3ff:fe0e:e44a%en0
|
||||||
|
pair 17:53 ^----------------------------------^ IPv6_LL=fe80::c62c:3ff:fe0e:e44a%en0
|
||||||
|
key 54:58 ^--^ IPV6
|
||||||
|
ipv6 59:98 ^-------------------------------------^ fdca:a98b:b1bf:e05d:7c49:35b3:5949:ec6f
|
||||||
|
val 59:98 ^-------------------------------------^ fdca:a98b:b1bf:e05d:7c49:35b3:5949:ec6f
|
||||||
|
pair 54:98 ^------------------------------------------^ IPV6=fdca:a98b:b1bf:e05d:7c49:35b3:5949:ec6f
|
Loading…
Reference in New Issue