Robot
/
lnav
mirror of https://github.com/tstack/lnav.git
2
0
Fork 0
Code Issues Releases Wiki Activity

[cmd] add :write-jsonlines-to command

This commit is contained in:
Timothy Stack 2020-08-21 23:18:51 -07:00
parent 2a37a9ce7f
commit d695febe14
6 changed files with 99 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
NEWS
View File

@ -23,6 +23,8 @@ lnav v0.8.6:
raise an error in an lnav script to stop execution and notify the user.
* Added the json_concat() function to make it easier to append/concatenate
values onto arrays.
* Added the ":write-jsonlines-to" command that writes the result of a SQL
query to a file in the JSON Lines format.
Interface Changes:
* Data piped into lnav is no longer dumped to the console after exit.

6
src/help.txt
View File

@ -662,6 +662,12 @@ COMMANDS
to write to standard out. Use '-' to write the data to
the terminal.
write-jsonlines-to <file>
Write the results of a SQL query to a JSON-Lines-formatted
file. Each row in the result set will be a single line of
JSON in the output and each column will be a property in
that object. Use '-' to write the data to the terminal.
pipe-to <shell-cmd>
Send the currently marked lines to the given shell command
for processing and open the resulting file for viewing.

60
src/internals/cmd-ref.rst
View File

@ -49,7 +49,7 @@
**See Also:**
:ref:`echo`, :ref:`eval`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
:ref:`echo`, :ref:`eval`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
----
@ -75,7 +75,7 @@
**See Also:**
:ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
:ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
----
@ -222,7 +222,7 @@
**See Also:**
:ref:`create_search_table`, :ref:`create_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
:ref:`create_search_table`, :ref:`create_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
----
@ -249,7 +249,7 @@
**See Also:**
:ref:`create_logline_table`, :ref:`create_logline_table`, :ref:`delete_search_table`, :ref:`delete_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
:ref:`create_logline_table`, :ref:`create_logline_table`, :ref:`delete_search_table`, :ref:`delete_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
----
@ -312,7 +312,7 @@
**See Also:**
:ref:`create_logline_table`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`create_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
:ref:`create_logline_table`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`create_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
----
@ -338,7 +338,7 @@
**See Also:**
:ref:`create_logline_table`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`create_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
:ref:`create_logline_table`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`create_search_table`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`
----
@ -430,7 +430,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`eval`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`eval`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
----
@ -502,7 +502,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`echo`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`echo`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
----
@ -876,7 +876,7 @@
**See Also:**
:ref:`append_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
:ref:`append_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
----
@ -902,7 +902,7 @@
**See Also:**
:ref:`append_to`, :ref:`pipe_line_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
:ref:`append_to`, :ref:`pipe_line_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_to`
----
@ -979,7 +979,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
----
@ -1377,7 +1377,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
----
@ -1403,7 +1403,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
----
@ -1429,7 +1429,33 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
----
.. _write_jsonlines_to:
:write-jsonlines-to *path*
^^^^^^^^^^^^^^^^^^^^^^^^^^
Write SQL results to the given file in JSON Lines format
**Parameters:**
* **path** --- The path to the file to write
**Examples:**
To write SQL results as JSON Lines to /tmp/table.json:
.. code-block:: lnav
:write-jsonlines-to /tmp/table.json
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
----
@ -1455,7 +1481,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_screen_to`, :ref:`write_to`, :ref:`write_to`
----
@ -1481,7 +1507,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_to`, :ref:`write_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`create_logline_table`, :ref:`create_search_table`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_to`, :ref:`write_to`
----
@ -1507,7 +1533,7 @@
**See Also:**
:ref:`alt_msg`, :ref:`append_to`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`
:ref:`alt_msg`, :ref:`append_to`, :ref:`echo`, :ref:`eval`, :ref:`pipe_line_to`, :ref:`pipe_to`, :ref:`redirect_to`, :ref:`redirect_to`, :ref:`write_cols_to`, :ref:`write_cols_to`, :ref:`write_csv_to`, :ref:`write_csv_to`, :ref:`write_json_to`, :ref:`write_json_to`, :ref:`write_jsonlines_to`, :ref:`write_jsonlines_to`, :ref:`write_raw_to`, :ref:`write_raw_to`, :ref:`write_screen_to`, :ref:`write_screen_to`
----

31
src/lnav_commands.cc
View File

@ -658,6 +658,7 @@ static Result<string, string> com_save_to(exec_context &ec, string cmdline, vect
if (args[0] == "write-csv-to" ||
args[0] == "write-json-to" ||
args[0] == "write-jsonlines-to" ||
args[0] == "write-cols-to") {
if (dls.dls_headers.empty()) {
return ec.make_error("no query result to write, use ';' to execute a query");
@ -796,6 +797,23 @@ static Result<string, string> com_save_to(exec_context &ec, string cmdline, vect
}
}
}
else if (args[0] == "write-jsonlines-to") {
yajlpp_gen gen;
yajl_gen_config(gen, yajl_gen_beautify, 0);
yajl_gen_config(gen,
yajl_gen_print_callback, yajl_writer, outfile);
for (size_t row = 0; row < dls.dls_rows.size(); row++) {
if (ec.ec_dry_run && row > 10) {
break;
}
json_write_row(gen, row);
yajl_gen_reset(gen, "\n");
line_count += 1;
}
}
else if (args[0] == "write-screen-to") {
bool wrapped = tc->get_word_wrap();
vis_line_t orig_top = tc->get_top();
@ -4179,6 +4197,19 @@ readline_context::command_t STD_COMMANDS[] = {
"/tmp/table.json"
})
},
{
"write-jsonlines-to",
com_save_to,
help_text(":write-jsonlines-to")
.with_summary("Write SQL results to the given file in JSON Lines format")
.with_parameter(help_text("path", "The path to the file to write"))
.with_tags({"io", "scripting", "sql"})
.with_example({
"To write SQL results as JSON Lines to /tmp/table.json",
"/tmp/table.json"
})
},
{
"write-cols-to",
com_save_to,

6
test/expected_help.txt
View File

@ -662,6 +662,12 @@ COMMANDS
to write to standard out. Use '-' to write the data to
the terminal.
write-jsonlines-to <file>
Write the results of a SQL query to a JSON-Lines-formatted
file. Each row in the result set will be a single line of
JSON in the output and each column will be a property in
that object. Use '-' to write the data to the terminal.
pipe-to <shell-cmd>
Send the currently marked lines to the given shell command
for processing and open the resulting file for viewing.

11
test/test_cmds.sh
View File

@ -574,6 +574,17 @@ check_output "write-json-to is not working" <<EOF
EOF
run_test ${lnav_test} -n \
-c ";select * from access_log" \
-c ':write-jsonlines-to -' \
${test_dir}/logfile_access_log.0
check_output "write-jsonlines-to is not working" <<EOF
{"log_line":0,"log_part":null,"log_time":"2009-07-20 22:59:26.000","log_idle_msecs":0,"log_level":"info","log_mark":0,"log_comment":null,"log_tags":null,"log_filters":null,"c_ip":"192.168.202.254","cs_method":"GET","cs_referer":"-","cs_uri_query":null,"cs_uri_stem":"/vmw/cgi/tramp","cs_user_agent":"gPXE/0.9.7","cs_username":"-","cs_version":"HTTP/1.0","sc_bytes":134,"sc_status":200}
{"log_line":1,"log_part":null,"log_time":"2009-07-20 22:59:29.000","log_idle_msecs":3000,"log_level":"error","log_mark":0,"log_comment":null,"log_tags":null,"log_filters":null,"c_ip":"192.168.202.254","cs_method":"GET","cs_referer":"-","cs_uri_query":null,"cs_uri_stem":"/vmw/vSphere/default/vmkboot.gz","cs_user_agent":"gPXE/0.9.7","cs_username":"-","cs_version":"HTTP/1.0","sc_bytes":46210,"sc_status":404}
{"log_line":2,"log_part":null,"log_time":"2009-07-20 22:59:29.000","log_idle_msecs":0,"log_level":"info","log_mark":0,"log_comment":null,"log_tags":null,"log_filters":null,"c_ip":"192.168.202.254","cs_method":"GET","cs_referer":"-","cs_uri_query":null,"cs_uri_stem":"/vmw/vSphere/default/vmkernel.gz","cs_user_agent":"gPXE/0.9.7","cs_username":"-","cs_version":"HTTP/1.0","sc_bytes":78929,"sc_status":200}
EOF
# By setting the LNAVSECURE mode before executing the command, we will disable
# the access to the write-json-to command and the output would just be the
# actual display of select query rather than json output.