## Adding new builtin languages for syntax highlighting

Should you find that a particular syntax is not available within `bat` and think it should be included in `bat` by default, you can follow the instructions outlined below.

`bat` uses the excellent [syntect](https://github.com/trishume/syntect) library to highlight source
code. As a basis, syntect uses [Sublime Text](https://www.sublimetext.com/) syntax definitions
in the `.sublime-syntax` format.

**Important:** Before proceeding, verify that the syntax you wish to add meets the [criteria for inclusion](#Criteria-for-inclusion-of-new-syntaxes).

1. Find a Sublime Text syntax for the given language, preferably in a separate Git repository
   which can be included as a submodule (under `assets/syntaxes`) using
   `git submodule add <https github link> ./assets/syntaxes/02_Extra/<repo name>`, replacing
   the contents of the angle brackets as appropriate.

2. If the Sublime Text syntax is only available as a `.tmLanguage` file, open the file in
   Sublime Text and convert it to a `.sublime-syntax` file via *Tools* -> *Developer* ->
   *New Syntax from XXX.tmLanguage...*. Save the new file in the `assets/syntaxes` folder.
   If only `.tmLanguage.json` or `.tmLanguage.yml` file is available, use
   [PackageDev](https://packagecontrol.io/packages/PackageDev) to convert it to `.tmLanguage.plist`
   format and then rename the converted file to `.tmLanguage` file.

3. Run the `assets/create.sh` script. It calls `bat cache --build` to parse all available
   `.sublime-syntax` files and serialize them to a `syntaxes.bin` file.

4. Re-compile `bat`. At compilation time, the `syntaxes.bin` file will be stored inside the
   `bat` binary.

5. Use `bat --list-languages` to check if the new languages are available.

6. Add a syntax test for the new language. See [below](#Syntax-tests) for details.

7. If you send a pull request with your changes, please do *not* include the changed `syntaxes.bin`
   file. A new binary cache file will be created once before every new release of `bat`. This
   avoids bloating the repository size unnecessarily.

### Syntax tests

`bat` has a set of syntax highlighting regression tests in `tests/syntax-tests`. The main idea is
make sure that we do not run into issues we had in the past where either (1) syntax highlighting
for some language is suddenly not working anymore or (2) `bat` suddenly crashes for some input (due
to `regex` incompatibilities between `syntect` and Sublime Text).

In order to add a new test file, please follow these steps (let's take "Ruby" as an example):

1. Make sure that you are running the **latest version of `bat`** and that `bat` is available on
   the path. If you are creating a syntax test for a new builtin syntax (see above), make sure that
   your version of `bat` already has the new syntax builtin.
2. Find an example Ruby source file or write one yourself. If possible, the file should aim to be
   "comprehensive" (i.e. include a lot of the possible syntax), but this is not strictly necessary.
   A simple file is better than none at all. Also, the files shouldn't be gigantic.
3. Save the file in `tests/syntax-tests/source/Ruby` (adapt for your language). The file name could
   be `test.rb` (adapt extension) but can also be adapted if that is necessary in order for `bat` to
   highlight it correctly (e.g. `Makefile`).
4. If you have copied the file from somewhere else, please make sure that the file *may* be copied
   under the respective license and that the license is compatible with `bat`s license. If it
   requires attribution, please add a `LICENSE.md` in the same folder with a text like this:
    ```
    The `test.rb` file has been added from [enter source here] under the following license:

    [add license text here]
    ```
5. Go to `tests/syntax-tests` and run the `update.sh` Bash script. A new file should be generated
   in the `highlighted` folder (e.g. `highlighted/Ruby/test.rb`).
6. Use `cat` or `bat --language=txt` to display the content of this file and make sure that the
   syntax highlighting looks correct.
7. `git add` the new files in the `source` folder as well as the autogenerated files in the
   `highlighted` folder.

### Troubleshooting

Make sure that the local cache does not interfere with the internally stored syntaxes and
themes (`bat cache --clear`).

## Criteria for inclusion of new syntaxes

* More than 10,000 downloads at [Package Control](https://packagecontrol.io)

### Manual modifications

The following files have been manually modified after converting from a `.tmLanguage` file:

* `Apache.sublime_syntax`=> removed `conf` and `CONF` file types.
* `Dart.sublime-syntax` => removed `#regex.dart` include.
* `DotENV.sublime-syntax` => added `.env.template`, `env` and `env.*` file types ([upstream PR](https://github.com/zaynali53/DotENV/pull/17)).
* `INI.sublime-syntax` => added `.coveragerc`, `.pylintrc`, `.gitlint`, `.hgrc`, `hgrc`, and `desktop` file types and support for comments after section headers.
* `Org mode.sublime-syntax` => removed `task` file type.
* `Robot.sublime_syntax` => changed name to "Robot Framework", added `.resource` extension.
* `SML.sublime_syntax` => removed `ml` file type.
* `wgsl.sublime-syntax` => added `wgsl` file extension.

### Non-submodule additions

* `Assembly (x86_64)` has been manually added from https://github.com/13xforever/x86-assembly-textmate-bundle due to `git clone` recursion problems
* `Nim.sublime-syntax` has been added manually from https://github.com/getzola/zola/blob/master/sublime_syntaxes/Nim.sublime-syntax as there was no suitable Git repository for it. The original syntax seems to originate from https://github.com/Varriount/NimLime
* `Rego.sublime-syntax` has been added manually from https://github.com/open-policy-agent/opa/blob/master/misc/syntax/sublime/rego.sublime-syntax
   as it is not kept in a standalone repository. The file is generated from
   https://github.com/open-policy-agent/opa/blob/master/misc/syntax/textmate/Rego.tmLanguage
* `SML.sublime_syntax` has been added manually from
   https://github.com/seanjames777/SML-Language-Definitiona as it is not
   kept in a standalone repository. The file generated is from
   https://github.com/seanjames777/SML-Language-Definition/blob/master/sml.tmLanguage
* `Cabal.sublime_syntax` has been added manually from
  https://github.com/SublimeHaskell/SublimeHaskell/ - we don't want to include the whole submodule because it includes other syntaxes ("Haskell improved") as well.
* `Lean.sublime-syntax` has been added manually from https://github.com/leanprover/vscode-lean/blob/master/syntaxes/lean.json via conversion.