mesv

A type-focused CSV parsing and formatting library with extensive configuration options and documentation examples.

I wanted the name to be short and a single word, because I dislike snake_case. Adding the fact that one of the (planned) features of this library was adding frontmatter metadata to the generated CSV file, I somehow mangled the two words metadata and csv, and eventually arrived at mesv.

Important!

This library is still in active development, and although it's functional (check the unit tests to make sure), the types, function names, signatures, and behaviours are liable to change between minor versions until version 1.0.0.

If you do end up using this library, be careful when updating to avoid sudden changes!

Installation

To use in your project, use the gleam command line tool to add mesv.

gleam add mesv

Examples

Basic examples of both formatting (converting data to String) and parsing (reading String and converting to some data type).

Formatting

As formatting is the simpler use case of the two available, I will go over it first.

To generate a default CSV file from some data, you first need to create a Formatter. To do so, simply call the format.build function and pass in a function that converts your data type to a List(String).

import gleam/int
import mesv/format

const data: List(#(String, Int, Bool)) = [
  #("Adam", 20, True),
  #("Beatrice", 25, True),
  #("Colin", 2, False),
]

pub fn main() -> Nil {
  let formatter =
    // First create a formatter
    format.build(fn(val: #(String, Int, Bool)) -> List(String) {
      let #(name, age, adult) = val
      [
        name,
        int.to_string(age),
        case adult {
          True -> "true"
          False -> "false"
        },
      ]
    })

  // Then, use that formatter on the data you want to format
  let formatted_data = format.run(formatter, data)

  // By default, the formatter uses the comma as a column separator,
  // newline as the row separator, and doublequotes for escaping cells
  assert formatted_data == "Adam,20,true\nBeatrice,25,true\nColin,2,false"
}

Then just call the format.run with it, and convert a List of data into a String.

Setting custom separators and escapers

As mentioned in the comments of the above example, the Formatter is initialized with some default values.

Specifically:

• , is used as the column separator
• \n is used as the row separator
• " is used as the escaper, to wrap cells that contain separators or escapers inside

However, all of these can be changed using the set_ functions:

• set_col_sep
• set_row_sep
• set_escaper

Go to the mesv/format module documentation through the links to learn more.

// Setup code is hidden to reduce bloat

pub fn main() -> Nil {
  let formatter =
    // [...]
    |> format.set_col_sep("|")
    |> format.set_row_sep(";")
    |> format.set_escaper("'") // Effect not visible here

  let formatted_data = format.run(formatter, data)

  // As you can see, the formatted CSV is different,
  // though the data is identical.
  assert formatted_data == "Adam|20|true;Beatrice|25|true;Colin|2|false"
}

When are cells escaped?

By default, cells are only escaped (wrapped in escapers) if they contain a separator or an escaper.

In addition, if a cell contains an escaper, it is then replaced with two escapers, to maintain an even number.

import mesv/format

const data: List(#(String, String)) = [
  #("Adam", "Cool"),
  #("Beatrice", "It's |neat|"),
  #("Colin", "I guess it's fine;"),
]

pub fn main() -> Nil {
  let formatted_data =
    format.build(fn(val: #(String, String)) -> List(String) { [val.0, val.1] })
    |> format.set_col_sep("|")
    |> format.set_row_sep(";")
    |> format.set_escaper("'")
    |> format.run(data)

  // Only cells that need to be escaped will be wrapped in escapers.
  assert formatted_data
    == "Adam|Cool;Beatrice|'It''s |neat|';Colin|'I guess it''s fine;'"
}

Using the function set_escape_all, it's possible to make a formatter that escapes all cells. However, I don't know where this would be useful - I added it because I was on a roll, without thinking too much.

Headers

By default, the output will not contain headers - however, you can set headers by using the format.set_headers function and passing in a List(String), which will be prepended to the CSV String, joined with the column separators and appropriately escaped if necessary.

// [...]
const data: List(#(String, Int, Bool)) = [
  #("Adam", 20, True),
  #("Beatrice", 25, True),
  #("Colin", 2, False),
]

pub fn main() -> Nil {
  let formatted_data =
    // [...]
    |> format.set_headers(["Name", "Age", "Is an adult"])
    // ^ Setting headers 
    |> format.run(data)

  // The specified headers are prepended to the CSV string
  assert formatted_data
    == "Name,Age,Is an adult\nAdam,20,true\nBeatrice,25,true\nColin,2,false"
}

Parsing

Parsing CSV is a bit more complicated, as for all data types other than String, it's necessary to account for failing to parse a specific cell.

However, most of this behaviour is hidden away in the internal functions, so usage is relatively easy.

import gleam/int
import mesv
import mesv/parse

const expected_data: List(#(String, Int, Bool)) = [
  #("Adam", 20, True),
  #("Beatrice", 25, True),
  #("Colin", 2, False),
]

pub fn main() -> Nil {
  let parsed_data =
    parse.build({
      // Create a parsing function using `mesv.parsed`
      // to construct a curried parsing function
      use name <- mesv.parsed
      use age <- mesv.parsed
      use adult <- mesv.parsed

      // If any value fails (ie, returns Error(Nil)),
      // the parsing of a row will stop.
      // However, if it reaches here,
      // it returns the following data type
      #(name, age, adult)
    })
    |> parse.column(Ok)
    |> parse.column(int.parse)
    |> parse.column(fn(val: String) -> Result(Bool, Nil) {
      case val {
        "true" -> Ok(True)
        "false" -> Ok(False)
        _ -> Error(Nil)
      }
    })
    // Pass in the CSV String to parse
    |> parse.run(
      "Adam,20,true\nBeatrice,25,true\nColin,2,false",
    )

  // The returned data is wrapped a bit weirdly, which I'm considering changing
  assert parsed_data == Ok(#(expected_data, []))
}

First, use the parse.build function to construct a Parser. This function expects a curried function - that is, a function that returns a function - with as many arguments as there are inputs to construct the final output.

Then, call parse.column to specify a function to use when parsing that column, and transform the Parser into one where one argument is filled.

It is possible to call parse.run using a Parser without any columns specified - however, doing so would result in a List of curried functions without any data inside.

In the above example, the succesive functions do this:

1. First, parse.build creates a Parser(fn(String) -> fn(Int) -> fn(Bool) -> #(String, Int, Bool)).
2. Then, parse.column specifies that to transform the first element of a row from String to Result(String, Nil), use the function Ok.
3. The next parse.column says that to turn the second element from String to Result(Int, Nil), use the function int.parse.
4. The last parse.column is a custom one for parsing Bool - Return Ok(Bool) for true or false, and Error(Nil) for anything else.
5. Then, call the parse.run function to use the above specified functions on each row, and if they succeed transform them into the specified data type, #(String, Int, Bool).

Headers

As in the Formatting section, handling headers is one of the features of mesv.

To specify to the Parser how to deal with headers, use the parse.set_expected_headers function and pass in an ExpectedHeaders value. This data type describes what your expectations are for the headers in the provided CSV file.

• Skip - don't care about the headers, and whatever is in the first row, skip it.
• Empty - in the CSV file, there will be no headers - start parsing directly from the first row as data.
• InOrderExact(List(String)) - you expect headers in the first row, and the parsed list of columns must be identical to what you passed into this constructor. Basically, we check if found_headers.starts_with(expected_headers), and if false, throw an error.
• HeadersMustContain(List(String)) - you expect headers in the first row, and all of the headers passed to this type must be found in the first row, in whatever order. If any one of them is not found, throw an error.
• InOrderMustPass(List(fn(String) -> Bool)) - you expect headers in the first row, and the parsed list of columns must all return True for their respective checks. Useful if you expect the first row to have some text, but don't care about capitalization, or you want to perform other checks on each header.
• HeadersMustContainPassing(List(fn(String) -> Bool)) - Analogue of the HeadersMustContain variation for the InOrderMustPass. Basically, all of the functions passed in must return True for any of the found headers.

When not specified, the Parser uses the value Empty - so it expects the first row of the CSV file to be data.

Lastly, if you specified any value expect for Skip or Empty, if the checks don't pass, the parse.run function wll return Error(ExpectedHeadersMismatch).

// expected_data is identical
pub fn main() -> Nil {
  let parsed_data =
    // [...] parser is the exact same
    // Specify that the first row is the headers,
    // and if they don't match what is specified, 
    // the parsing will fail
    |> parse.set_expected_headers(
      InOrderExact(["Name", "Age", "Is an adult"])
    )
    |> parse.run(
      "Name,Age,Is an adult\n"
      <> "Adam,20,true\n"
      <> "Beatrice,25,true\n"
      <> "Colin,2,false",
    )

  assert parsed_data == Ok(#(expected_data, []))
}

To add to the above explanation, this would be inserting a step between 4 and 5: 5. The parse.set_expected_headers says to the parser that the first row will be headers, and that they should start with the values passed in. If they don't, the parsing returns an Error.

Setting custom separators and escapers

Just as with a formatter, a parser can also be configured to expect custom colum and row separators, and escapers.

It is important to mention however, that while row and column separators can be whatever character or word you want, for escapers, there is some ambiguity.

According to the CSV specification, the grammar of a CSV file specifies that if a cell (or field, as it's referred to) is escaped, it means that there are single doublequotes (escapers) on both sides, and if any doublequotes(escapers) appear inside of the cell, they should appear doubled.

And here lies the issue - when changing the escaper to be a single character, there is no ambiguity - when doubling a single character, there are two characters.

But imagine that you set the escaper to be ==. Now, as per the specification, if an instance of the escaper appears within a cell, it must be duplicated. So, take for example the cell a==b. Then, to escape it, wrap it in escapers and duplicate the instance of the escaper inside.

So, should that be ==a====b==, or ==a===b==? When trying to count instances of the escaper in the first one, there are 5, because there are three ways to find == in ====. But then, if it were the second, it doesn't really follow the specification either.

I'm sure there does exist a foolproof and reliable method of escaping cells with multi-character escapers, but I haven't found one, and I don't think there is need of it.

Tldr; use single character escapers.

Notes

The parse.run function returns a rather strange type, that being Result(List(Result(a, parse.ParsingError)), parse.ParsingError). This structure is liable to change, as I'm not very happy with how I handled parsing errors.

However, to explain it a bit - if the Parser had headers specified using set_expected_headers function and the first row didn't pass the checks, an Error(ExpectedHeadersMismatch) value is returned. If they did match or were not specified, the function will always return an Ok value.

In that Ok value is a list of Results. Each of those rows can either succeed in parsing and return a, or fail, and return parse.ParsingError describing what happened.

If you're interested only in the successfully parsed values, then use the parse.get_parsed function to filter them out.

Documentation

Further documentation can be found at hexdocs/mesv, in the descriptions of the relevant modules.

Roadmap

As mentioned several times in this README, this project is still rather young, and there are several features I still want to implement - first among them the namesake of this library, adding metadata.

Here is a roadmap, describing the planned features:

• Alpha (0.1.0) - Done! Basic features, a usable library for parsing CSV
• Beta (0.2.0) - Typed frontmatter support, features to change parsing behaviour depending on metadata
• Release (1.0.0) - Finalized types and error handling, two-in-one constructors (create formatter and parser using the same builder), feature for column-based cell parsing in addition to order-based
• Streaming (1.1.0) - Support for stream-based inputs and outputs to reduce the memory requirements for large files