imag User Documentation

December 2019 Version: 0.10.0

Introduction

This document is the user documentation for imag, the personal information management suite for the commandline.

Basically: This is Hobby stuff. Expect incompleteness, outdated documentation, false statements and generally read with grain of salt.

If you have any objections, suggestions for improvements, bugs, etc, please file them (See Section 8). A way to reach out to the imag project maintainer(s) is described in the Section 8 section.

The Problem

The problem this project tries to solve is to provide a modular commandline application for personal information management.

It targets “power users” or “commandline users”, uses plain text as a storage format and tries to be as scriptable as possible. imag offers the ability to link data from different “PIM aspects” (such as “diary”, “contacts” and “bookmark” for example).

One major goal of imag is to make the PIM data traverseable and queryable. For example: a wiki article can be linked to an appointment which is linked to a todo which is linked to a note which is linked to a contact.

imag wants to offer an all-in-one scriptable modular commandline personal information management suite for all PIM aspects one could possibly think of. Because imag uses plain text (TOML headers for structured data and plain text which can be rendered using markdown, for example, for continuous text) the user is always able to access their data without the imag tools at hand.

The Approach

The approach “imag” takes on solving this problem is to store content in a “store” and persisting content in a unified way. Meta-information is attached to the content which can be used to store structured data. This can be used to implement a variety of “domain modules” using the store. While content is stored in one place, imag does not duplicate content. imag does not copy or move icalendar files, emails, vcard files, music or movies to the store, but tries to remember the actual files are and stores meta-information about them in the store.

Detailed explanation on this approach follows in the chapters of this work.

Implementation

The program is written in the Rust programming language.

The program consists of libraries which can be re-used by other projects to implement and adapt imag functionality. An external program may use a library of the imag distribution to store content in the store of imag and make it visible to imag this way.

This is a technical detail a user does not necessarily need to know, but as imag is intended for power-users anyways, we would say it fits here.

Alternative Projects

imag is not the only project which tries to solve that particular problem. For example there is org mode for the emacs text editor. There is also zim, a desktop wiki editor which is intended to be used for a personal wiki.

The difference between imag and the mentioned projects is: * emacs orgmode is (from what I know and see) for orgabizing things. imag is intended not only for organizing, but also for recording, tracking and querying. * zim is a wiki, which could be used for PIM but is not specialized for it. Recording habits might be possible, but not that simple as with imag

imag is not there yet, though. Some parts can be used, though it is far away from being feature-complete.

In addition: imag is text-editor independent and other tools than imag might be used to access data stored in the imag store. For example, one could “grep”, “awk” and “sed” entries without much hassle and even write bash scripts for automatically filling imag entries with data.

Architecture of the imag code

The imag codebase has a rather simple overall architecture. In this chapter the types of crates, architecture of an imag module and the type structure are described.

Crate types

There are different types of crates in the imag world:

These are all libraries. There are also binary crates in the imag project (though they are technically also library crates):

Architecture of an imag module

With the things from above, a module could have the following architecture:

+-----------------------------------+---------+
        |  imag-foo                         |         |
        +-----------------------------------+         |
        |  libimagfoo                       |         |
        +-----------------+-----------------+         |
        |                 |                 |         |
        | libimagentrybar | libimagentrybaz |         |
        |                 |                 |   lib   |
        +-----------------+-----------------+         |
        |                                   |         |
        |  <more library crates>            |         |
        |                                   |   imag  |
        +-----------------------------------+         |
        |                                   |         |
        | libimagstore                      |    rt   |
        |                                   |         |
        +-----------------------------------+---------+

External dependencies are not listed in this graphic.

The foundation of all imag modules is the store, as one can see in the visualization from above. Above the store level, entry libraries and domain libraries are used to implement functionality. The highest level of all imag modules is the commandline interface on top of the domain library. In between can be any number of entry extension libraries, or none if not needed. libimagrt is used by the binary to construct the runtime, which itself constructs and initializes the Store, so this library is used in the full stack more or less.

Theoretically, the commandline interface crate could be replaced to build a terminal user interface, graphical user interface or web interface.

Types

The imag core, hence the libimagstore and libimagrt, provide a set of types that a user (as in a library writer) should be aware of.

First of all, there is the Runtime type which is provided by the libimagrt. It provides basic access to whether debugging or verbosity is enabled as well as the most important core object: The Store.

The Store type is provided by the libimagstore library, the heart of everything.

When interacting with the store, two types are visible: FileLockEntry and Entry whereas the former derefs to the latter, which basically means that the former wraps the latter. The FileLockEntry is a necessary wrapper for ensuring that when working concurrently with the store, an entry is only borrowed once from the store. It also ensures that the object is alive as long as the store is.

The Entry type provides functionality like reading the actual content, its header and so on. Extensions for its functionality are implemented on this type, not on the FileLockEntry.

The Entry provides access to its header, which is a toml::Value, where toml is the toml-rs crate (external project). Convenience functionality is provided via the toml-query crate, an external project which was initiated and extracted from the imag project.

The Store

The store is where all the good things happen. The store is basically just a directory on the filesystem imag manages and keeps its state in.

One could say that the store is simply a database, and it really is. We opted to go for plain text, though, as we believe that plain text is the only sane way to do such a thing, especially because the amount of data which is to be expected in this domain is in the lower Megabytes range and even if it is really much won’t exceed the Gigabytes ever.

Having a storage format which is plain-text based is the superior approach, as text editors will always be there.

A user should always be able to read her data without great effort and putting everything in a real database like sqlite or even postgresql would need a user to install additional software just to read his own data. We don’t want that. Text is readable until the worlds end and we think it is therefore better to store the data in plain text.

The following sections describe the store and the file format we use to store data. One may skip the following sections, they are included for users who want to dig into the store with their editors.

File Format

The contents of the store are encoded in UTF-8. A normal text editor (like vim or the other one) will always be sufficient to dig into the store and modify files. For simple viewing even a pager (like less) is sufficient.

Each entry in the store consists of two parts:

  1. Header
  2. Content

The following section describe their purpose.

Header Format

The header format is where imag stores its data. The header is an area at the top of every file which is seperated from the content part by three dashes (---). Between these three dashes there is structured data. imag uses TOML as data format for this structured data.

The header can contain any amount of data, but modules (see Section 6) are restricted (by convention) in their way of altering the data.

Normally there are several sections in the header. One section ([imag]) is always present, it is automatically created by the store and contains a version field, which tells imag which version this file was created with. The store automatically verifies that it is compatible (satisfying semver) with the version of imag an entry was created with, and if it is not, it fails loading the entry.

Other sections are named like the modules which created them. Every module is allowed to store arbitrary data under its own section and a module may never read or write other sections than its own.

Content Format

The content is the part of the file where the user is free to enter any textual content. The content may be rendered as Markdown or other markup format for the users convenience. The store does never expect any specific markup.

Technically it would be possible that the content part of a file is used to store binary data. We don’t want this, though, as it is contrary to the goals of imag.

Example

An example for a file in the store follows.


        ---
        [imag]
        version = "0.10.0"
        
        [note]
        name = "foo"
        
        [link]
        internal = ["some/other/imag/entry"]
        ---
        
        This is an example text, written by the user.
        

File organization

The “Entries” are stored as files in the “Store”, which is a directory the user has access to. The store may exist in the users Home-directory or any other directory the user has read-write-access to.

Each module stores its data in an own subdirectory in the store, by convention.

The path to a file in the store is named “Store id” (or short “ID”) and we refer to it by using the store location as root. So if the store exists in /home/user/.imag/store/, a file with the storepath example.file is (on the filesystem) located at /home/user/.imag/store/example.file.

By convention, each libimagentry<name> and libimag<name> module stores its entries in in <name>/.

So, the pattern for the storepath is

<module name>/<optional sub-folders>/<file name>

Any number of subdirectories may be used, so creating folder hierarchies is possible and valid. A file “example” for a module “module” could be stored in sub-folders like this:

module/some/sub/folder/example

The above is not enforced or a strict rule, but rather a “rule of thumb”.

Backends

The store itself also has a backend. This backend is the “filesystem abstraction” code.

Note: This is a very core thing. Casual users might want to skip this section.

Problem

First, we had a compiletime backend for the store. This means that the actual filesystem operations were compiled into the store either as real filesystem operations (in a normal debug or release build) but as a in-memory variant in the ‘test’ case. So tests did not hit the filesystem when running. This gave us us the possibility to run tests concurrently with multiple stores that did not interfere with each other.

This approach worked perfectly well until we started to test not the store itself but crates that depend on the store implementation. When running tests in a crate that depends on the store, the store itself was compiled with the filesystem-hitting-backend. This was problematic, as tests could not be implemented without hitting the filesystem and mess up other currently-running tests.

Hence we implemented store backends.

Implementation

The filesystem is abstracted via a trait FileAbstraction which contains the essential functions for working with the filesystem.

Two implementations are provided in the code:

whereas the first actually works with the filesystem and the latter works with an in-memory HashMap that is used as filesystem.

Further, the trait FileAbstractionInstance was introduced for functions which are executed on actual instances of content from the filesystem, which was previousely tied into the general abstraction mechanism.

So, the FileAbstraction trait is for working with the filesystem, the FileAbstractionInstance trait is for working with instances of content from the filesystem (speak: actual Files).

In case of the FSFileAbstractionInstance, which is the implementation of the FileAbstractionInstance for the actual filesystem-hitting code, the underlying resource is managed like with the old code before. The InMemoryFileAbstractionInstance implementation is corrosponding to the InMemoryFileAbstraction implementation - for the in-memory “filesystem”.

Conventions, best practices

This section explains conventions used in the imag codebase. It is mainly focused on developers, but a user may read it for getting to know how imag works.

Lets work our way up from the store and how to extend it to the commandline user interface.

Versioning

All imag crates are versioned with the same version number until we reach some "1.0.0" version. This means that all imag tools are only tested for compatibility with libraries and such if their version numbers match. It might not be possible to import one imag library in version 0.3.0 and another one in 0.4.0 and make them work together. It also means that if new tools are introduced into the imag codebase, they might start with their first version not at 0.1.0 but at something like 0.5.0.

Store and Entry functionality

A Entry does not offer much functionality by itself. So its the job of libraries to extend its functionality. This should never be done by wrapping the Entry type itself but by providing and implementing an extension trait on it.

Same goes for extending the Store type: never wrap it, always provide an extension trait for it.

These two rules ensure that the type does not lose any functionality from a wrapping. Deref could do that, but not over muliple levels, so extension traits it is. It also most likely results in functions inside the extension trait which all return a Result<_, _>.

Libraries

In the next few sections, conventions and best practices for writing a imag library are written down.

A developer of imag should read this carefully, a user may skip this section or cross-read it for better understanding of the imag project.

Library naming

Libraries which provide functionality for entries or the store but no domain-functionality should be named “libimagentrything” whereas “thing” stands for what the library provides.

All domain libraries should be prefixed with “libimag”.

Library scope

A library should never introduce utility functionality which could be useful for other libraries as well. If there is no such functionality available, the “libimagutil” or “libimagentryutil” might be a place where such a function would go to.

If a library has to introduce free functions in its public interface, one should think hard whether this is really necessary.

Library error types/kinds

Libraries must use “failure” to create error objects.

Libraries with commandline frontends

Libraries with commandline frontends provide end-user functionality. They are called “domain” libraries. Normally, they depend on one or more “libimagentrything” libraries. They should be named “libimagthing”, though. For example: “libimagdiary”, “libimagtimetrack” or “libimagwiki”, whereas the commandline frontends would be “imag-diary”, “imag-timetrack” and “imag-wiki”, respectively.

If such a library needs to depend on another “libimagthing”, for example if “libimagdiary” needs to depend on “libimagnote”, one should think about this and whether the functionality could be outsourced to a more general “libimagentrything”.

Library testing

All libraries should be tested as much as possible. Sometimes it may not be possible without a lot of effort, but still: more tests = better!

Commandline tools

The commandline tools are the CLI-frontends for their respective libraries. So libimagdiary has a CLI frontend imag-diary.

Those CLI frontends use functionality from libimagrt to build a commandline interface which is consistent with the rest of the ecosystem.

Commandline applications use the runtime interfaces for receiving IDs from the CLI or IDs which are piped into the application. Commandline applications use the ‘stdin’/‘stdout’/‘stderr’ wrappers provided by the runtime (see section below).

Commandline applications are only interactive when specified by the user (normally via a --interactive flag). An application must provide the full functionality via its commandline interface, thus it is not allowed to provide functionality which is only usable in interactive mode.

IO

There are minor restrictions how imag tools should do IO. A good rule of thumb is (but most certainly only applicable when programming an imag tool in Rust): use libimagrt to do IO of any kind.

For more information, or if not using Rust as programming language: the documentation of libimagrt describes how IO should happen (which output stream to use, how input should be done).

Writing an imag module

So you want to write a module for imag. That’s nice.

This guide helps you getting started. It also can help you understanding how imag modules work, so even if you do not want to write a full new module, but extend or alter one, this guide may help you.

Data layout

First, you have to think about what data you want to store. What functionality do you want to provide and what data that creates.

In this example, we’re writing a module that stores numbers. We’re writing the appropriate library for that as well as a commandline frontend.

libimagnumberstorage

We’re writing a libimagnumberstorage which provides the core functionality of our module: Storing numbers.

That library can then be used by other library authors and by the commandline interface implementation.

Setup

So what do we need to do to write this library:

  1. Create a new “lib” crate. Because we’re writing a “domain” library, we’re doing this in the lib/domain subdirectory: cd lib/domain; cargo new --lib libimagnumberstorage.
  2. After creating the library, we have to add the new library to the /Cargo.toml field and add the missing metadata in the new /lib/domain/libimagnumberstorage/Cargo.toml file.

That was the setup part. Now we can implement our functionality. For that, we need to extend two types from libimagstore, so we have our first dependency here.

Dependencies to other libraries

  1. Put libimagstore as a dependency in the /lib/domain/libimagnumberstorage/Cargo.toml file. By using libimagstore = { version = "0.10.0", path = "../../../lib/core/libimagstore" } we automatically get all the goodness of Cargo, so that releases automagically work as expected, but when developing locally, the local version of libimagstore is used. Of course, the version has to be the latest released version.
  2. For error handling, we also need to import libimagerror.
  3. For easy header-editing, we import toml and toml-query.
  4. For error-type creating, we import error-chain.

Interface

  1. Then, we have to extend two types:
    1. libimagstore::store::Store has to be extended so we can implement a CRUD interface for our special entries.
    2. libimagstore::store::Entry has to be extended so we can get our stored numbers in a convenient way.

Our interface should roughly look like this:

store.get_stored_number("5") -> Result<FileLockEntry, _>
        store.store_number("5")      -> Result<FileLockEntry, _>
        store.delete_number("5")     -> Result<(), _>

You notice that the Store returns FileLockEntry objects rather than Entry objects. And that’s ok. A FileLockEntry is a Entry, but ensures that we are the only ones editing that entry. So, we have to implement our number-storing-interface on Entry as well:

entry.get_number() -> Result<usize>
        entry.set_number(usize) -> Result<()>

All those “extensions” are implemented as traits which are then implemented for Store and Entry.

Normally, we create new files for that, as well as for the error types we need:

where store.rs contains a trait NumberStorage and entry.rs contains a trait NumberEntry. error.rs contains the invocation of the error_chain!{} macro. Error types from libimagstore and others are linked in.

Modules

A module is a functionality of the program. There is a huge list of modules available in the imag core distribution.

Some of the modules shipped with imag cover core functionality such as linking, tagging or references to files outside of the store or even the store interface itself. Others cover things like diary, notes, wiki or bookmarks. These are also called “domains”.

We try really hard to offer a consistent commandline user interface over all of these modules.

The following sections describe each module in detail, including its purpose and its provided functionality.

Bookmarks

The Bookmarks module is for keeping URLs as bookmarks, tagging and categorizing them and finally also open them in the browser.

Category

A tool to create categories and set/get them for entries.

The difference between a category and a tag is that a category must exist before it can be used and all entries of a category are linked to the “category entry” internally.

Diary

The diary module is for keeping your diary notes.

The diary module giv3s you the possibility to write your diary in imag. It offers daily, hourly and minutely entries (the latter beeing more like a private tumble-blog).

Exporting the diary is possible, so one can write it in markdown and later pass that to pandoc, if desired, to generate a website or book from it.

Edit

The imag-edit command is for simply editing store entries with the $EDITOR.

It is based on libimagentryedit (Section 7.8).

Entry

Plumbing tool for modifying and querying structured data in entries.

Init

This is the only imag-* command which does not set up a runtime and check whether the store is available. This command can be used to set up a imag store.

It also puts a default configuration in the right place and initializes a git repository, if there is a git command in $PATH (via calling git on the commandline, not via libgit2 or some other library).

The linking module imag-link is one of the plumbing modules. It offers the possibility to link entries in the store.

It also offers the functionality to link to external sources. This functionality can be used to link to external URLs, but the bookmarking module should be used to do this (see Section 6.1).

The linking module offers functionality to add, remove and list both internal (store entry to store entry) and external (store entry to URL) links.

Internal linking

External linking

A store entry can only have one external link. Therefor, when you create an external link, the linking module creates a new entry in the store which links to this URL. The linking module then links you entry with this new entry by using an internal link. This way one entry can have multiple external links attached to it and external links are deduplicated automatically.

Log

The “imag-log” module is a lightweight interface to the “imag-diary” command.

It is intended as a tumbeblog-like diary, where one does not care to fire up an editor and type in a long text, but rather type a few words and forget about it:

Usage

Logs can be created via an entry in the configuration file in the section log:

[log]
        logs = ["work", "hobby", "music"]
        default = "hobby"

The default key is required and the name which is used here must appear in the logs array.

In the above configuration snippet, the logs work, hobby and music are created. The user may now log to one of these logs with:

imag log --to <logname> "Some message"
        # or
        imag log -t <logname> "Some message"
        # or, to the default log:
        imag log "Some message"

Logs can be read by naming the log:

imag log show work

which prints one log per line (including time it was logged).

Mails


NOTE: This is mostly a todo-list for the imag-mail command. Nothing shown here is implemented. This “documentation-to-be” should be moved to imag-mail --help eventually. This list might be incomplete, details might be not possible to implement in the way described or other dragons.

Target audience: People who want to implement imag-mail.


The Mails module implements a commandline email client. Emails can be written (via $EDITOR) and viewed, also in threads. Emails can be crawled for creating new contacts.

A Text User Interface is not planned, but might be there at some point.

The mail module implements a minimal Email client. It does not handle IMAP syncing or SMTP things, it is just a viewer for emails (a MUA).

The goal of the initial implementation is only a CLI, not a TUI like mutt offers, for example (but that might be implemented later). As this is an imag module, it also creates references to mails inside the imag store which can be used by other tools then (for example imag-link to link an entry with a mail - or the imag entry representing that mail).

So this module offers functionality to read (Maildir) mailboxes, search for and list mails and mail-threads and reply to mails (by spawning the $EDITOR).

Outgoing mails are pushed to a special directory and can later on be send via imag-mail which calls a MTA (for example msmtp) and also creates store entries for the outgoing mails.

Configuration

The following configuration variables are available for the imag-mail command:

CLI

The CLI of the imag-mail module is planned as follows:

Format specifiers

The imag-mail command supports formatting output automatically and via predefined formats in the configuration file or by passing formatting specifications via CLI.

The available formatting variables are:

Notes

The Notes module is intended to keep notes. These notes can be inserted as plain text, markdown or other markup languages.

The notes module offers:

Reference

The Reference module.

Store

The Store module.

Tagging

The Tagging module.

A valid tag matches the regex [a-zA-Z][0-9a-zA-Z]*.

Timetrack

The Timetrack module implements a timewarrior-like timetracking functionality for imag.

Each timetracking is a ‘tag’ which can be started and stopped. These tags are no tags as in imag-tag, but timetracking-tags.

Summaries can be printed, also filtered by tags if desired.

Todo

The “todo” module implements a task manager.

View

The View module.

Wiki

The Wiki module provides a personal wiki implementation.

The wiki entries are markdown-formatted files in the imag store. All entries are automatically searched for links and those links are automatically added to the header (or as external link, depending on the format).

Wiki entries can have no or one category and a arbitrary number of tags. Entries can be listed (as a “tree” shape) and filtered by content, category and tag.

Libraries

This section of the documentation is only relevant for developers and you might skip it if you’re only a user of the imag tool.

The following sections contain a short documentation on what the several libraries are supposed to do. It is generated from the README.md files of each library and only gives a general overview what can be done with the library. For a more comprehensive documentation of the library, one might consult the appropriate documentation generated from the source of the library itself.

The documentation of the libraries is sorted alphabetically.

libimagbookmark

This library crate implements functionality for bookmarks.

It uses libimagentrylink to create external links and therefor deduplicates equivalent external links (libimagentrylink deduplicates - you cannot store two different store entries for https://imag-pim.org in the store).

It supports bookmark collections and all basic functionality that one might need.

libimagcalendar

This library helps tracking (vcard) events in imag.

It does nothing more than create one imag entry per VEVENT UID, giving the user the ability to “link” to each event individually.

It uses the libimagentryref library for refering to the actual file holding the data.

libimagcontacts

The contact library basically only creates references to the actual vcard files, though it also can parse (via the vobject crate) the information and return it from an entry directly.

The architecture of indirections is as follows:


        +--------------------------------+
        |                                |
        |     Store, as ContactStore     |
        |                                |
        +----------------+---------------+
                         |
                         | Provides access to
                         |
        +----------------v---------------+
        |                                |
        |   (FileLock)Entry as Contact   |
        |                                |
        |      which is actually a:      |
        |                                |
        |     (FileLock)Entry as Ref     |
        |                                |
        +----------------+---------------+
                         |
                         | refers to
                         |
        +----------------v---------------+
        |                                |
        |   vcard file (outside store)   |
        |                                |
        +----------------+---------------+
                         |
                         | contains
                         |
        +----------------v---------------+
        |                                |
        |            vcard data          |
        |                                |
        +--------------------------------+

libimagdiary

This library crates implements a full diary.

One can have one or more diaries in the store, each diary can have unlimited entries.

The diary provides daily, hourly and even minutely diary entries. For twitter-like log keeping, have a look at “libimaglog”, though.

libimagentryannotation

This library provides annotation functionality for entries.

Annotations are normal Store entries, but their header at annotation.is_annotation is set to true.

Annotations are linked to an entry (as in libimagentrylink).

Library functionality

The library features two traits: One to extend an Entry with annotation functionality and another one for extending the Store with functionality to get annotations of an entry and all annotations in the store.

libimagentrycategory

This library provides category functionality for entries.

libimagentrydatetime

Provides date/time functionality for entries.

libimagentryedit

Provides edit (as in spawning an $EDITOR) functionality for entries.

libimagentryfilter

Helper library to filter lists of entries by certain predicated. Offers filters for filtering by header values and other predicates.

A commandline-to-filter DSL is planned for this, so commandline applications can use this to implement a uniform filter interface.

Linking library for linking entries with other entries.

libimagentrymarkdown

Helper crate to add useful functionality in a wrapper around hoedown for imag.

Adds functionality to extract links, parse content into HTML and other things which might be useful for markdown rendering in imag.

libimagentryref

This library crate contains functionality to generate references within the imag store.

Problem

The problem this library solves is the following: A user wants to refer to a file which exists on her filesystem from within imag. But unfortunately, the user has several devices and the filesystem layout (the way the $HOME is organized) is not the same on every device. With this library, the user is able to refer to a file, but without specifying the whole path.

Each device can have a different “base path”, files are re-found via their hashes and file names, assuming that the files are equal on different devices or have at least the same name.

User Story / Usecase

Alice has a music library on her workstation and on her notebook. On her workstation, the music collection is at home/alice/music, on the notebook, it exists in /home/al/media/music.

From within imag, alice wants to create a link to a file $music_store/Psy_trance_2018_yearmix.mp3.

libimagentryref helps her, because she can provide a “base path” in the imag configuration file of each device and then link the file. imag only stores data about the file and its relative path, but not its abolute path.

When moving the imag store from the workstation to the notebook, the base path for the music collection is not /home/alice/music anymore, but /home/al/media/music and imag can find the file automatically.

Solution, Details

libimagentryref does store the following data:

[ref]
        basepath = "music"
        is_ref = true # marker that this entry is actually a "ref"
        relpath = "Psy_trance_2018_yearmix.mp3"
        
        [ref.hash]
        sha1 = "<sha1 hash of the file>"

The filehash is stored so that libimagentryref can re-find the file whenever it was moved. The sha1 key is added to be able to upgrade hashes later to other hashing algorithms. relpath is the part of the path that when joined with the “base” path from the configuration results in the full path of the file for the current machine. The “collection” key hints to the configuration key in the imag config file.

The configuration section for the collections looks like this:

[ref.basepathes]
        music = "/home/alice/music"
        documents = "/home/alice/doc"

libimagentryref provides functionality to get the file. libimagentryref also offers functionality to find files only using their filename (x)or filehash and correct the filehash or filename respectively (automatically or explicitely).

Limits

As soon as the file is renamed and modified, this fails. This does also not cover the use case where the same file has different names on different machines.

libimagentrytag

Library for tagging entries. Used in “imag-tag” but should be used in all other modules which contain tagging functionality, so the backend and frontend look the same for all modules.

libimagentryutil

This library contains utilities for working with libimagstore::store::Entry objects where the functionality does not necessarily belong into libimagstore.

libimagentryview

Provides viewing (as in piping to stdout, opening in $EDITOR or in $BROWSER) functionality for entries.

libimagerror

In imag, we do not panic.

Whatever we do, if we fail as hard as possible, the end-user should never ever see a backtrace from a panic!().

Anyways, the user might see a error trace generated by imag. That is because imag is software for power-users, for nerds (I use the term “nerd” because for me it is a good thing - I do not want to offend anyone by using it). This target group can read backtraces without getting confused. IO Error and Permission denied Error are things that nerds can understand and they already know what to do in the most obvious cases (such as Permission denied Error).

This library crate is for generating error types and handle them in a nice way. It can be seen as mini-framework inside imag which was written to work with error types in a specified way. All imag crates must use this library if they can return errors in any way, except the libimagutil - which is for the most basic utilities.

libimaghabit

The habit library implements a habit tracker.

A habit can be instantiated with a name and a time-period in which it should be fullfilled (eg. daily, ever 3 days, weekly…).

The module offers ways to generate statistics about habits.

libimaginteraction

A crate for more general interaction with the user (interactive commandline interface).

Offers functions for asking the user Y/N questions, for (numeric) values, etc.

libimaglog

A small extension over libimagdiary which strips down the functionality of libimagdiary to some defaults for writing a log (a tumbleblog like diary) with rather short messages.

Provides only basic functionality over libimagdiary, most notably the “log.is_log” header entry, so the imag-log CLI can distinguish between “logs” and “diary entries”.

libimagmails

The mail library implements everything that is needed for being used to implement a mail reader (MUA).

It therefor provides reading mailboxes, getting related content or mails, saving attachments to external locations, crafting new mails and responses,…

It also offers, natively, ways to search for mails (which are represented as imag entries).

For more information on the domain of the imag-mail command, look at the documentation of the Section 6.9 module.

libimagnotes

libimagrt

This library provides utility functionality for the modules and the binary frontends, such as reading and parsing the configuration file, a builder helper for the commandline interface and such.

It also contains the store object and creates it from configuration.

the libimagrt::runtime::Runtime object is the first complex object that comes to live in a imag binary.

IO with libimagrt

libimagrt also provides IO primitives which should be used by all imag tools and libraries:

The IO story in imag is pretty easy. As imag is mainly a CLI tool, IO is either stdout or stderr and stdin.

Output

libimagrt provides getters for an output stream for “normal” output, like logging, status information, etc. It also provides an output for “touched entries”.

Whenever an imag tool touches an entry in any way (either reading or writing), it should report this to libimagrt. libimagrt then does “the right thing” which is printing it to stdout or swallowing the output. Normal output (logging, status information, explicitely queried information) goes to the right sink automatically, that is:

With these two settings in place, calls to imag can be chained and mixed with external tools pretty easily:

imag -O ids where 'some.header == 1' | \
        imag -I -O tag add foo               | \
        imag -I -O category set bar          | \
        fzf                                  | \
        imag -I tag add baz

The first line gets all imag entries where some.header equals 1. The touched entries are printed to stdout (-O). The second line tags all entries which are passed via stdin (-I) with foo and prints them to stdout (-O) The third line sets the category for all entries which are passed via stdin with bar and prints them to stdout. The fourth line calls the fzf program and lets the user select one entry and the last line reads that entry via stdin and tags it with baz.

Automatically detecting the appropriate input/output settings is possible, but hidden behind a environment-flag, as it is considered experimental. To test this, set IMAG_IO_EXPERIMENTAL=1 in your environment. Note that stdin may be detected as “store id stream” when it is actually not. libimagrt can take care of this when passing --interactive.

Input

libimagrt also provides primitives for input. As documented in the paragraph on “Output”, imag tools may get store ids passed via stdin. Hence, imag tools may/can not interactive when passing store ids via stdin. libimagrt provides functionality to query data from the user. These functions automatically fail if the user passes store-ids via stdin.

The next paragraph documents the details of this and may be skipped.

The user tells imag that stdin contains store-ids by setting the -I (--ids-in) flag on the commandline. If that flag is given, the interactive functionality of libimagrt automatically returns an Err(_) which can be used to tell the user what happened and exit the program accordingly. The user may also provide --interactive to tell imag via libimagrt that stdin is indeed not a stream of store-ids even if a pipe is detected.

libimagstore

The store is the heart of everything. Here lives the data, the complexity and the performance bottleneck.

The store offeres read/write access to all entries.

The store itself does not offer functionality, but has a commandline interface “imag-store” which can do basic things with the store.

libimagtimetrack

A library for tracking time events in the imag store.

Store format

Events are stored with a store id like this:

/timetrack/<insert-date-year>/<insert-date-month>/<insert-date-day>/<insert-date-time>-<tag>.ext

Timetrackings contain

by default and might be extended with more header fields as one likes.

The header of a timetrack “work” entry looks like this:

[event]
        tag = "work"
        start = "2017-01-02T03:04:05"
        end = "2017-01-02T06:07:08"

Normal tags (as in libimagentrytag) are explicitely not used for tagging, so the user has the possibility to use normal tags on these entries as well.

The tag field is of type string, as for one tag, one entry is created. This way, one can track overlapping tags, as in:

imag timetrack start foo
        imag timetrack start bar
        imag timetrack stop foo
        imag timetrack start baz
        imag timetrack stop bar
        imag timetrack stop baz

The end field is, of course, only set if the event already ended.

Library functionality

The library uses the libimagentrydatetime::datepath::DatePathBuilder for building StoreId objects.

The library offers two central traits:

The library does not provide functionality to implement imag-timetrack or so, as the core functionality is already given and the commandline application can implement the missing bits in few lines of code.

Aggregating functionality might be provided at a later point in time.

libimagtodo

The library for the todo module which provides functionality to implement/implements a todomanager in imag.

Implementation details

One todo entry is stored as one imag entry. The ID of the imag entry is generated by appending a unique ID (UUID) to “todo/”.

The unique ID identifies the todo entry.

Stored data

A todo entry stores the following information:

The description of the todo entry is stored as plain text.

Data not stored

Some data is explicitely not stored by the library because there are other libraries fullfilling that purpose. These are:

Header format

The header partial for libimagtodo is as follows:

[todo]
        uuid = "string"
        status = "enum { 'deleted', 'done', 'pending' }"
        scheduled = "<NaiveDateTime>" // optional
        hidden = "<NaiveDateTime>" // optional
        due = "<NaiveDateTime>" // optional
        priority = "enum { 'h', 'm', 'l' }" // optional

Functionality

The provided functionality of this library includes, but is not limited to:

libimagutil

Utility library. Does not depend on other imag crates.

libimagwiki

The wiki library implements a complete wiki for personal use.

This basically is a note-taking functionality combined with linking.

Layout

The basic structure and layout is as simple as it gets:

/wiki holds all wikis. The default wiki is /wiki/default. Below that there are entries. Entries can be in sub-collections, so /wiki/default/cars/mustang could be an entry.


        +-------------+
        |             |
        |  WikiStore  |
        |             |
        +------+------+
            1  |
               |
               | n
        +------v------+
        |             |
        |    Wiki     |
        |             |
        +------+------+
             1 |
               |
               | n
        +------v------+
        |             | n
        |    Entry    <------+
        |             |      |
        +------+------+      |
             1 |             |
               |             |
               |             |
               +-------------+

The store offers an interface to get a Wiki. The wiki offers an interface to get entries from it.

Each Entry might link to a number of other entries within the same wiki. Cross-linking from one wiki entry to an entry of another wiki is technically possible, but not supported by the Entry itself (also read below).

When creating a new wiki, the main page is automatically created.

Autolinking

The Entry structure offers an interface which can be used to automatically detect links in the markdown. The links are then automatically linked (as in libimagentrylink).

Contributing to imag

So you want to contribute to imag! Thank you, that’s awesome!

All contributors agree to the developer certificate of origin by contributing to imag.

Feel free to contact us via our mailinglist and/or submit patches via mail (use git format-patch and git send-email, always add a cover letter to describe your submission). You don’t have to send patches via mail, though. As long as I can git pull your changes (without having to login or register at the remote) or git am your patchset, I’m fine. I’d encourage you, though, to use git-send-email or at least git-request-pull

Make sure to test-compile your patchset and run tests if there are any for the code you changed.

Prerequisites

The prerequisites are simple: cargo and rustc in current version (stable) or newer (we do not use nighly features though).

Build dependencies for building are listed in the default.nix file, though you do not have to have the nix package manager installed to build imag. Everything else will be done by cargo.

Commit guidelines

Make sure your patchset does not contain “Fixup” commits when publishing it, but feel free to send “Fixup” commits in the review process. If squashing fails I will come back to you.

Also ensure that each commit has a “Signed-off-by:” line. By adding that line, you agree to our developer certificate of origin. If you do not add the “Signed-off-by:” line, I reserve the right to kindly reject your patch.

Code of Conduct

We use the same code of conduct as the rust community does.

Basically: Be kind, encourage others to ask questions - you are encouraged to ask questions as well!

Developer Certificate of Origin

Developer Certificate of Origin
        Version 1.1
        
        Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
        660 York Street, Suite 102,
        San Francisco, CA 94110 USA
        
        Everyone is permitted to copy and distribute verbatim copies of this
        license document, but changing it is not allowed.
        
        
        Developer's Certificate of Origin 1.1
        
        By making a contribution to this project, I certify that:
        
        (a) The contribution was created in whole or in part by me and I
            have the right to submit it under the open source license
            indicated in the file; or
        
        (b) The contribution is based upon previous work that, to the best
            of my knowledge, is covered under an appropriate open source
            license and I have the right under that license to submit that
            work with modifications, whether created in whole or in part
            by me, under the same open source license (unless I am
            permitted to submit under a different license), as indicated
            in the file; or
        
        (c) The contribution was provided directly to me by some other
            person who certified (a), (b) or (c) and I have not modified
            it.
        
        (d) I understand and agree that this project and the contribution
            are public and that a record of the contribution (including all
            personal information I submit with it, including my sign-off) is
            maintained indefinitely and may be redistributed consistent with
            this project or the open source license(s) involved.

Changelog

This section contains the changelog.

Some things, like typo fixes, version string updates and such are not stated in the changelog (though updating of dependencies is). Please note that we do not have a “Breaking changes” section as we are in Version 0.y.z and thus we can break the API like we want and need to.

0.10.0

Some things cannot linked to because many commis were necessary for introducing them to the imag codebase:

These are just the major changes. From v0.9.0 to v0.10.0, more than 650 smaller changes and fixes where added to the codebase as well. All in all, over 700 commits were added to the master branch since v0.9.0.

0.9.3

Bugfix release for fixing:

0.9.2

Bugfix release for fixing:

0.9.1

Bugfix release for fixing:

0.9.0

In the process of these major changes, small bugfixes and improvements were applied to the codebase. Too much to list all of them here, though.

The merge messages of the respective feature branches contain more details on the changes.

0.8.0

After the last release (0.7.0), we changed how we keep our changelog from manual adding things to the list, to using git-notes. Hence, there’s no categorization anymore.

0.7.1

Bugfix release for fixing:

0.7.0

0.6.4

Bugfix release for fixing:

0.6.3

Bugfix release for fixing:

0.6.2

Bugfix release for fixing:

0.6.1

Bugfix release for fixing two severe bugs in imag-init:

0.6.0

0.5.0

0.4.0

0.3.0

Note: As this file was written after the 0.3.0 release, we simply list the merges here instead of explaining what changed.

0.2.0

0.1.0