sakalWeb IS RESTful API

Also abbreaviated as swis-api or swapi, is an in-memory database experimental project. It suits like a huge structured database with very low latency. The project is written in Go (1.20).

• https://github.com/savla-dev/swis-api

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

GET /

{
    "app_env": {
        "alpine_version": "3.17",
        "app_mode_environment": "production",
        "app_version": "5.3.1",
        "golang_version": "1.20.2",
        "instance_name": "f20a59cc875b"
    },
    "code": 200,
    "header": "sakalWebIS v5 RESTful API -- root route",
    "message": "welcome to swis, krusty!",
    "timestamp": 1682719078,
    "user": {
        "acl": [
            "backups",
            "business",
            "depots",
            "dish",
            "finance",
            "infra",
            "links",
            "news",
            "projects",
            "roles",
            "six",
            "users"
        ],
        "roles": [
            "operator",
            "infrastructure",
            "gdpr",
            "golang",
            "swis"
        ]
    }
}

history

It all started in circa 2009 with a project called Šakal (‘shah-kahl’; abbreaviation of two authors’ names), a simple draft for a content management system (CMS) written in PHP language, with a very simple CSS layout. Years later, the project was renamed to sakalWeb as it had been powering up some of the other projects (as their engines) at a time.

versioning

Below listing some of the projects’ names and concurrently used sakalWeb versions.

implementation

The backend itself is written in Go v1.20. A single binary executable is then exported from a staging docker image to a clean alpine image, resulting in docker image size being about 20 MB (alpine image itself is about 5.5 MB in size). Single built binary is about 11 MB (x86_64 arch).

The system is build on Gin-Gonic framework, making it both fast, and highly reliable from the beginning. As far as the system model is concerned, the system is modular — core engine is just gin and its middlewares with other modules exposing their routes, controllers and models to the core. The core then links each route via gin.RouterGroup to the main HTTP server handler.

As the name suggets, CRUD (Create, Read, Update, Delete) model is implemented for each module making it truly RESTful (mostly).

data storage

Data are virtually stored in data structures and models’ vars (arrays of structures) — in the runtime memory. After (re)start, the system is flushed clean, making it mandatory for data to be imported afterwards… At the moment, we do keep data backups as JSON files, being available to import using

1
2

GET  /module
POST /module/restore

API calls.

The largest import we have fullfilled so far took swapi about 1.5 ms (miliseconds) to load! (The file had about 40 kB in size, carrying more than 350 lines of JSON objects.)

development

For our development, we use swapi instances being spined off locally, allowing one to access modules and their routes immediately. As SCM we use GitHub with an automatic (on-push) CI/CD pipelines being run/executed in our own infrastructure using Github Actions Runners.

modules

The IS has its core in handling all routes, and the middleware such as Gin logger, panic recovery, and custom auth middleware (more below).

alvax

alvax is our internal Telegram bot for savla.dev folks to entertain. This module serves alvax’s command list for it to fetch, decode and serve itself.

auth

This module has no routes, but it serves as custom middleware in terms of AAA (at least the first A’s as in Authentication and Authorization). Every request goes through this middleware, being checked for X-Auth-Token header contents. If empty, HTTP 401 Unauthorized code and message are returned. The same comes when the token is invalid, or the user is not activated. Only when one enters a valid token, middleware pushes the app context to the next service.

For further database access, explicit list (ACL) of modules has to be set for each user. Default methods GET and OPTIONS are enabled by default on every module. Methods PUT, PUSH, UPDATE can be used with power Role. Roles serve as the other layer of authorization. Only admin roles can use DELETE method over such module.

example

User wants to list/read/get contents of the depots loaded module. One has to fulfill all the listed items:

• a record (account) in users module
• activated account
• valid SHA256 token
• ACL with such module name assigned
• no special Role is required for this HTTP method

business

Business module stores all information about customers (business-to-business!) in terms of name, VAT ID, addresses and contact. These are usualy used when generating invoices for example.

depot

Depot(s) module is meant to be “an interface” for storing storage inforamtion like Item name, description and location. This idea comes from the old swis. One can see the information when looking for a particular item (like old notebook), just to find out that that object is not in his area at the time (they left it elsewhere).

dish

This module serves socket lists to savla-dish instances. Ususally, a dish can request “its sockets” by dialing

1

GET /dish/sockets/:dish_name

API call, Only unmuted and matching sockets are then exported and returned as JSON list to dish to load the stream.

docs

This module is a shadow one, because it does not really serves anything directly, it just acts like “a storage” for swagger exported documentation files. Swagger UI then acts like a frontend for those data — files are linked to container and read-only’d.

finance

Financial accounts module stores all internal account info, bank codes etc. It however does not store any explicit bank card details and so on. Moreover, financial records could be included, so some sort of remote Bank API is to be implemented and used to sync the records themselves.

groups

As the name suggests, this module is to be used to moderate groups within the organization.

infra

This module stores information about infrastructure as a whole. It includes models for domains, network, and nodes logic. One is being able to link them together and to construct a diagram out of given JSON.

Fig. 1: Grafana panel with a simple infrastructure overview using Diagram panel.

news

News module have two so-called submodules: news sources and the actual news, in form of a model to link with RSS feed stream input. The first submodule gives the information of one’s news sources — what one wants to fetch, which RSS sources to use. The second one allows one to execute an RSS fetching mechanism allowing swapi to parse and return given news as JSON.

sources example output

1

GET /news/sources/krusty

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
    "code": 200,
    "message": "ok, dumping news sources",
    "news": [
        {
            "news_user": "krusty",
            "news_sources": [
                {
                    "source_id": "",
                    "source_name": "ČT24 Hlavní zprávy",
                    "source_url": "http://www.ceskatelevize.cz/ct24/rss/hlavni-zpravy"
                },
                {
                    "source_id": "",
                    "source_name": "iRozhlas.cz",
                    "source_url": "https://www.irozhlas.cz/rss/irozhlas"
                },
                {
                    "source_id": "",
                    "source_name": "Seznam Zprávy",
                    "source_url": "https://api.seznamzpravy.cz/v1/documenttimelines/5ac49a0272c43201ee1d957f?rss=1"
                }
            ]
        }
    ]
}

news (RSS) feed (parsed on other fronted)

Example of parsed swapi JSON output (CZ news, screenshot):

Fig. 2: News module formatted output example.

projects

Project(s) module is to serve list(s) of local projects being worked on in such organization. It includes project’s name, its repository, URL, status of publishment, project leader and so on.

swife

swife stands for sakalWeb IS frontend(s). This module stores raw page data as base64 strings, to fetch them by forntend engines, and to easily edit them on the backend. Note that this is only a temporary module.

users

The main module of them all — user management system, unified interface for adding, fetching, editing and deleting user records. It also stores user’s access tokens needed to be allowed into swapi. When there are no users loaded, the root token (generated at start/or loaded from the local environment at start — required) has to be used for initial setups, and for data imports (restores).

clients

As swapi acts like a JSON API, any frontend app can be built above the data base backend. For this purposes, we use a simple desktop-ready frontend built on SB Admin 2 and Django framework (called swjango); and a very simple mobile-ready PWA called swAPP. Both applications run on the internal network only, hence the VPN connection is a must there.

• bbs-go (simple telnet BBS experiment in Go)
• savla-gw (API gateway experiment in Go)
• swapp (mobile-webapp-first UI, PWA in Go)
• swife-xp (XP.css powered fronted in Vue and Node.js)
• swjango (desktop webapp, Django)