diff --git a/README.md b/README.md
index ce46e03c3d430c6ceb2d846898cf4e34d052def5..7cb97ee7f5db3f4e896bf0e0e0535b30325e103c 100644
--- a/README.md
+++ b/README.md
@@ -9,45 +9,67 @@ This implementation has several features to make usage as simple as possible.
* Rustus is robust, since it uses asynchronous Rust;
* It can store information about files in databases;
* You can specify directory structure to organize your uploads;
+* It has a lot of hooks options, and hooks can be combined.
* Highly configurable;
-### Supported info storages
+## Installation
-* FileSystem
-* PostgresSQL
-* Mysql
-* SQLite
-* Redis
+You can download binaries from a [releases page](https://github.com/s3rius/rustus/releases).
-### Supported data storages
+If you want to use docker, you can use official images from [s3rius/rustus](https://hub.docker.com/r/s3rius/rustus/):
+```bash
+docker run --rm -it -p 1081:1081 s3rius/rustus:latest
+```
-* FileSystem
+If we don't have a binary file for your operating system you can build it with [cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html).
-## Installation
+```bash
+git clone https://github.com/s3rius/rustus.git
+cd rustus
+cargo install --path . --features=all
+```
+
+### Supported data storages
-Since I haven't configured build automation yet, you can build it
-from source using `cargo`.
+Right now you can only use `file-storage` to store uploads data.
+The only two options you can adjust are:
+* uploads directory
+* directory structure
+
+To upload files in a custom directory other than `./data`
+you can provide a `--data-dir` parameter.
```bash
-cargo install --path .
+rustus --data-dir "./files"
```
-Or you can use a docker image.
+If you have a lot of uploads, you don't want to store all your files in
+a flat structure. So you can set a directory structure for your uploads.
```bash
-docker run --rm -it -p 1081:1081 s3rius/rustus:latest
+rustus --dir-structure="{env[HOSTNAME]}/{year}/{month}/{day}"
```
-Docker image and binaries will be available soon.
+```bash
+tree data
+data
+├── 0bd911d4054d41c6a3ad54be67ee3e66.info
+├── 5bc9c62384494c439e2a064b82a39cc6.info
+└── rtus-68cb5b8746-5mgw9
+ └── 2022
+ └── 1
+ └── 8
+ ├── 0bd911d4054d41c6a3ad54be67ee3e66
+ └── 5bc9c62384494c439e2a064b82a39cc6
-## Architecture
+```
-Files and info about them are separated from each other.
-In order to modify original file rustus searches for information about
-the file in information storage.
+**Important note:** if you use variable that doesn't exist or incorrect like invalid env variable, it
+results in an error and the directory structure will become flat again.
-However, automatic migration between different information
-storages is not supported yet.
+As you can see all info files are stored in a flat structure. It cannot be changed if
+you use file info storage. In order to get rid of those `.info` files use different
+info storages.
## Info storages
@@ -55,7 +77,7 @@ The info storage is a database or directory. The main goal is to keep track
of uploads. Rustus stores information about download in json format inside
database.
-File storage is a default one. You can customize the directory of an .info files
+File storage is used by default. You can customize the directory of an .info files
by providing `--info-dir` parameter.
```bash
@@ -90,27 +112,225 @@ you have to use webhooks or File hooks.
Hooks have priorities: file hooks are the most important, then goes webhooks and AMQP hooks have the least priority.
If pre-create hook failed, the upload would not start.
-Of course, since AMQP is a protocol that doesn't allow you to track responses.
-We can't validate anything to stop uploading.
-
-
-### Roadmap
-
-* [x] Data storage interface;
-* [x] Info storage interface;
-* [x] Core TUS protocol;
-* [x] Extensions interface;
-* [x] Creation extension;
-* [x] Creation-defer-length extension;
-* [x] Creation-with-upload extension;
-* [x] Termination extension;
-* [x] Route to get uploaded files;
-* [x] Database support for info storage;
-* [x] Redis support for info storage;
-* [x] Notification interface;
-* [x] Notifications via http hooks;
-* [x] Notifications via RabbitMQ;
-* [X] Executable files notifications;
-* [ ] S3 as data storage store support;
-* [ ] Rustus helm chart;
-* [ ] Cloud native rustus operator.
+Of course, since AMQP is a protocol that doesn't allow you to track responses
+we can't validate anything to stop uploading.
+
+Hooks can have 2 formats
+
+default:
+```json
+{
+ "upload": {
+ "id": "",
+ "offset": 0,
+ "length": 39729945,
+ "path": null,
+ "created_at": 1641620821,
+ "deferred_size": false,
+ "metadata": {
+ "filename": "38MB_video.mp4",
+ "meme": "hehe2"
+ }
+ },
+ "request": {
+ "URI": "/files",
+ "method": "POST",
+ "remote_addr": "127.0.0.1",
+ "headers": {
+ "accept-encoding": "gzip, deflate",
+ "connection": "keep-alive",
+ "host": "localhost:1081",
+ "upload-metadata": "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=",
+ "tus-resumable": "1.0.0",
+ "content-length": "0",
+ "upload-length": "39729945",
+ "user-agent": "python-requests/2.26.0",
+ "accept": "*/*"
+ }
+ }
+}
+```
+
+tusd:
+```json
+{
+ "Upload": {
+ "ID": "",
+ "Offset": 0,
+ "Size": 39729945,
+ "IsFinal": true,
+ "IsPartial": false,
+ "PartialUploads": null,
+ "SizeIsDeferred": false,
+ "Metadata": {
+ "filename": "38MB_video.mp4",
+ "meme": "hehe2"
+ },
+ "Storage": {
+ "Type": "filestore",
+ "Path": null
+ }
+ },
+ "HTTPRequest": {
+ "URI": "/files",
+ "Method": "POST",
+ "RemoteAddr": "127.0.0.1",
+ "Header": {
+ "host": [
+ "localhost:1081"
+ ],
+ "user-agent": [
+ "python-requests/2.26.0"
+ ],
+ "accept": [
+ "*/*"
+ ],
+ "content-length": [
+ "0"
+ ],
+ "upload-metadata": [
+ "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ="
+ ],
+ "connection": [
+ "keep-alive"
+ ],
+ "tus-resumable": [
+ "1.0.0"
+ ],
+ "upload-length": [
+ "39729945"
+ ],
+ "accept-encoding": [
+ "gzip, deflate"
+ ]
+ }
+ }
+}
+```
+
+### File hooks
+
+Rustus can work with two types of file hooks.
+
+1. Single file hook;
+2. Hooks directory.
+
+The main difference is that hook name is passed as a command line parameter to a
+single file hook, but if you use hooks directory then hook name is used to determine a
+file to call. Let's take a look at the examples
+
+Example of a single file hook:
+
+```bash
+#!/bin/bash
+
+# Hook name would be "pre-create", "post-create" and so on.
+HOOK_NAME="$1"
+MEME="$(cat /dev/stdin | jq ".upload .metadata .meme" | xargs)"
+
+# Here we check if name in metadata is equal to pepe.
+if [[ $MEME = "pepe" ]]; then
+ echo "This meme isn't allowed" 1>&2;
+ exit 1
+fi
+```
+
+As you can see it uses first CLI parameter as a hook name and all hook data is received from stdin.
+
+Let's make it executable
+```bash
+chmod +x "hooks/unified_hook"
+```
+
+To use it you can add parameter
+```bash
+rustus --hooks-file "hooks/unified_hook"
+```
+
+This hook is going to ignore any file that has "pepe" in metadata.
+
+Let's create a hook directory.
+
+```bash
+⯠tree hooks
+hooks
+├── post-create
+├── post-finish
+├── post-receive
+├── post-terminate
+└── pre-create
+```
+
+Every file in this directory has an executable flag.
+So you can specify a parameter to use hooks directory.
+
+```bash
+rustus --hooks-dir "hooks"
+```
+
+In this case rustus will append a hook name to the directory you pointed at and call it as
+an executable.
+
+Information about hook can be found in stdin.
+
+### Http Hooks
+
+Http hooks use http protocol to notify you about an upload.
+You can use HTTP hooks to verify Authorization.
+
+
+Let's create a FastAPI application that listens to hooks and checks the
+authorization header.
+
+```bash
+# Installing dependencies
+pip install fastapi uvicorn
+```
+
+```python
+# server.py
+from fastapi import FastAPI, Header, HTTPException
+from typing import Optional
+
+app = FastAPI()
+
+@app.post("/hooks")
+def hook(
+ authorization: Optional[str] = Header(None),
+ hook_name: Optional[str] = Header(None),
+):
+ print(f"Received: {hook_name}")
+ if authorization != "Bearer jwt":
+ raise HTTPException(401)
+ return None
+```
+
+Now we can start a server.
+```bash
+uvicorn server:app --port 8080
+```
+
+Now you can start rustus, and it will check if Authorization header has a correct value.
+```bash
+rustus --hooks-http-urls "http://localhost:8000/hooks" --hooks-http-proxy-headers "Authorization"
+```
+
+
+### AMQP hooks
+
+All hooks can be sent with an AMQP protocol.
+
+For example if you have a rabbitMQ you can use it.
+
+```bash
+rustus --hooks-amqp-url "amqp://guest:guest@localhost" --hooks-amqp-exchange "my_exchange"
+```
+
+This command will create an exchange called "rustus" and queues for every hook.
+
+Every hook is published with routing key "rustus.{hook_name}" like
+"rustus.post-create" or "rustus.pre-create" and so on.
+
+The problem with AMQP hooks is that you can't block the upload.
+To do this you have to use HTTP or File hooks. But with AMQP your
+uploads become non-blocking which is definitely a good thing.
\ No newline at end of file
diff --git a/src/config.rs b/src/config.rs
index d27bc42a93bb68b020fae394de91cca83d790f84..15c1ab7b567e88378086832fb1594725e39e4e24 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -4,6 +4,7 @@ use std::path::PathBuf;
use chrono::{Datelike, Timelike};
use lazy_static::lazy_static;
+use log::error;
use structopt::StructOpt;
use crate::info_storages::AvailableInfoStores;
@@ -36,11 +37,11 @@ pub struct StorageOptions {
///
/// This directory is used to store files
/// for all *file_storage storages.
- #[structopt(long, default_value = "./data")]
+ #[structopt(long, env = "RUSTUS_DATA_DIR", default_value = "./data")]
pub data_dir: PathBuf,
- #[structopt(long, default_value = "")]
- pub dis_structure: String,
+ #[structopt(long, env = "RUSTUS_DIR_STRUCTURE", default_value = "")]
+ pub dir_structure: String,
}
#[derive(StructOpt, Debug, Clone)]
@@ -91,8 +92,8 @@ pub struct NotificationsOptions {
///
/// This format will be used in all
/// messages about hooks.
- #[structopt(long, default_value = "default", env = "RUSTUS_NOTIFICATION_FORMAT")]
- pub notification_format: Format,
+ #[structopt(long, default_value = "default", env = "RUSTUS_HOOKS_FORMAT")]
+ pub hooks_format: Format,
/// Enabled hooks for notifications.
#[structopt(
@@ -236,8 +237,10 @@ impl RustusConf {
vars.insert("year".into(), now.year().to_string());
vars.insert("hour".into(), now.hour().to_string());
vars.insert("minute".into(), now.minute().to_string());
- strfmt::strfmt(self.storage_opts.dis_structure.as_str(), &vars)
- .unwrap_or_else(|_| "".into())
+ strfmt::strfmt(self.storage_opts.dir_structure.as_str(), &vars).unwrap_or_else(|err| {
+ error!("{}", err);
+ "".into()
+ })
}
/// List of extensions.
diff --git a/src/protocol/core/routes.rs b/src/protocol/core/routes.rs
index c81e32a304006001f4c50cd9887bd545cea4e64a..7ba5919abe84b1595ac7c49de3ad2d2c0189692c 100644
--- a/src/protocol/core/routes.rs
+++ b/src/protocol/core/routes.rs
@@ -81,7 +81,7 @@ pub async fn write_bytes(
if app_conf.hook_is_active(hook) {
let message = app_conf
.notification_opts
- .notification_format
+ .hooks_format
.format(&request, &file_info)?;
let headers = request.headers().clone();
tokio::spawn(async move {
diff --git a/src/protocol/creation/routes.rs b/src/protocol/creation/routes.rs
index f9ee9cca7410bc419890bdbc9a418f1307d9d5ec..b24ca73503b53b006860bce45ea9ae0d46130efd 100644
--- a/src/protocol/creation/routes.rs
+++ b/src/protocol/creation/routes.rs
@@ -79,7 +79,7 @@ pub async fn create_file(
let initial_file_info = FileInfo::new("", length, None, meta.clone());
let message = app_conf
.notification_opts
- .notification_format
+ .hooks_format
.format(&request, &initial_file_info)?;
let headers = request.headers();
notification_manager
@@ -110,7 +110,7 @@ pub async fn create_file(
if app_conf.hook_is_active(Hook::PostCreate) {
let message = app_conf
.notification_opts
- .notification_format
+ .hooks_format
.format(&request, &file_info)?;
let headers = request.headers().clone();
// Adding send_message task to tokio reactor.
diff --git a/src/protocol/termination/routes.rs b/src/protocol/termination/routes.rs
index 272fc2154a43c7fc4b383df1512246925cb3545d..696feb2d44a824e6bb04d18512ea92a8d2884f6f 100644
--- a/src/protocol/termination/routes.rs
+++ b/src/protocol/termination/routes.rs
@@ -20,7 +20,7 @@ pub async fn terminate(
if app_conf.hook_is_active(Hook::PostTerminate) {
let message = app_conf
.notification_opts
- .notification_format
+ .hooks_format
.format(&request, &file_info)?;
let headers = request.headers().clone();
tokio::spawn(async move {