diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d193a7af580b316299bcce30f97dcd3f6b321a71..7a07c2e4587e68248f4a0f272ff169d7ebde4d45 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -140,6 +140,36 @@ jobs: target_dir: helm_releases app_version: ${{env.APP_VERSION}} + upload_docs: + runs-on: ubuntu-latest + needs: + - upload_helm + steps: + - uses: actions/checkout@v2 + - uses: actions/setup-python@v2 + with: + python-version: 3.x + - name: Install dependencies + run: pip install mkdocs-material pygments + - name: Build docs + run: mkdocs build -d /tmp/docs + - uses: actions/checkout@v2 + with: + ref: gh-pages + - name: Commit files + run: | + ls -A | grep -vE "^(.git|.gitignore|helm_releases)" | xargs rm -rfv + cp -a /tmp/docs/. . + git config --local user.email "s3rius@users.noreply.github.com" + git config --local user.name "s3rius" + git add . + git commit -m "Docs update" + - name: Push changes + uses: ad-m/github-push-action@master + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + branch: gh-pages + publish_crate: runs-on: ubuntu-latest steps: diff --git a/.gitignore b/.gitignore index 31bd9be85d7d6656702340c2683eb0deb978e48a..cb95e64ee31c2cd1f1d3771bbc6a9324bb409cb7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,7 @@ .idea/ +.vscode/ /target data/ tarpaulin-report.html -lcov.info \ No newline at end of file +lcov.info +site diff --git a/README.md b/README.md index 9023bf0134326aded0556529d58b0ef3c6c1ea41..94837937d791f0bc8371c4e073c650c3e9f66ecb 100644 --- a/README.md +++ b/README.md @@ -19,327 +19,67 @@ This implementation has several features to make usage as simple as possible. * It has a lot of hooks options, and hooks can be combined. * Highly configurable; -## Installation +Please check out [docs](https://s3rius.github.io/rustus/) for more information about configuration and deploy. -You can download binaries from a [releases page](https://github.com/s3rius/rustus/releases). +## Installation -If you want to use docker, you can use official images from [s3rius/rustus](https://hub.docker.com/r/s3rius/rustus/): +You can install rustus by 4 different ways. -```bash -docker run --rm -it -p 1081:1081 s3rius/rustus:latest -``` +### From source -If we don't have a binary file for your operating system you can build it -with [cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html). +To build it from source rust must be installed. +Preferred version is 1.59.0. ```bash git clone https://github.com/s3rius/rustus.git cd rustus cargo install --path . --features=all ``` +Also you can speedup build by disabling some features. -### Supported data storages - -Right now you can only use `file-storage` to store uploads data. The only two options you can adjust are: - -* uploads directory -* directory structure - -To upload files in a custom directory other than `./data` -you can provide a `--data-dir` parameter. - -```bash -rustus --data-dir "./files" -``` - -If you have a lot of uploads, you don't want to store all your files in a flat structure. So you can set a directory -structure for your uploads. - -```bash -rustus --dir-structure="{env[HOSTNAME]}/{year}/{month}/{day}" -``` - -```bash -tree data -data -├── 0bd911d4054d41c6a3ad54be67ee3e66.info -├── 5bc9c62384494c439e2a064b82a39cc6.info -└── rtus-68cb5b8746-5mgw9 - └── 2022 - └── 1 - └── 8 - ├── 0bd911d4054d41c6a3ad54be67ee3e66 - └── 5bc9c62384494c439e2a064b82a39cc6 - -``` - -**Important note:** if you use variable that doesn't exist or incorrect like invalid env variable, it results in an -error and the directory structure will become flat again. - -As you can see all info files are stored in a flat structure. It cannot be changed if you use file info storage. In -order to get rid of those `.info` files use different info storages. - -## Info storages - -The info storage is a database or directory. The main goal is to keep track of uploads. Rustus stores information about -download in json format inside database. - -File storage is used by default. You can customize the directory of an .info files by providing `--info-dir` parameter. - -```bash -rustus --info-dir "./info_dir" -``` - -If you want to choose different storage you have to specify its type and connection string. - -```bash -# Redis info storage -rustus --info-storage redis-info-storage --info-db-dsn "redis://localhost" -# PostgreSQL info storage -rustus --info-storage db-info-storage --info-db-dsn "postgres://rustus:rustus@192.168.1.89:5440/rustus" -# SQLite3 info storage -rustus --info-storage db-info-storage --info-db-dsn "sqlite:////test.sqlite3" -# MySQL -rustus --info-storage db-info-storage --info-db-dsn "mysql://rustus:rustus@192.168.1.89:3306/rustus" -``` - -## Hooks - -Rustus supports several event hooks, such as: - -* File hooks; -* HTTP hooks; -* AMQP hooks. - -You can combine them, but you have to be careful, since AMQP hooks won't block uploading. - -If you want to check the "Authorization" header value or validate some information, you have to use webhooks or File -hooks. - -Hooks have priorities: file hooks are the most important, then goes webhooks and AMQP hooks have the least priority. If -pre-create hook failed, the upload would not start. Of course, since AMQP is a protocol that doesn't allow you to track -responses we can't validate anything to stop uploading. - -Hooks can have 2 formats - -default: - -```json -{ - "upload": { - "id": "", - "offset": 0, - "length": 39729945, - "path": null, - "created_at": 1641620821, - "deferred_size": false, - "metadata": { - "filename": "38MB_video.mp4", - "meme": "hehe2" - } - }, - "request": { - "URI": "/files", - "method": "POST", - "remote_addr": "127.0.0.1", - "headers": { - "accept-encoding": "gzip, deflate", - "connection": "keep-alive", - "host": "localhost:1081", - "upload-metadata": "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=", - "tus-resumable": "1.0.0", - "content-length": "0", - "upload-length": "39729945", - "user-agent": "python-requests/2.26.0", - "accept": "*/*" - } - } -} -``` - -tusd: - -```json -{ - "Upload": { - "ID": "", - "Offset": 0, - "Size": 39729945, - "IsFinal": true, - "IsPartial": false, - "PartialUploads": null, - "SizeIsDeferred": false, - "Metadata": { - "filename": "38MB_video.mp4", - "meme": "hehe2" - }, - "Storage": { - "Type": "filestore", - "Path": null - } - }, - "HTTPRequest": { - "URI": "/files", - "Method": "POST", - "RemoteAddr": "127.0.0.1", - "Header": { - "host": [ - "localhost:1081" - ], - "user-agent": [ - "python-requests/2.26.0" - ], - "accept": [ - "*/*" - ], - "content-length": [ - "0" - ], - "upload-metadata": [ - "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=" - ], - "connection": [ - "keep-alive" - ], - "tus-resumable": [ - "1.0.0" - ], - "upload-length": [ - "39729945" - ], - "accept-encoding": [ - "gzip, deflate" - ] - } - } -} -``` - -### File hooks - -Rustus can work with two types of file hooks. - -1. Single file hook; -2. Hooks directory. - -The main difference is that hook name is passed as a command line parameter to a single file hook, but if you use hooks -directory then hook name is used to determine a file to call. Let's take a look at the examples - -Example of a single file hook: - -```bash -#!/bin/bash - -# Hook name would be "pre-create", "post-create" and so on. -HOOK_NAME="$1" -HOOK_INFO="$2" -MEME="$(echo "$HOOK_INFO" | jq ".upload .metadata .meme" | xargs)" - -# Here we check if name in metadata is equal to pepe. -if [[ $MEME = "pepe" ]]; then - echo "This meme isn't allowed" 1>&2; - exit 1 -fi -``` - -As you can see it uses first CLI parameter as a hook name and all hook data is received from stdin. - -Let's make it executable - -```bash -chmod +x "hooks/unified_hook" -``` - -To use it you can add parameter - -```bash -rustus --hooks-file "hooks/unified_hook" -``` - -This hook is going to ignore any file that has "pepe" in metadata. +Available features: -Let's create a hook directory. +* `amqp_notifier` - adds amqp protocol support for notifying about upload status; +* `db_info_storage` - adds support for storing information about upload in different databases (Postgres, MySQL, SQLite); +* `http_notifier` - adds support for notifying about upload status via http protocol; +* `redis_info_storage` - adds support for storing information about upload in redis database; +* `hashers` - adds support for checksum verification; +* `all` - enables all rustus features. -```bash -⯠tree hooks -hooks -├── post-create -├── post-finish -├── post-receive -├── post-terminate -└── pre-create -``` - -Every file in this directory has an executable flag. So you can specify a parameter to use hooks directory. - -```bash -rustus --hooks-dir "hooks" -``` +All precompiled binaries have all features enabled. -In this case rustus will append a hook name to the directory you pointed at and call it as an executable. +### With cargo -Information about hook is passed as a first parameter, as if you call script by running: +If you have cargo installed maybe it would be easier to +install it directly from crates.io. ```bash -./hooks/pre-create '{"id": "someid", ...}' +cargo install rustus --features=all ``` -### Http Hooks +### Binaries -Http hooks use http protocol to notify you about an upload. You can use HTTP hooks to verify Authorization. - -Let's create a FastAPI application that listens to hooks and checks the authorization header. +All precompiled binaries available on github releases page. +You can download binaries from [here](https://github.com/s3rius/rustus/releases), unpack it and run. ```bash -# Installing dependencies -pip install fastapi uvicorn +./rustus ``` -```python -# server.py -from fastapi import FastAPI, Header, HTTPException -from typing import Optional - -app = FastAPI() +Make sure that you download version for your cpu and os. +### Using docker -@app.post("/hooks") -def hook( - authorization: Optional[str] = Header(None), - hook_name: Optional[str] = Header(None), -): - print(f"Received: {hook_name}") - if authorization != "Bearer jwt": - raise HTTPException(401) - return None -``` +One of the most simple ways to run rustus is docker. -Now we can start a server. +Rustus has two containers for each version. +1. debian based image +2. alpine based image -```bash -uvicorn server:app --port 8080 -``` +Alpine based images are more lightweight than debian -Now you can start rustus, and it will check if Authorization header has a correct value. +To run rustus you just need to run this command ```bash -rustus --hooks-http-urls "http://localhost:8000/hooks" --hooks-http-proxy-headers "Authorization" -``` - -### AMQP hooks - -All hooks can be sent with an AMQP protocol. - -For example if you have a rabbitMQ you can use it. - -```bash -rustus --hooks-amqp-url "amqp://guest:guest@localhost" --hooks-amqp-exchange "my_exchange" -``` - -This command will create an exchange called "rustus" and queues for every hook. - -Every hook is published with routing key "rustus.{hook_name}" like -"rustus.post-create" or "rustus.pre-create" and so on. - -The problem with AMQP hooks is that you can't block the upload. To do this you have to use HTTP or File hooks. But with -AMQP your uploads become non-blocking which is definitely a good thing. \ No newline at end of file +docker run --rm -p "1081:1081" -d s3rius/rustus --log-level "DEBUG" +``` \ No newline at end of file diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000000000000000000000000000000000000..5e6f7f71b4760037d1c9784d7a81b8dc57c4e5b7 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,252 @@ +--- +title: Configuration +description: "How to configure Rusts" +--- + +Rustus is highly configurable you can configure rustus with CLI or you can use environment variables. + +!!! info + Some options can be passed only through as CLI parameters + +!!! info + + Information about hooks you can find on [Hooks page](../hooks). + + +## Configuring server + +We use actix to run server. +You can configure on wich `host` and `port` rustus is listenging. +Also you can configure number of actix `workers` that handle connections. + +`--max-body-size` is the max number of bytes that users can send in request body. + +`--url` is a base URL for all tus requests. + +`--workers` by default is euqal to number of physical CPU cores. Edit it carefully. + +=== "CLI" + + ``` bash + rustus --host "0.0.0.0" \ + --port 1081 \ + --workers 8 \ + --max-body-size 1000000 \ + --url "/files" \ + --log-level "INFO" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_SERVER_HOST="0.0.0.0" + export RUSTUS_SERVER_PORT="1081" + export RUSTUS_SERVER_WORKERS="8" + export RUSTUS_MAX_BODY_SIZE="1000000" + export RUSTUS_URL="/files" + export RUSTUS_LOG_LEVEL="INFO" + + rustus + ``` + + +## Configuring data storage + + +!!!info + + Currently only file storage is available, + so if you pass to `--storage` parameter other than `file-storage` you will get an error. + + Also you **can not** pass `--force-fsync` through environment variables. + + +`--storage` is a type of data storage to be used. + +`--data-dir` is a path to the directory where all files are stored. + +`--dir-structure` is a pattern of a directory structure inside data dir. +You can use variables within the pattern. + +Available variables: + +* `{year}` - current year; +* `{month}` - current month number from 1 to 12; +* `{day}` - current day number from 1 to 31; +* `{hour}` - hour number from 0 to 23; +* `{minute}` - minute number from 0 to 59; +* `{env[ENV_NAME]}` - environment variable where `ENV_NAME` is name of your variable. + +!!! note + + All environment variables are saved in memory during rustus startup. + So you cannot change variable dynamically. Even if you change env used in + structure pattern it won't change. + +For example if you use `{env[HOSTNAME]}/{year}/{month}/{day}` as your dir-structure, rustus stores files like: + +``` bash +$ tree data +data +└── rtus-68cb5b8746-5mgw9 + └── 2022 + └── 1 + └── 8 + ├── 0bd911d4054d41c6a3ad54be67ee3e66 + └── 5bc9c62384494c439e2a064b82a39cc6 +``` + +=== "CLI" + + ``` bash + rustus --force-fsync \ + --storage "file-storage" \ + --data-dir "./data/" \ + --dir-structure "{year}/{month}/{day}" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_STORAGE="file-storage" + export RUSTUS_DATA_DIR="./data/" + export RUSTUS_DIR_STRUCTURE="{year}/{month}/{day}" + + rustus --force-fsync + ``` + +## Configuring info storage + +Info storages are used to store information +about file uploads. These storages **must** be persistent, +because every time chunk is uploaded rustus updates information +about upload. And when someone wants to download file, information +about it requested from storage to get actual path of an upload. + +Available info storages: + +* `file-info-storage` - stores information in files on disk; +* `redis-info-storage` - information is stored in redis; +* `db-info-storage` - information is stored in database; + +### File info storage + +file info storage stores information in files on disk. +It's default info storage. Every download has it's own associated file. +All .info files stored in flat structure so it's the least preferable way of +storing information about uploads. But if you don't plan to have many uploads, it may fit well. + +`--info-dir` - directory where all .info file will be stored (default is `./data`). + + +=== "CLI" + + ``` bash + rustus --force-fsync \ + --storage "file-info-storage" \ + --info-dir "./data" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_INFO_STORAGE="file-info-storage" + export RUSTUS_INFO_DIR="./data" + + rustus + ``` + +### Redis info storage + +Redis db is a good way to store information. + +!!! note + + If you're using redis as a cluster + you must provide connection string for master redis server. + Since rustus need to have latest information and it writes a lot. + +`--info-db-dsn` - connection string for your redis database. +It's required if redis-info-storage is chosen. + +=== "CLI" + + ``` bash + rustus --force-fsync \ + --storage "redis-info-storage" \ + --info-db-dsn "redis://localhost/0" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_INFO_STORAGE="redis-info-storage" + export RUSTUS_INFO_DB_DSN="redis://localhost" + + rustus + ``` + + +### DB info storage + +Rustus can store information about upload in database. + +It's a good and reliable option. But rustus can't work +with replicas since it requires most recent information +about uploads. + +You can use `postgresql`, `mysql` or even `sqlite` schemas to +connect to database. + +`--info-db-dsn` - connection string for your database. + +=== "CLI" + + ``` bash + rustus --force-fsync \ + --storage "db-info-storage" \ + --info-db-dsn "postgresql://user:password@localhost/db" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_INFO_STORAGE="redis-info-storage" + export RUSTUS_INFO_DB_DSN="postgresql://user:password@localhost/db" + + rustus + ``` + +## Configuring TUS + +Since tus protocol offers extensibility you can turn off some protocol extensions. + +Available extensions: + +* `getting` - rustus specific extension that helps you download uploaded files with get request; +* `creation` - helps you to create files (It's like a core feature you better have this enabled); +* `termination` - allows you to delete uploads with DELETE request; +* `creation-with-upload` - allows you to write first bytes of a file while creating; +* `creation-defer-length` - allows you to create file without specifying file length; +* `concatenation` - allows you to concatenate finished partial uploads. +* `checksum` - allows you to verify checksum of every batch. + +You can read more about extensions on [official web-site](https://tus.io/protocols/resumable-upload.html#protocol-extensions). + +`--tus-extensions` - a list of enabled extensions. + +By default all extensions are enabled. + +=== "CLI" + + ``` bash + rustus --tus-extensions "getting,creation,termination,creation-with-upload,creation-defer-length,concatenation,checksum" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_TUS_EXTENSIONS="getting,creation,termination,creation-with-upload,creation-defer-length,concatenation,checksum" + + rustus + ``` diff --git a/docs/deploy.md b/docs/deploy.md new file mode 100644 index 0000000000000000000000000000000000000000..2d180e029ca170f86175484305a51445012b0d97 --- /dev/null +++ b/docs/deploy.md @@ -0,0 +1,171 @@ +--- +title: "Deployment" +description: "How to deploy rustus" +--- + +# Deployment + +Deploying an application is always a challenge. Rustus was made to make deployment as easy as possible. +Since this application works with files so if you want to scale number of rustus instances you +have to somehow make different rustus instances to work with the same data or info directory. + +## Docker compose + +``` yaml title="docker-compose.yml" +# This is super simple configuration +version: "3.7" + +services: + rustus: + image: s3rius/rustus + volumes: + # Volume mouted to default data directory + # So it's available across multiple containers. + - rustus_data_volume:/app/data + +volumes: + rustus_data_volume: +``` + +After running `docker compose up` you will see rustus startup logs. + +If you want to deploy multiple rustus instances you can simply +use config as this one: + +``` yaml title="docker-compose.yml" +version: "3.7" + +services: + proxy: + image: jwilder/nginx-proxy:alpine + container_name: proxy + # Actual proxy ports. + ports: + - 8080:80 + volumes: + # This thing helps to locate containers + # within this composition to generate nginx config. + - /var/run/docker.sock:/tmp/docker.sock:ro + + rustus: + image: s3rius/rustus + ports: + # Ports definition + # To generate correct nginx config. + - 1081 + volumes: + # Volume mouted to default data directory + # So it's available across multiple containers. + - rustus_data_volume:/app/data + environment: + # Idk why but without this variable + # load balancing with jwilder/nginx-proxy doesn't work. + VIRTUAL_HOST: localhost + +volumes: + rustus_data_volume: # This is named volume +``` + +The main idea is that traffic that comes into nginx-proxy +is routed in one of multiple rustus containers. +Here I used `jwilder/nginx-proxy` but you can use other +reverse-proxies such as raw `nginx proxy` or `traefik`. + +Now you can run multiple rustus instnaces like this. + +```bash +docker compose up --scale rustus=3 +``` + +After that you can upload files to `http://localhost:8080/files` + +## Kubernetes + +Configuration for kubernetes is almost the same as docker. +But the most preferable way is an official helm chart. + +Load balancing is done by kubernetes so you just have to +create volume to mount data and info directories. + +## Helm + +You can install rustus by running this set of commands: +``` bash +helm repo add "rustus" "https://s3rius.github.io/rustus/helm_releases" +helm repo update +helm repo install "rustus/rustus" +``` + +### Configuration +But of course it can be configured. + +``` bash +# You can download basic configuration by running +helm show values "rustus/rustus" > values.yml +``` + +By editing values.yml you can configure many different options. + +!!! warning + + For production use you must provide and mount PersistentVolumeClaim + in order to scale rustus. + + This helm chart has only one replica by default. + +### Persistence + +You can add pvc mount by editing `persistence` section. +The most preferable way is to create `PersistentVolume` and `PersistentVolumeClaim` +before installing this chart. + +After you created claim you can apply this values file to mount your claim into rustus. +``` yaml title="values.yml" +persistence: + enabled: true + existingClaim: "rustus-pvc" +``` + +!!! warning + + Currently there's no ability to create multiple mounts + and if you use file info storage you must specify the same direcotry + as you specified for data storage. + + But it would be better to use other type of info-storage. + +### Subcharts + +For example if you want to use redis as your info storage. + +``` yaml title="values.yml" +env: + RUSTUS_INFO_STORAGE: redis-info-storage + RUSTUS_INFO_DB_DSN: redis://:pass@rustus-redis-master/0 + +redis: + enabled: true +``` + +`redis`, `postgersql` and `mysql` are subcharts. + +You can find information about configuration these subcharts here: + +* [Repo](https://github.com/bitnami/charts/tree/master/bitnami/redis) for redis; +* [Repo](https://github.com/bitnami/charts/tree/master/bitnami/mysql) for mysql; +* [Repo](https://github.com/bitnami/charts/tree/master/bitnami/postgresql) for postgresql. + +In production you may ignore these subcharts to deploy your own redis or mysql or postgresql. + +After you done editing `values.yml` you can apply the configuration like this: + +``` bash +helm upgrade \ +--install \ # Install chart if it's not installed +--namespace rustus \ # k8s namespace +--create-namespace \ # Creates namespace if it doesn't exist +--atomic \ # Ensures that everything is deployed correctly +--values "values.yml" \ # Link to values.yml file +"rustus" \ # name of a release +"rustus/rustus" # Name of the chart +``` \ No newline at end of file diff --git a/docs/hooks.md b/docs/hooks.md new file mode 100644 index 0000000000000000000000000000000000000000..56e3ff2397806c60ca65e7f2228fa06eef09c6ea --- /dev/null +++ b/docs/hooks.md @@ -0,0 +1,334 @@ +--- +title: Setting up hooks +desctiption: Setting up hooks to notify other systems about uploads +--- + +Rustus can notify about uploads using hooks. +This is useful when you integrate rustus in your architecture. +Apps can keep track of every upload using this feature. + +Rustus has different event types for different moments of an upload's lifecycle. + +* `pre-create` - This hook means that someone wants to create an upload; +* `post-create` - someone successfully created an upload; +* `post-receive` - someone uploaded a new part of an upload; +* `post-terminate` - someone deleted upload; +* `post-finish` - someone finished uploading file. + +!!! note + + Pre-create hook is very important. + If at least one of hooks fails, upload is canceled. + + But AMQP hooks won't cancel the upload, since it's non blocking type of hooks. + +You can disable some hooks by using `--hooks` parameter. + +=== "CLI" + + ``` bash + rustus --hooks "pre-create,post-create,post-receive,post-terminate,post-finish" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_HOOKS="pre-create,post-create,post-receive,post-terminate,post-finish" + + rustus + ``` + + +## Fomat + +Information about every hook using `JSON` format. +Format can be configured using `--hooks-format` parameter or `RUSTUS_HOOKS_FORMAT` environment variable. + +Available formats: + +* default +* tusd + +=== "default" + + ``` json + { + "upload": { + "id": "", + "offset": 0, + "length": 39729945, + "path": null, + "created_at": 1641620821, + "deferred_size": false, + "metadata": { + "filename": "38MB_video.mp4", + "meme": "hehe2" + } + }, + "request": { + "URI": "/files", + "method": "POST", + "remote_addr": "127.0.0.1", + "headers": { + "accept-encoding": "gzip, deflate", + "connection": "keep-alive", + "host": "localhost:1081", + "upload-metadata": "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=", + "tus-resumable": "1.0.0", + "content-length": "0", + "upload-length": "39729945", + "user-agent": "python-requests/2.26.0", + "accept": "*/*" + } + } + } + ``` + +=== "tusd" + + ``` json + { + "Upload": { + "ID": "", + "Offset": 0, + "Size": 39729945, + "IsFinal": true, + "IsPartial": false, + "PartialUploads": null, + "SizeIsDeferred": false, + "Metadata": { + "filename": "38MB_video.mp4", + "meme": "hehe2" + }, + "Storage": { + "Type": "filestore", + "Path": null + } + }, + "HTTPRequest": { + "URI": "/files", + "Method": "POST", + "RemoteAddr": "127.0.0.1", + "Header": { + "host": [ + "localhost:1081" + ], + "user-agent": [ + "python-requests/2.26.0" + ], + "accept": [ + "*/*" + ], + "content-length": [ + "0" + ], + "upload-metadata": [ + "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=" + ], + "connection": [ + "keep-alive" + ], + "tus-resumable": [ + "1.0.0" + ], + "upload-length": [ + "39729945" + ], + "accept-encoding": [ + "gzip, deflate" + ] + } + } + } + ``` + +## Hook types + +Rustus offers multiple types of Hooks. We'll take a brief look on each type. + +### File hooks + +Rustus can work with two types of file hooks. + +* Single file hook +* Hooks directory + +The main difference is that in case if use single file hook, hook name is passed as a command line argument +to an executable file, but if you use hooks directory then hook name is used to determine a file to call. Let's take a look at the examples. + +Parameters: +* `--hooks-file` - path to an executable file; +* `--hooks-dir` - path to a directory with executable files. + +``` bash title="single_file_hook.sh" +#!/bin/bash + +# Hook name would be "pre-create", "post-create" and so on. +HOOK_NAME="$1" +HOOK_INFO="$2" +MEME="$(echo "$HOOK_INFO" | jq ".upload .metadata .meme" | xargs)" + +# Here we check if name in metadata is equal to pepe. +if [[ $MEME = "pepe" ]]; then + echo "This meme isn't allowed" 1>&2; + exit 1 +fi +``` + +As you can see it uses first CLI parameter as a hook name and all hook data is received from the second one. +Let's make it executable and make rustus use this hook. + +=== "CLI" + + ``` bash + chmod +x "hooks/single_file_hook.sh" + + rustus --hooks-file "hooks/single_file_hook.sh" + ``` + +=== "ENV" + + ``` bash + chmod +x "hooks/single_file_hook.sh" + export RUSTUS_HOOKS_FILE="hooks/single_file_hook.sh" + + rustus + ``` + +If you would like to use directory hooks you must create directory with the following structure: + +```tree +hooks +├── post-create +├── post-finish +├── post-receive +├── post-terminate +└── pre-create +``` + +!!! warning + If some hook file isn't found, rustus throws an error. + In case with `pre-create` hook it can be fatal. + +### Http Hooks + +Http hooks use HTTP to send `POST` requests to some endpoint. + +Configuration parameters: + +* `--hooks-http-proxy-headers` - list of headers to proxy (separated by commas) to listener's endpoint; +* `--hooks-http-urls` - list of absolute urls to send request to (separated by commas). + +!!! note + Hook names are passed as header called `Hook-Name`. + +=== "CLI" + + ``` bash + rustus --hooks-http-urls "https://httpbin.org/post" \ + --hooks-http-proxy-headers "Authorization" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_HOOKS_HTTP_URLS="https://httpbin.org/post" + export RUSTUS_HOOKS_HTTP_PROXY_HEADERS="Authorization" + + rustus + ``` + +#### Example application + +To be more verbose let's create simple web server that +handles uploads using [FastAPI](https://fastapi.tiangolo.com/). + +At first we need to install dependencies using pip. + +```bash +pip install fastapi uvicorn +``` + + +```python title="server.py" +from fastapi import FastAPI, Header, HTTPException +from typing import Optional + +app = FastAPI() + + +@app.post("/hooks") +def hook( + authorization: Optional[str] = Header(None), + hook_name: Optional[str] = Header(None), +): + print(f"Received: {hook_name}") + if authorization != "Bearer jwt": + raise HTTPException(401) + return None +``` + +Now we can run this server using uvicorn. + +```bash +uvicorn server:app --port 8080 +``` + +Let's configure rustus to use this server as a hook reciever. + +=== "CLI" + + ``` bash + rustus --hooks-http-urls "http://localhost:8000/hooks" \ + --hooks-http-proxy-headers "Authorization" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_HOOKS_HTTP_URLS="http://localhost:8000/hooks" + export RUSTUS_HOOKS_HTTP_PROXY_HEADERS="Authorization" + + rustus + ``` + +That's it. + + +### AMQP hooks + +AMQP hooks are used to store information about uploads using RabbitMQ. + +Configuration parameters: + +* `--hooks-amqp-url` - connection string to RabbitMQ; +* `--hooks-amqp-queues-prefix` - prefix for queues for every event queue; +* `--hooks-amqp-exchange` - name of exchange to use. + +This hook will send every message in an exchange with routing keys +like queues names. + +Queues are named like `{prefix}.{event type}`. Eg `rustus.pre-create` and so on. + +!!! warning + + Since we can't really track message delivery and responses + Rustus doesn't stop in any case. + +=== "CLI" + + ``` bash + rustus --hooks-amqp-url "amqp://guest:guest@localhost:5672" \ + --hooks-amqp-queues-prefix "rustus_queue" \ + --hooks-amqp-exchange "rustus" + ``` + +=== "ENV" + + ``` bash + export RUSTUS_HOOKS_AMQP_URL="amqp://guest:guest@localhost:5672" + export RUSTUS_HOOKS_AMQP_QUEUES_PREFIX="rustus_queue" + export RUSTUS_HOOKS_AMQP_EXCHANGE="rustus" + + rustus + ``` \ No newline at end of file diff --git a/docs/imgs/logo_horizontal.svg b/docs/imgs/logo_horizontal.svg new file mode 100644 index 0000000000000000000000000000000000000000..7a0aad50ef98ca2d77913d97329174cf5e5180a3 --- /dev/null +++ b/docs/imgs/logo_horizontal.svg @@ -0,0 +1 @@ +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 392 392"><defs><style>.cls-1{fill:url(#БезымÑнный_градиент_9);}.cls-2{fill:url(#БезымÑнный_градиент_9-2);}.cls-3{fill:#FFFFFF;}.cls-4{fill:#FFFFFF;}</style><linearGradient id="БезымÑнный_градиент_9" x1="213.3" y1="392.31" x2="213.3" y2="6.18" gradientUnits="userSpaceOnUse"><stop offset="0.25" stop-color="#FFFFFF"/><stop offset="0.34" stop-color="#FFFFFF"/><stop offset="0.48" stop-color="#FFFFFF"/><stop offset="0.66" stop-color="#FFFFFF"/><stop offset="0.75" stop-color="#FFFFFF"/></linearGradient><linearGradient id="БезымÑнный_градиент_9-2" x1="112.8" y1="392.31" x2="112.8" y2="6.18" xlink:href="#БезымÑнный_градиент_9"/></defs><g id="Слой_2" data-name="Слой 2"><g id="Слой_1-2" data-name="Слой 1"><path class="cls-1" d="M213.11,392.31q-53.36,0-83-30t-29.64-85.94V49.86c10.11-9.42,14.29-13.31,23.56-22h0c11.55-10.71,16.8-15.6,23.56-21.68V274.52q0,75.84,65.14,75.82,65.51,0,65.5-75.82v-120A23.55,23.55,0,0,1,301.81,131h.73a23.55,23.55,0,0,1,23.56,23.55V276.37q0,56-29.8,85.94T213.11,392.31Z"/><path class="cls-2" d="M29.8,362.31Q0,332.33,0,276.37V154.55A23.55,23.55,0,0,1,23.56,131h.73a23.55,23.55,0,0,1,23.56,23.55v120q0,75.84,65.5,75.82,65.15,0,65.14-75.82V6.18c6.76,6.08,12,11,23.56,21.68h0c9.27,8.69,13.45,12.58,23.56,22V276.37q0,56-29.64,85.94t-83,30Q59.62,392.31,29.8,362.31Z"/><path class="cls-3" d="M77.73,125.52A23,23,0,0,1,62.05,85.68l85.33-79.5a23,23,0,0,1,31.36,33.66L93.41,119.35A23,23,0,0,1,77.73,125.52Z"/><path class="cls-3" d="M248.37,125.52a23,23,0,0,1-15.68-6.17L147.38,39.84A23,23,0,1,1,178.74,6.18l85.31,79.5a23,23,0,0,1-15.68,39.84Z"/></g></g></svg> \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000000000000000000000000000000000000..eacebf7f944924e37485d1b9eb8ad9863d7715e5 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,82 @@ +--- +title: "Welcome page" +description: Rustus docs +--- + +<div align="left"> + <img src="https://raw.githubusercontent.com/s3rius/rustus/master/imgs/logo_horizontal.svg" alt="logo" width="500"> + <div> + <p></p> + <img alt="Docker Image Size (latest by date)" src="https://img.shields.io/docker/image-size/s3rius/rustus?sort=date&style=for-the-badge"> + <img alt="Docker Image Version (latest semver)" src="https://img.shields.io/docker/v/s3rius/rustus?style=for-the-badge"> + <img alt="GitHub" src="https://img.shields.io/github/license/s3rius/rustus?style=for-the-badge"> + </div> +</div> + +Rustus is a [TUS](https://tus.io) protocol implementation that helps you handle file uploads. + +This project has many features that makes it easy to integrate in your service. + + +## Installation + +You can install rustus by 4 different ways. + +### From source + +To build it from source rust must be installed. +Preferred version is 1.59.0. + +```bash +git clone https://github.com/s3rius/rustus.git +cd rustus +cargo install --path . --features=all +``` +Also you can speedup build by disabling some features. + +Available features: + +* `amqp_notifier` - adds amqp protocol support for notifying about upload status; +* `db_info_storage` - adds support for storing information about upload in different databases (Postgres, MySQL, SQLite); +* `http_notifier` - adds support for notifying about upload status via http protocol; +* `redis_info_storage` - adds support for storing information about upload in redis database; +* `hashers` - adds support for checksum verification; +* `all` - enables all rustus features. + +All precompiled binaries have all features enabled. + +### With cargo + +If you have cargo installed maybe it would be easier to +install it directly from crates.io. + +```bash +cargo install rustus --features=all +``` + +### Binaries + +All precompiled binaries available on github releases page. +You can download binaries from [here](https://github.com/s3rius/rustus/releases), unpack it and run. + +```bash +./rustus +``` + +Make sure that you download version for your cpu and os. + +### Using docker + +One of the most simple ways to run rustus is docker. + +Rustus has two containers for each version. +1. debian based image +2. alpine based image + +Alpine based images are more lightweight than debian + +To run rustus you just need to run this command + +```bash +docker run --rm -p "1081:1081" -d s3rius/rustus --log-level "DEBUG" +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000000000000000000000000000000000000..f70e948d6511683124eb87ee1da0a327a65d8699 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,50 @@ +site_name: "rustus" +site_url: "https://s3rius.github.io/rustus" +repo_url: https://github.com/s3rius/rustus +repo_name: s3rius/rustus +edit_uri: "" + +theme: + icon: + repo: fontawesome/brands/github + logo: imgs/logo_horizontal.svg + name: material + language: en + features: + - navigation.instant + - navigation.tracking + - navigation.expand + - search.suggest + - header.autohide + palette: + - media: "(prefers-color-scheme: light)" + scheme: default + primary: indigo + toggle: + icon: material/weather-night + name: Switch to dark mode + - media: "(prefers-color-scheme: dark)" + scheme: slate + primary: blue + toggle: + icon: material/weather-sunny + name: Switch to light mode + +plugins: + - search: + lang: en + + +extra: + generator: true + +markdown_extensions: + - admonition + - markdown.extensions.meta + - pymdownx.highlight: + anchor_linenums: true + - pymdownx.inlinehilite + - pymdownx.snippets + - pymdownx.superfences + - pymdownx.tabbed: + alternate_style: true \ No newline at end of file