Skip to content

Add anubis to filter spam/bot traffic#3788

Merged
mouse-reeve merged 2 commits into
bookwyrm-social:mainfrom
ilkka-ollakka:feat/use_anubis
Feb 2, 2026
Merged

Add anubis to filter spam/bot traffic#3788
mouse-reeve merged 2 commits into
bookwyrm-social:mainfrom
ilkka-ollakka:feat/use_anubis

Conversation

@ilkka-ollakka

@ilkka-ollakka ilkka-ollakka commented Jan 10, 2026

Copy link
Copy Markdown
Contributor

Description

Adds anubis (https://anubis.techaro.lol/) between nginx and django to reduce bot traffic.

Contains small default config to allow bookwyrm user-agent and configuration is based on anubis documentation default examples otherwise.

Anubis works with nginx in a flow that nginx terminates https/http connection, uses anubis as subrequest authenticator for request (https://anubis.techaro.lol/docs/admin/configuration/subrequest-auth) and if that succeeds, passes request to upstream django.

Using subrequest authentication flow seems to make the nginx config littlebit simpler to follow, as we are also doing caching and things within nginx.

What type of Pull Request is this?

  • Bug Fix
  • Enhancement
  • Plumbing / Internals / Dependencies
  • Refactor

Does this PR change settings or dependencies, or break something?

  • This PR changes or adds default settings, configuration, or .env values
  • This PR changes or adds dependencies
  • This PR introduces other breaking changes

Details of breaking or configuration changes (if any of above checked)

Adds anubis in request path between nginx and django, modifies nginx configs.

Documentation

  • New or amended documentation will be required if this PR is merged
  • I have created a matching pull request in the Documentation repository
  • I intend to create a matching pull request in the Documentation repository after this PR is merged

Needs documentation update on dockerless-installation.

Tests

  • My changes do not need new tests
  • All tests I have added are passing
  • I have written tests but need help to make them pass
  • I have not written tests and need help to write them

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

One issue with Anubis with this configuration is that with rootless-podman anubis doesn't get correct remote ip-address. This is due nginx doesn't get correct remote address in this flow.

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

One issue with Anubis with this configuration is that with rootless-podman anubis doesn't get correct remote ip-address. This is due nginx doesn't get correct remote address in this flow.

This should not be an issue in default use, as anubis doesn't do geo-ip/ASN-based filtering by default (https://anubis.techaro.lol/docs/admin/thoth#thoth-features)

@ilkka-ollakka ilkka-ollakka marked this pull request as ready for review January 10, 2026 18:53
@mouse-reeve

Copy link
Copy Markdown
Member

My brain's still kind of coming back online so I haven't had a chance to really parse this, but I'm very excited about anything that helps with the spam situation! Thank you so much for working on it

@ilkka-ollakka ilkka-ollakka changed the title Add anubis to nginx path Add anubis to filter spam/bot traffic Jan 11, 2026
@ilkka-ollakka ilkka-ollakka force-pushed the feat/use_anubis branch 2 times, most recently from 457e7a0 to 5ad3bdb Compare January 12, 2026 14:31
@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

I change the flow so that nginx uses anubis as subrequest autenticator (https://anubis.techaro.lol/docs/admin/configuration/subrequest-auth) to simplify how it is integrated in existing nginx configuration.

@ilkka-ollakka ilkka-ollakka force-pushed the feat/use_anubis branch 2 times, most recently from 96cf507 to 5c59432 Compare January 18, 2026 11:00
@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

I squashed the commits so it only contains the subrequest authentication changes, so it doesn't confuse what is actually changing. Otherwise I think it works quite ok and with my limited testing I didn't have spotted any case where it would have stopped legitimate use.

@mouse-reeve

Copy link
Copy Markdown
Member

I'm running into this issue creating the Anubis image locally:

bookwyrm-anubis-1  | {"time":"2026-01-22T22:43:29.117345339Z","level":"INFO","source":{"function":"main.main","file":"github.com/TecharoHQ/anubis/cmd/anubis/main.go","line":279},"msg":"starting up Anubis"}
bookwyrm-anubis-1  | {"time":"2026-01-22T22:43:29.117639089Z","level":"INFO","source":{"function":"main.main","file":"github.com/TecharoHQ/anubis/cmd/anubis/main.go","line":341},"msg":"loading policy file","fname":"/data/cfg/botPolicy.yaml"}
bookwyrm-anubis-1  | 2026/01/22 22:43:29 can't parse policy file: can't parse policy file /data/cfg/botPolicy.yaml: config is not valid:
bookwyrm-anubis-1  | store: configuration is invalid: bbolt: can't write to path

@ilkka-ollakka

ilkka-ollakka commented Jan 22, 2026

Copy link
Copy Markdown
Contributor Author

I'm running into this issue creating the Anubis image locally:

bookwyrm-anubis-1  | {"time":"2026-01-22T22:43:29.117345339Z","level":"INFO","source":{"function":"main.main","file":"github.com/TecharoHQ/anubis/cmd/anubis/main.go","line":279},"msg":"starting up Anubis"}
bookwyrm-anubis-1  | {"time":"2026-01-22T22:43:29.117639089Z","level":"INFO","source":{"function":"main.main","file":"github.com/TecharoHQ/anubis/cmd/anubis/main.go","line":341},"msg":"loading policy file","fname":"/data/cfg/botPolicy.yaml"}
bookwyrm-anubis-1  | 2026/01/22 22:43:29 can't parse policy file: can't parse policy file /data/cfg/botPolicy.yaml: config is not valid:
bookwyrm-anubis-1  | store: configuration is invalid: bbolt: can't write to path

hmmm.. looks like for some reason the anubis_data is not mounted to /data or something with the volume permissions are off :/

Does it make difference if you change the /data mount to something else and change the path accordingly in botPolicy.yaml file? Could be something different on volume definitions between podman and docker.

Other option is to try just take out the storage section in config and run it as memory storage. Which should be fine, it only means mostly that browsers are rechecked if anubis restarts.

@mouse-reeve

Copy link
Copy Markdown
Member

How do I change it in the policy? It seems like it's loading that file from github?

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

How do I change it in the policy? It seems like it's loading that file from github?

it is the anubis/botPolicy.yaml file that has all the configs. It references to github to see what the defaults are,but it doesnt load things from there.

I'll add comment on changed files to highlight what to change.

Comment thread anubis/botPolicy.yaml Outdated
Comment thread docker-compose.yml
@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

@mouse-reeve let me know if you have still issues with this PR, I can also modify the PR changes to simplify it so it doesn't need separate volume.

@mouse-reeve

Copy link
Copy Markdown
Member

Removing lines 11 to 14 in the botPolicy file got the anubis image running, but I'm hitting this error in nginx:

bookwyrm-nginx-1  | 2026/01/23 19:12:35 [emerg] 1#1: host not found in upstream "anubis:8000" in /etc/nginx/conf.d/default.conf:11
bookwyrm-nginx-1  | nginx: [emerg] host not found in upstream "anubis:8000" in /etc/nginx/conf.d/default.conf:11

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

Removing lines 11 to 14 in the botPolicy file got the anubis image running, but I'm hitting this error in nginx:

bookwyrm-nginx-1  | 2026/01/23 19:12:35 [emerg] 1#1: host not found in upstream "anubis:8000" in /etc/nginx/conf.d/default.conf:11
bookwyrm-nginx-1  | nginx: [emerg] host not found in upstream "anubis:8000" in /etc/nginx/conf.d/default.conf:11

that sounds strange, as docker/podman should resolve the container-name same with anubis what it does with web -container on upstreams. There are no other errors visible from anubis or nginx ?

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

One thing that could be, both web and anubis binds to 8000 port in this setup. If the docker enviroment pushes those both to local interfaces they could conflict. I'll try to check docker setup for myself to check if it works differently in this case than podman.

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

I was able to reproduce the problem, and issue is that I forgot to add the network: part in anubis service similar what nginx and other have.

So in podman it didn't matter but in docker it did, so it caused anubis and nginx ending up in different networks in docker side. I'll update the PR accordingly.

@ilkka-ollakka ilkka-ollakka force-pushed the feat/use_anubis branch 2 times, most recently from adc94de to ab68d0d Compare January 24, 2026 15:18
@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

Rebased on top of current main, reworked config so it works correctly with docker also and removed persistent key/storage use, as I don't think we need it currently.

@ilkka-ollakka ilkka-ollakka force-pushed the feat/use_anubis branch 2 times, most recently from d85382b to 7330ceb Compare January 24, 2026 18:37
@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

Removed the domain config from anubis and host in redirect so it works in dev-mode where port is defined and domain is localhost

@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

just note for myself, currently goes haywire with dev-setup in nonstandard port, i.e. localhost:1333 directly.

Contains small default config and configuration is based on anubis documentation default example
* add link to default config content in config
* configure anubis to work as subrequest authenticatior, this
  makes the nginx configuration littlebit more straightforward to follow
* turn off absolute_redirect in nginx, this caused odd issues with development setup where location
  is localhost:1333
* use relative path-only redirect from anubis instead of absolute url for same reason

anubix fixup
@ilkka-ollakka

Copy link
Copy Markdown
Contributor Author

Turns out nginx absolute_redirect on; setting seemed to cause the odd behaviour, when turning it off, things seems to work correctly with http://localhost:1333, reverse_proxy with ngrok and https setups.

Protection is not as tight as it could be, but I think in this case it is more compatible in all the flows and improves by default things. Setting REDIRECT_DOMAINS etc can be done each admin per site if they see the need for it.

@mouse-reeve

Copy link
Copy Markdown
Member

I tried this again and I'm very sorry to report that I'm still getting

bookwyrm-nginx-1  | 2026/02/02 21:36:40 [emerg] 1#1: host not found in upstream "anubis:8000" in /etc/nginx/conf.d/default.conf:11
bookwyrm-nginx-1  | nginx: [emerg] host not found in upstream "anubis:8000" in /etc/nginx/conf.d/default.conf:11

@mouse-reeve

Copy link
Copy Markdown
Member

Wait nevermind!! I just hadn't properly restarted. It seems to be working!

@mouse-reeve mouse-reeve merged commit ce14408 into bookwyrm-social:main Feb 2, 2026
9 checks passed
@ilkka-ollakka ilkka-ollakka deleted the feat/use_anubis branch February 3, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use anubis with bookwyrm to reduce spam/bots

3 participants