Add anubis to filter spam/bot traffic#3788
Conversation
0060d1b to
c715bdc
Compare
|
One issue with Anubis with this configuration is that with rootless-podman anubis doesn't get correct remote ip-address. This is due nginx doesn't get correct remote address in this flow. |
This should not be an issue in default use, as anubis doesn't do geo-ip/ASN-based filtering by default (https://anubis.techaro.lol/docs/admin/thoth#thoth-features) |
|
My brain's still kind of coming back online so I haven't had a chance to really parse this, but I'm very excited about anything that helps with the spam situation! Thank you so much for working on it |
457e7a0 to
5ad3bdb
Compare
|
I change the flow so that nginx uses anubis as subrequest autenticator (https://anubis.techaro.lol/docs/admin/configuration/subrequest-auth) to simplify how it is integrated in existing nginx configuration. |
96cf507 to
5c59432
Compare
|
I squashed the commits so it only contains the subrequest authentication changes, so it doesn't confuse what is actually changing. Otherwise I think it works quite ok and with my limited testing I didn't have spotted any case where it would have stopped legitimate use. |
|
I'm running into this issue creating the Anubis image locally: |
hmmm.. looks like for some reason the anubis_data is not mounted to /data or something with the volume permissions are off :/ Does it make difference if you change the /data mount to something else and change the path accordingly in botPolicy.yaml file? Could be something different on volume definitions between podman and docker. Other option is to try just take out the storage section in config and run it as memory storage. Which should be fine, it only means mostly that browsers are rechecked if anubis restarts. |
|
How do I change it in the policy? It seems like it's loading that file from github? |
it is the anubis/botPolicy.yaml file that has all the configs. It references to github to see what the defaults are,but it doesnt load things from there. I'll add comment on changed files to highlight what to change. |
|
@mouse-reeve let me know if you have still issues with this PR, I can also modify the PR changes to simplify it so it doesn't need separate volume. |
|
Removing lines 11 to 14 in the botPolicy file got the anubis image running, but I'm hitting this error in nginx: |
that sounds strange, as docker/podman should resolve the container-name same with anubis what it does with web -container on upstreams. There are no other errors visible from anubis or nginx ? |
|
One thing that could be, both web and anubis binds to 8000 port in this setup. If the docker enviroment pushes those both to local interfaces they could conflict. I'll try to check docker setup for myself to check if it works differently in this case than podman. |
|
I was able to reproduce the problem, and issue is that I forgot to add the So in podman it didn't matter but in docker it did, so it caused anubis and nginx ending up in different networks in docker side. I'll update the PR accordingly. |
adc94de to
ab68d0d
Compare
|
Rebased on top of current main, reworked config so it works correctly with docker also and removed persistent key/storage use, as I don't think we need it currently. |
d85382b to
7330ceb
Compare
|
Removed the domain config from anubis and host in redirect so it works in dev-mode where port is defined and domain is localhost |
|
just note for myself, currently goes haywire with dev-setup in nonstandard port, i.e. localhost:1333 directly. |
Contains small default config and configuration is based on anubis documentation default example * add link to default config content in config * configure anubis to work as subrequest authenticatior, this makes the nginx configuration littlebit more straightforward to follow * turn off absolute_redirect in nginx, this caused odd issues with development setup where location is localhost:1333 * use relative path-only redirect from anubis instead of absolute url for same reason anubix fixup
7330ceb to
01e0f0b
Compare
|
Turns out nginx Protection is not as tight as it could be, but I think in this case it is more compatible in all the flows and improves by default things. Setting REDIRECT_DOMAINS etc can be done each admin per site if they see the need for it. |
|
I tried this again and I'm very sorry to report that I'm still getting |
|
Wait nevermind!! I just hadn't properly restarted. It seems to be working! |
Description
Adds anubis (https://anubis.techaro.lol/) between nginx and django to reduce bot traffic.
Contains small default config to allow bookwyrm user-agent and configuration is based on anubis documentation default examples otherwise.
Anubis works with nginx in a flow that nginx terminates https/http connection, uses anubis as subrequest authenticator for request (https://anubis.techaro.lol/docs/admin/configuration/subrequest-auth) and if that succeeds, passes request to upstream django.
Using subrequest authentication flow seems to make the nginx config littlebit simpler to follow, as we are also doing caching and things within nginx.
What type of Pull Request is this?
Does this PR change settings or dependencies, or break something?
Details of breaking or configuration changes (if any of above checked)
Adds anubis in request path between nginx and django, modifies nginx configs.
Documentation
Needs documentation update on dockerless-installation.
Tests