/t/ - POWBlock - Developing a system agnostic Proof of Work backend

Options
Subject
Message	Max message length: 12000
files	Drag files here to upload or click here to select them 0.00 / 50.00 MB Max file size: 32.00 MB Total max file size: 50.00 MB Max files: 5 Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password	(used to delete files and posts)
Misc

POWBlock - Developing a system agnostic Proof of Work backend Anonymous 07/08/2024 (Mon) 01:36:13 No. 15564

You know, it occurs to me that despite being the site's sysadmin, and this being my bro Codexx's board, I literally never post my tech autism here. Maybe I should change that. If you'll pardon me making the equivalent of a /blog thread, I want to talk about some of my current ideas. As some anons are aware, over the past 4 and a half years I've become something of a guru at programming Varnish VCL. 8chan's entire network is bespoke and we've had to deal with requirements that no normal, Cuckflare-and-cloud-storage website needs. As a result I gained an absolute shitfuckload of hands on experience in hacking Varnish code. This has led to a minor obsession with pushing its boundaries, and making it do useful things that it was never designed to do. One thing that's been on my mind for literally *years* was the idea to make Varnish do the Cloudflare thing with a frontal POW and bot detection system. You can do this with Nginx and setups like this: https://github.com/C0nw0nk/Nginx-Lua-Anti-DDoS which is what Josh used as the basis of his "Kiwiflare" system. But no such tool exists for Varnish. I wanted to change that, and funnily enough I pulled it off several weeks back. It can be done though careful hacks of the error handling subsystem and abusing the object cache to track user session keys. But then I got to thinking, why hoard it? Since my original system was already using an external validator script (the prototype was written in Bash using netcat rofl) why couldn't I just move the system itself to a fast little backend that was designed to hook up with *any* frontend proxy? Not only would it save me a lot of work in the big picture, but other people might use it too. This became the idea for POWBlock. The first thing I decided was that I wanted it to run as a script. There are several reasons for this but the big ones are: Portability, lack of dependencies, and simplicity. The Bash prototype is obviously garbage from a production standpoint, but Python is fat and fucking slow. There is a reason why the Nginx POW system uses Lua. Lua is small, light, fast, low overhead because it was meant for embedded systems, its in every server repo, and has an HTTP server module with solid security and concurrency. Its not good enough to run a whole website off of, but if you want a zippy little script-based HTTP application like this it is far and away the best choice if its something you actually mean to deploy. Since I wasn't a Luafag I needed to start learning. After a week of reading and basic practice, I decided to move forward with some help from ChatGPT, henceforth called "the nig." I've found over the past year that this is great for practicing code. You learn the very basics, design your program yourself and break it into key sections and functions, throw your ideas and functions at the nig, and then learn as you go because you have to debug its shit and occasionally hand-write functions where the bot can't get it right. Its shockingly effective hands-on learning and I've been picking up Lua in record time. Its also nice to just hand the nig a big block of code you banged out and say "comment all this shit for me." Saves a lot of writing. Now you have the backstory, so let's get into the concept. POWBlock is a Lua-based mini-server that has exactly one job: It runs on a box somewhere and serves as an alternate backend for an existing frontend proxy. When a user visits the shielded site, the proxy can send them to POWBlock and force them through a Cloudflare-style Proof of Work browser check (and eventually other checks), give them a cookie to prove they passed, and send them back to the site. You can do this with Varnish, Nginx, and even fucking Apache. Control is handled by the frontend proxy, where you configure rate limiting and other basic anti-DoS functions to protect POWBlock itself from being flooded out, and it uses a system of custom HTTP headers to communicate and for security. My next post will be the prototype code for the server, so you can get an idea what it does.

Anonymous 07/08/2024 (Mon) 01:38:34 No. 15565

local http = require("http")
local crypto = require("crypto")
local json = require("json")

-- Define the PoW challenge parameters
local difficulty = 4  -- Number of leading zeros required in the hash
local cookie_name = "POW"  -- Name of the cookie
local cookie_max_age = 8 * 60 * 60  -- 8 hours in seconds

-- Rate limiting parameters
local MAX_ATTEMPTS_PER_MINUTE = 1
local attempts = {}

-- Predefined value for X-Lua-Pow header
local EXPECTED_X_LUA_POW = "magic"

-- Define maximum allowed hash size (in bytes)
local MAX_HASH_SIZE = 1024  -- Adjust as per your system's capacity and requirements

-- Session storage expiration time (in minutes)
local session_store_time = 10

-- URL store expiration time (in minutes)
local URL_STORE_EXPIRATION_TIME = 10
local url_store = {}

-- Map to store server nonces
local server_nonces = {}

-- Function to clear expired URLs from url_store
local function clear_expired_urls()
    local current_time = os.time()
    for client_id, entry in pairs(url_store) do
        if current_time - entry.timestamp > URL_STORE_EXPIRATION_TIME * 60 then
            url_store[client_id] = nil  -- Remove expired entry
        end
    end
end

-- Periodically clean up expired URLs (every minute)
local function periodic_url_cleanup()
    clear_expired_urls()
    -- Schedule the next cleanup after 60 seconds
    http.sleep(60 * 1000, periodic_url_cleanup)
end

-- Start periodic cleanup of expired URLs
http.start_time_thread(periodic_url_cleanup)

[Expand Post]
-- Function to handle PoW challenge request
local function handle_pow_request(request, response)
    -- Check X-Lua-Pow header first
    local x_lua_pow_value = request.headers["X-Lua-Pow"]
    if x_lua_pow_value ~= EXPECTED_X_LUA_POW then
        response.status = 403
        response:send("Forbidden.")
        return
    end
    
    if request.method == "GET" and request.path == "/pow" then
        -- Save the original requested URL
        local original_url = request.headers["X-Original-URL"] or (request.headers["X-Forwarded-Host"] .. request.path) -- Adjust this based on your server's header
        local client_id = request.headers["X-Client-ID"] or tostring(request.client.ip) -- Use client ID or IP as session identifier

        -- Generate a unique 2-byte server nonce for the client
        local server_nonce = crypto.rand_bytes(2)
        server_nonces[client_id] = {
            nonce = server_nonce,
            timestamp = os.time()
        }

        url_store[client_id] = {
            url = original_url,
            timestamp = os.time()
        }

        -- Serve the PoW challenge page with client-side PoW computation
        local html_response = [[
            <html>
            <head>
                <title>PoW Challenge</title>
                <script>
                    // Function to generate PoW challenge using Hashcash
                    function generate_pow_challenge() {
                        var version = "1";  // Hashcash version
                        var bits = "20";     // Bits parameter (controls difficulty)
                        var resource = "example.com";  // Resource to tie the challenge to (example)
                        var timestamp = Date.now();  // Generate a timestamp (milliseconds since epoch)
                        var message = version + ":" + bits + ":" + timestamp + ":" + resource;

                        var nonce = 0;
                        var pow_challenge = "";

                        while (true) {
                            nonce++;
                            var candidate = message + ":" + nonce;
                            var hash = sha256(candidate);  // Use SHA-256 for Hashcash

                            // Check if the hash meets the difficulty requirement
                            if (validate_hash(hash, ]] .. difficulty .. [[)) {
                                pow_challenge = candidate;
                                break;
                            }
                        }

                        return pow_challenge;
                    }

                    // Function to validate hash difficulty (number of leading zeros)
                    function validate_hash(hash, difficulty) {
                        return hash.substring(0, difficulty) === Array(difficulty + 1).join("0");
                    }

                    // Function to submit PoW solution to server
                    async function submitSolution(solution) {
                        try {
                            let response = await fetch('/verify', {
                                method: 'POST',
                                headers: {
                                    'Content-Type': 'application/json'
                                },
                                body: JSON.stringify({ solution, serverNonce: "]] .. server_nonce .. [[" })
                            });
                            let result = await response.text();
                            console.log(result);
                            if (result.includes("successfully")) {
                                alert("PoW challenge passed successfully!");

                                // Redirect back to the original URL
                                var originalURL = "]] .. original_url .. [[";
                                window.location.href = originalURL;
                            } else {
                                alert("Invalid PoW solution.");
                            }
                        } catch (error) {
                            console.error('Error submitting PoW solution:', error);
                            alert("Error submitting PoW solution. Please try again.");
                        }
                    }

                    // Function to handle submission click
                    function handleSubmission() {
                        var pow_challenge = generate_pow_challenge();
                        console.log("Generated PoW challenge:", pow_challenge);
                        submitSolution(pow_challenge);
                    }

                    // Function to enable the submit button after challenge computation
                    window.onload = function() {
                        var submitButton = document.getElementById('submitBtn');
                        submitButton.disabled = true;  // Initially disable the button

                        // Generate POW challenge
                        var pow_challenge = generate_pow_challenge();
                        console.log("Generated PoW challenge:", pow_challenge);

                        // Enable submit button once challenge is computed
                        submitButton.disabled = false;
                    };
                </script>
            </head>
            <body>
                <h1>Please wait while we validate your access...</h1>
                <p>Only one submission allowed per minute.</p>
                <button id="submitBtn" onclick="handleSubmission()" disabled>Submit Solution</button>
            </body>
            </html>
        ]]

        response.headers["Content-Type"] = "text/html"
        response:write(html_response)
        response:send()
    elseif request.method == "POST" and request.path == "/verify" then
        -- Validate PoW solution sent by client
        local data = request.body

        -- Check rate limit
        local client_ip = request.client.ip
        local timestamp = os.time()

        attempts[client_ip] = attempts[client_ip] or { count = 0, timestamp = 0 }

        if attempts[client_ip].count >= MAX_ATTEMPTS_PER_MINUTE and timestamp - attempts[client_ip].timestamp < 60 then
            response.status = 429  -- Too Many Requests
            response:send("Too many validation attempts. Please try again later.")
            return
        end

        -- Validate and process PoW solution
        if not data or not data.solution or type(data.solution) ~= "string" then
            response.status = 400
            response:send("Bad Request: Invalid solution format.")
            return
        end
        
        local solution = data.solution
        local client_server_nonce = data.serverNonce

        -- Validate server nonce
        local stored_nonce = server_nonces[client_ip]
        if not stored_nonce or stored_nonce.nonce ~= client_server_nonce then
            response.status = 403
            response:send("Forbidden: Invalid server nonce.")
            return
        end

        local is_valid = validate_pow_solution(solution)
        
        if is_valid then
            -- Successful PoW validation
            response:send("PoW challenge passed successfully!")

            -- Set POW cookie with X-Pow value
            local x_pow_value = request.headers["X-Pow"] or ""
            response.headers["Set-Cookie"] = cookie_name .. "=" .. x_pow_value .. "; Path=/; Max-Age=" .. cookie_max_age
        else
            -- Invalid PoW solution
            response.status = 403  -- or 401 Unauthorized based on scenario
            response:send("Unauthorized: Invalid PoW solution.")
        end

        -- Update attempt count and timestamp
        attempts[client_ip].count = attempts[client_ip].count + 1
        attempts[client_ip].timestamp = timestamp

        -- Remove server nonce after successful validation
        server_nonces[client_ip] = nil
    else
        -- Redirect invalid or non-existent paths to the PoW challenge
        response.status = 302  -- Redirect
        response.headers["Location"] = "/pow"
        response:send("")
    end
end

-- Function to validate PoW solution against the challenge
function validate_pow_solution(solution)
    if type(solution) ~= "string" then
        return false, "Invalid solution format."
    end

    -- Calculate hash size in bytes
    local hash_size = #solution

    -- Check if hash size exceeds the maximum allowed
    if hash_size > MAX_HASH_SIZE then
        return false, "Hash size exceeds allowed limit."
    end

    -- Perform actual hash validation logic (using SHA-256 as an example)
    local hash = crypto.digest("sha256", solution)
    return validate_hash(hash, difficulty), "Validation successful."
end

-- Create HTTP server instance
local server = http.create_server("0.0.0.0", 8081, function(request, response)
    -- Read X-Pow header for cached random string from Varnish
    local x_pow_value = request.headers["X-Pow"] or ""

    handle_pow_request(request, response)
end)

-- Start listening
server:listen()

print("PoW server listening on http://0.0.0.0:8081/")

Edited last time by Acidadmin on 07/08/2024 (Mon) 03:19:52.

Anonymous 07/08/2024 (Mon) 01:59:01 No. 15566

I'll break down some of what's going on here. >local EXPECTED_X_LUA_POW = "magic" The X-Lua-POW header passes a secret key, or magic number, that authenticates the traffic as coming from your actual frontend. This is to prevent clients from accessing the POWBlock server directly even if they learn its IP address. You configure your frontend to inject this header and set the value you want there. There's some limiting and cleanup functions to make sure that very large requests or a sudden barrage of requests don't just overwhelm the script, and we make sure to clean up the few items we store in memory, like the client's original URL, every X minutes. Lua is good but we still want to lighten the load on it where ever we can. We serve a real basic HTML page with the javascript for the POW challenge embedded right in. We also generate a weak but unique "server nonce" that is part of the POW calculation. This is a random string that changes for each client request, and makes it harder to try and pre-solve POW challenges. We also force the user to manually click a Submit button when the challenge is done, a very tiny form of captcha. And we rate limit submissions to one per user per minute. We do a bunch of basic sanitizing and sanity checking to filter out malicious requests and data. We force any user that accesses this as a backend to discard the original url they were coming in on and send them to the POW page, but we can pick up that original URL via a header from the frontend and use it to bounce them there once this is done. And of course we validate their POW challenge and give them a POW cookie when they pass. The frontend is configured to check for this cookie, make sure its value is a valid one, and let them on into the site if it is. Pretty simple, *generally.* Now I'll talk about the magic. The frontend proxy just needs to be able to do four things: >Route users to one or more backends based on the value of a cookie. >Pull a value from a shared fast data store to facilitate the above matching. >Implement rate limiting and security triggers that can null a stored session token. and lastly but maybe most importantly: >Generate a custom header whose value is a random string. It turned out that a major key to making this work across any kind of frontend is to have the user's session token generated by the frontend itself. Then we both pass the generated value in the X-Pow HTTP header to POWBlock and send it to the frontend's shared data store in the form of a [userIP::X-PowValue] keypair. POWBlock takes the X-Pow header value and uses it as the value of the POW cookie that gets set for that client. After some thought and research, I eventually settled on using Redis for the shared datastore just due to its low overhead and easy compatibility with Lua (meaning Nginx and Apache via Lua modules) and Varnish via the libvmod-Lua extension. But you can literally use anything, the POWBlock server does not care. In the case of Varnish, I'll post the prototype VCL code that I worked up.

Edited last time by Acidadmin on 07/08/2024 (Mon) 03:28:14.

Anonymous 07/08/2024 (Mon) 02:00:27 No. 15567

# luapow.vcl - Varnish module for handling Proof of Work (PoW) authentication

import redis;

# Define a subroutine to handle POW control
sub POW {
    # Check for POW cookie in the request
    if (req.http.Cookie) {
        # Parse cookies
        cookie.parse(req.http.Cookie);

        # Get POW cookie value
        set req.http.Cookie-POW = cookie.get("POW");

        # Fetch cached POW value from Redis based on user's IP from X-Forwarded-For
        set redis_key = "user_pow:" + req.http.X-Forwarded-For;
        if (!redis.is_error(redis.get("RedisServer", "RedisPort", redis_key))) {
            set req.http.X-POW-Cached = redis.get("RedisServer", "RedisPort", redis_key);

            # Compare POW cookie value with cached value
            if (req.http.Cookie-POW == req.http.X-POW-Cached) {
                # Valid POW cookie, allow request to proceed through the other VCLs
                return;
            }
        }
    }

    # Generate an 8-character random string for X-Pow
    set req.http.X-Pow = std.randomstr(8, "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789");

    # Cache the X-Pow value in Redis based on user's IP from X-Forwarded-For
    set redis_key = "user_pow:" + req.http.X-Forwarded-For;
    redis.set("RedisServer", "RedisPort", redis_key, req.http.X-Pow, "EX", 28800);  # Cache for 8 hours (28800 seconds)

    # Set X-Lua-Pow header to a secret key value
    set req.http.X-Lua-Pow = "magic";

    # Set Lua server as backend
    set req.backend_hint = POWBlock;

    # Preserve requested URL
    set req.http.X-Original-URL = req.url;

    # Redirect user to the PoW page
    return (pass);
}

# Define Varnish backend

[Expand Post]backend POWBlock {
    .host = "POWServer";
    .port = "POWPort";
}

# Handle incoming requests
sub vcl_recv {
    # Reset headers
    unset req.http.X-Original-URL;
    unset req.http.X-Lua-Pow;
    unset req.http.Cookie-POW;
    unset req.http.X-Lua;
    unset req.http.X-POW-Cached;

    # Call POW subroutine to handle PoW authentication
    call POW;
}

Anonymous 07/08/2024 (Mon) 02:24:39 No. 15568

This is called a VCL Module. They plug into the main VCL code that drives the caching engine via a leading include"" statement. 8chan's network frontends use about 24 of these that I've written over the years, so this is just one more. But you can see some of what we need to handle: First we scrub all custom header values from the client's side, to make sure he can neither spoof nor see any of the headers or values that we're using. Then we parse the user's Cookies looking for POW, and if it exists we ping the Redis storage for the user's IP address to see if there's a stored value. If both exist, and match, the user is let into the site. If they don't match, we run the rest of the POW subroutine, which: >Generates an X-Pow header with a random 8 character session token >Sends the token to the Redis store keyed to the user's IP address >Sets the magic header >Switches the backend from the 8chan main CDN gateway to the POWBlock server >Saves the URL the user was coming in on and sends it in a header >Connects the user to POWBlock with all necessary information supplied What isn't shown: Forward rate limits, DoS throttling, URL locking, request sanitizing, header normalizing, and a bunch of other shit the main VCL code will do for security before it even gets to this spot. I'm sure it'll need to be tweaked for an actual deploy, but I'll cross that bridge when I get to it. And I need to write a nullification module so that if a user gets past this and fucks around, it will null the value of their key in the shared store and force them back to the POW or even ban them entirely. >Why bother with all this shit and why make a Lua server instead of just putting it on the site, maybe with the splash disclaimer? Because it can take upwards of 60% of the load off the frontends in the case of a major attack. POWBlock can run on its own shitbox server (and even have DDoS protection there) and if the site is getting hammered like hell, then its POWBlock taking most of the assraping instead of the servers we actually care about. And these things are so cheap and easy to set up (install Lua, install 3 Lua modules, stick the script in systemd, fucking done) that the servers it would run on can be wholly disposable. You could have 50 of em, each running a forking instance, and divide a big attack up between them. Its also expandable. Right now its just going to be a POW, but I can add simple user agent checks, non-invasive fingerprinting (click speed, mouse speed), dummy fields to catch the stupidest bots, and maybe a custom captcha similar to what Stephen uses in Lynxchan. It's not going to replace Cuckflare's massive CDN network and Terabit pipes, but I think this could become a super useful tool for all of us little guys out here. If anyone cares about this thread I'll post updates as I work on deploying the actual system and doing a real feasibility test. I have some free time over the next couple of weeks and have plenty of site work to do, but I'm gonna try to squeeze this in too. Auf wiedersehen!

Anonymous 07/13/2024 (Sat) 03:15:42 No. 15611

>>15566 >We also generate a weak but unique "server nonce" that is part of the POW calculation No it isn't you nigger. Also if you do this correctly the nonce controls the size of the table an attacker needs to precalculate challenges and 2 bytes is just 65k unique solutions you double nigger. Additionally I imagine you should also clean server_nonces in case users never complete the pow.

Quick Reply


Sage Bypass Check