IrisCTF 2025: ?x8 [rev] writeup

Writeup author: fsharp
(The heading above can't show more than 3 question marks so this will have to do…)

Introduction

I played IrisCTF 2025 with DeadSec and checked out a variety of challenges. This writeup is on the reverse engineering challenge I solved during the CTF, which is ???????? and has only 7 solves. We were the 6th team to solve it.

Problem description

Author: nope

where is this challenge? what is this challenge?
(must use webgl version to solve, you may need to shift+refresh to clear your cache to see the updated changes)

(Link provided: https://2025.irisc.tf/wgl)

Finding the challenge

To add some context, this year there were 2 versions of the IrisCTF website that players can navigate to: a WebGL version containing really cool graphics and animations, and a regular version that was also present in the previous 2 iterations of IrisCTF.

The description hints that the challenge is only solvable in the WebGL version of the website, and all we get is its link. Having participated in the first IrisCTF, I know that there was a challenge based on reversing a scoreboard Easter egg hidden in the CTF website, and I believed this one to be similar.

Since the scoreboard Easter egg was hidden in a JavaScript (JS) file, I logged into the WebGL version of the website, cleared the browser cache, and scoured its web source. The following stood out to me immediately:

JS file analysis

The first file (71a983c3dac9500a8a94a3f60d56302d.js) looks like this:

async function sha1Hex(input) {
  const encoder = new TextEncoder();
  const data = encoder.encode(input);
  const hashBuffer = await crypto.subtle.digest('SHA-1', data);
  const hashArray = Array.from(new Uint8Array(hashBuffer));
  return hashArray.map(b => b.toString(16).padStart(2, '0')).join('').substring(0, 32);
}
async function instantiate(module, imports = {}) {
  // snip...
  const { exports } = await WebAssembly.instantiate(module, adaptedImports);
  const memory = exports.memory || imports.env.memory;
  const adaptedExports = Object.setPrototypeOf({
    reality() {
      // snip...
      return __liftString(exports.reality() >>> 0);
    },
    recognition(jsSecretData) {
      // snip...
      exports.recognition(jsSecretData);
    },
    consciousness(catIdx) {
      // snip...
      const result = __liftString(exports.consciousness(catIdx) >>> 0);
      // snip...
    },
    awareness(str) {
      // snip...
      return exports.awareness(str) != 0;
    },
  }, exports);
  // snip...
  return adaptedExports;
}
async function loadWasm() {
  window['mind'] = await instantiate(
    await WebAssembly.compileStreaming(
        fetch(`/static/bg/${await sha1Hex('mind')}.wasm`)
    ), {}
  );
  return window['mind'];
}

The loadWasm() function fetches a WebAssembly (WASM) file located at /static/bg/${await sha1Hex('mind')}.wasm, loads it, and sets window['mind'] to a dictionary containing functions exported from that file. await sha1Hex('mind') calculates the SHA-1 hash of the byte-string mind, turns it into hexadecimal characters, and returns the first 32 characters, which is d42c0112962be8a77d278ebe747514e5.

The second file (f.min.js) is a minified JS file. The string mind is present in several functions:

function bb(a) {
    a.preventDefault();
    window.mind.awareness(a.target.elements.flag.value) ?
        D[F][1].z() :
        a.target.elements.flag.value = "incorrect";
    return !1
}

function cb(a, b, c) {
    // snip...
    let h = window.mind ? window.mind.perception() : -1;
    if (e)
        // snip...
    else {
        e = a.description;
        14 === a.id && 4 < h && (e = `irisctf{${window.mind.reality()}}`);
        // snip...
        1E7 < a.id && 5 > h ? w.onsubmit = bb : (w.method = "post", w.action = "api");
        // snip...
    }
    // snip...
    14 === a.id && 0 <= h && 4 >= h || k.appendChild(q)
}

async function Mb() {
    var a = (new TextEncoder).encode("feeling");
    a = await crypto.subtle.digest("SHA-1", a);
    return Array.from(new Uint8Array(a)).map(
            b => b.toString(16).padStart(2, "0")
        ).join("").substring(0, 32)
}

async function Nb(a, b, c) {
    if (document.cookie.includes("chal")) {
        window.mind || await loadWasm();
        if (-1 === window.mind.perception()) {
            var d = await fetch(`/static/bg/${await Mb()}.bmp`);
            d.ok &&
            (d = await d.arrayBuffer(), d = new Uint8Array(d),
             window.mind.recognition(d))
        }
        - 1 !== window.mind.perception() &&
        (c = window.mind.consciousness(c), null != c && cb(c, a, b))
    }
}

Nb(a, b, c) performs the following if the cookie data for the CTF website contains the string chal:

1. If window.mind is empty, loadWasm() is called. This is for window.mind to contain references to exported WASM functions.
2. If window.mind.perception() returns -1, a BMP image located at /static/bg/${await Mb()}.bmp is retrieved, and a buffer containing the image's bytes is passed into window.mind.recognition().
3. Now, if window.mind.perception() doesn't return -1, window.mind.consciousness(c) is called and its result is assigned to c.
4. If c isn't null, cb(c, a, b) is called.

await Mb() is essentially await sha1Hex('feeling'), which returns b3b6db05f1888b1603560e78a9564d9c. bb() seems to be a function that checks whether an entered flag is correct by passing it into window.mind.awareness(), and cb() seems to override the flag submission mechanism of a challenge based on specific conditions being met.

At this point, I was certain this challenge involves the JS, WASM and BMP files that I've written about.

WASM file: initial analysis

I downloaded both /static/bg/d42c0112962be8a77d278ebe747514e5.wasm and /static/bg/b3b6db05f1888b1603560e78a9564d9c.bmp, then loaded the WASM file into Ghidra with the Ghidra WASM plugin to decompile it.

The BMP image looks like this:

Huh. Interesting.

perception() and reality() return the global variables global_13 and global_14 respectively:

undefined4 export::perception(void)
{
    return global_13;
}

undefined4 export::reality(void)
{
    return global_14;
}

However, the exported reality() function from the first JS file returns __liftString(exports.reality() >>> 0). Perhaps global_14 is a pointer to a string? The second JS file contains:

e = `irisctf{${window.mind.reality()}}`

…so window.mind.reality() probably returns a string that is used for the flag of this challenge. It is only called when 14 === a.id and window.mind.perception() returns a number larger than 4. Moreover, if 1E7 < a.id and window.mind.perception() returns a number less than 5, the flag submission mechanism gets overriden with the bb() function. Checking the HTML source of the website, the challenge with ID 14 is ????????, so maybe its description will contain the flag at one point.

consciousness() takes an integer argument and does something with it:

undefined4 export::consciousness(int param1)
{
    // snip...

    if ( /* snip... */ ) {
        // snip...
        if (*(int *)(global_10 + 8) == 0) {
            iVar2 = 0;
        }
        else {
            // snip...
            iVar2 = *(int *)(global_12 + 8);
        }
        if ((((iVar2 == 0) ||
              (global_13 == 0 && param1 != 3)) || (global_13 == 1 && param1 != 5)) ||
              (((global_13 == 2 && param1 != 6 || (global_13 == 3 && param1 != 1)) ||
               ((global_13 == 4 && param1 != 8 || (4 < global_13)))))) {
            return 0x660;
        }
        if ( /* snip... */ ) {
            // many function calls, snip...
            if ( /* snip... */ ) {
                // snip...
                uVar1 = unnamed_function_26(uRam0000090c >> 2);
                return uVar1;
            }
        }
    }

    // snip...
}

2 kinds of values (0x660 and uVar1) are returned depending on whether specific conditions are satisfied. For uVar1 to be returned, the conditions to satisfy are:

1. iVar2 is not equal to 0.
2. When global_13 is 0, 1, 2, 3 or 4, the argument must be 3, 5, 6, 1 and 8 respectively.
3. global_13 must not be greater than 4.

Where do 3, 5, 6, 1 and 8 come from? Going back to d.min.js, the argument passed into window.mind.consciousness() can be traced back to its origin:

async function Nb(a, b, c) {
    // snip...
    (c = window.mind.consciousness(c), null != c && cb(c, a, b))
    // snip...
}

function Y(a, b) {
    // snip...
    await Nb(d, 0 === c.length, b);
    // snip...
}

(async function() {
    // snip...
    D.push(["welcome", Y("Welcome", 0), 0, 0, "Welcome"]),
    D.push(["binexp", Y("Binary Exploitation", 1), 0, 0, "Binary Exploitation"]),
    // snip...
})()

It turns out the argument is a number representing the challenge category currently selected in the website! 3, 5, 6, 1 and 8 correspond to forensics, networks, open-source intelligence, binary exploitation, and reverse engineering respectively. This matches the implicit meaning of the catIdx argument ('category index') for consciousness() in the first JS file analyzed.

What about recognition()? The first JS file names the argument as jsSecretData, so maybe the BMP image contains hidden data?

Ghidra decompiles the function as follows:

void export::recognition(uint param1)
{
    // snip...

    if ( /* snip... */ ) {
        // snip...
        param1_00 = (undefined4 *)unnamed_function_22(*(undefined4 *)(param1 + 8));
        // snip...
        global_10 = param1_00;
        if ( /* snip... */ ) {
            // snip...
            iVar1 = unnamed_function_24(param1);
            // snip...
            memory_copy(0,0,iVar1,*(undefined4 *)(param1 + 4),param1_00[1]);
            // snip...
            global_11 = 1;
            // snip...
            global_12 = unnamed_function_23(*global_10);
            global_13 = 0;
            return;
        }
    }

    // snip...
}

There's memory copying involved, and the global variables global_10 to global_13 are set. From this, I guess that either global_10 or global_12 stores a pointer to a byte buffer containing the BMP image's bytes.

To validate this, I use the Chrome browser's DevTools functionality to inspect the WASM's memory. The DevTools documentation contains good tutorials on doing so, which I'll link here and here. In a nutshell:

1. Install and enable the C/C++ DevTools Support (DWARF) Chrome extension.
2. Open DevTools (e.g. by pressing F12), head to the Sources tab, and click on the WASM file. If there is no WASM file, load it by creating a cookie named chal and refreshing the page.
3. Put a breakpoint on a WASM opcode by clicking on its address. In this case, I choose the return opcode used by recognition(), which is located at 0x208d.
4. Reload the page and wait until the breakpoint is hit. There should now be a 'Paused in debugger' box at the top of the webpage.
5. Underneath Scope > Module, the globals dropdown shows the values of global variables, and the memories dropdown has a button that opens the memory inspector. Click on that button.
6. Modify the memory address in the memory inspector to the value of either $global10 or $global12.

For me, $global10 = 0x598e0 and $global12 = 0xa99a0.

$global10 is quite close to the start of the BMP image bytes:

The BMP image is located at 0x59900. What does $global12 point to?

Hmm. Doesn't seem like the variables are useful… but wait! Looking at them again, both $global10 and $global12 point to 00 99 05 00, and interpreting those bytes as a 4-byte little-endian integer gives 0x59900, which is where the BMP image is! So, both global variables are pointers to a pointer to the BMP image.

Now let's take a look at awareness(). My previous analysis showed that it takes in an entered flag.

undefined4 export::awareness(undefined4 param1)
{
    undefined4 uVar1;
    // snip...

    if ( /* snip... */ ) {
        // snip...
        uVar1 = unnamed_function_31(param1);
        return uVar1;
    }

    // snip...
}

The flag is passed into unnamed_function_31(), and awareness() returns whatever is returned by that function. Also, from the first JS file, window.mind.awareness() returns exports.awareness(str) != 0. It's likely unnamed_function_31() returns a boolean depending on whether the flag's correct.

So what does that function decompile into?

undefined4 unnamed_function_31(uint param1)
{
    // snip...
    uVar7 = 0;
    // snip...

    param5 = unnamed_function_28(global_12,10);
    // snip...
    param3 = unnamed_function_28(global_12,0x12);
    // snip...
    param4 = unnamed_function_28(global_12,0x16);
    uVar3 = unnamed_function_29(0,global_13,param3,param4,param5);
    param2 = uVar3 >> 8 & 0xff;
    if (*(uint *)(param1 - 4) >> 1 == 0xc) {
        // snip...
        uVar3 = uVar3 & 0xff;
        iVar4 = unnamed_function_30(param1,param2,uVar3,param3,param4,param5);
        for (; iVar1 = global_14, uVar7 < 6; uVar7 = uVar7 + 1) {
            for (uVar2 = 0; uVar2 < 6; uVar2 = uVar2 + 1) {
                uVar5 = unnamed_function_29(uVar2 + param2,uVar7 + uVar3,
                                            param3,param4,param5);
                uVar6 = unnamed_function_29(param2 + 6 + uVar2,uVar7 + uVar3,
                                            param3,param4,param5);
                uVar8 = 1;
                if ((uVar5 & 0x100) == 0) {
                    uVar8 = uVar6 & 0x100;
                }
                iVar1 = 0;
                if (uVar8 == 0) {
                    iVar1 = iVar4;
                }
                iVar4 = iVar1;
            }
        }
        if (iVar4 != 0) {
            global_13 = global_13 + 1;
            // snip...
            if ( /* snip... */ ) {
                // snip...
                return 1;
            }
            // snip...
        }
        // snip...
        unnamed_function_30(param1,param2,uVar3,param3,param4,param5);
    }
    return 0;
}

From this, I noticed the following:

1. Some numbers are calculated from unnamed_function_28(global_12, <something>). They are passed into unnamed_function_29() and unnamed_function_30().
2. For unnamed_function_31() to return 1, *(uint *)(param1 - 4) >> 1 == 0xc and iVar4 != 0. global13 is also incremented by 1 if these conditions are satisfied.
3. The value of iVar4 is initialized as unnamed_function_30(param1,param2,uVar3,param3,param4,param5) and is modified in a for loop. The only way for iVar4 to not become 0 is if its initialized value is not 0, (uVar5 & 0x100) == 0, and (uVar6 & 0x100) == 0.

To summarize what I've found so far:

1. window.mind.perception() returns global_13, which should range from -1 to 5. When it's -1, the BMP image isn't loaded yet. When it's 0, the BMP image is loaded. When it's 5, the flag should be displayed in the description of the ???????? challenge. When it's between 0 and 4 (inclusive), the flag submission mechanism gets overriden for certain challenges.
2. window.mind.reality() should return a string used for the flag of this challenge.
3. window.mind.consciousness() accepts a challenge category index and returns something that depends on both the inputted index and global_13.
4. window.mind.recognition() accepts a byte buffer to the BMP image's bytes, and sets both global_10 and global_12 to values that point to a pointer to the start of those bytes.
5. window.mind.awareness() is called from the modified flag submission mechanism and accepts an entered flag. It returns a boolean that depends on both global_13 and the inputted flag. If it returns true, global_13 is incremented by 1.
6. The BMP image likely contains hidden data.

From all of this information, there seems to be 5 stages that need to be passed, with each stage requiring a specific flag.

The goal now is to focus on understanding what unnamed_function_31() does to solve this challenge.

Analyzing unnamed_function_31()

There must be a place to enter a flag for each stage. From my previous analysis, I noticed that 5 challenge categories are referenced, and the first one is forensics.

After loading the WASM and heading to that category, I see this at the bottom of the page:

That's probably it! At this point, I think I have to solve a special challenge from the forensics, networks, open-source intelligence, binary exploitation, and reverse engineering categories in that order.

At the start of unnamed_function_31(), there are several unnamed_function_28(global_12, <something>) calls. What do they return? If I set breakpoints at 0x1da1, 0x1db2 and 0x1dc3, which are opcodes right after those calls, then entered a flag, I can look at the stack and see that they return 0x8a, 0x140 and 0x100 respectively. The decompilation of the function is:

undefined4 unnamed_function_28(int param1,uint param2)
{
    // snip...

    if ( /* snip... */ ) {
        return *(undefined4 *)(param2 + *(int *)(param1 + 4));
    }

    // snip...
}

Given that param1 is global_12 and it's known that it's a pointer to a pointer to the BMP image bytes, param2 might be a byte offset from the start of the image. Indeed, checking the BMP image in a hex editor confirms this:

So, unnamed_function_28() returns a 4-byte little-endian integer located at a offset into a byte buffer.

Looking up the BMP image format, 0x8a is the byte offset where the image's pixel data is located, while 0x140 and 0x100 are the image's width and height respectively.

unnamed_function_29() seems to do something similar:

undefined4 unnamed_function_29(int param1,int param2,int param3,int param4,int param5)
{
    undefined4 uVar1;
    // snip...
    
    if (/* snip... */) {
        // snip...
        uVar1 = unnamed_function_28(global_12,
                                    param5 +
                                    ((param4 + -1) - param2) * param3 * 4 +
                                    param1 * 4);
        return uVar1;
    }

    // snip...
}

Checking all calls to unnamed_function_29(), it can be found that param3, param4 and param5 always equal to the image width, image height, and byte offset to pixel data respectively. As pixel data is stored bottom-up, this function is retrieving a 4-byte little-endian integer from the pixel data, with param1 and param2 being the X and Y coordinates of the pixel to read.

The first usage of this function is unnamed_function_29(0,global_13,param3,param4,param5), so it's retrieving a pixel at the leftmost column of the image. Since global_13 ranges from 0 to 4, it will retrieve the 1st to 5th pixels of that column for each stage. Going back to unnamed_function_31()'s decompilation, it can be seen that not all 4 bytes of the pixel read are used:

// snip...
uVar7 = 0;
// snip...

uVar3 = unnamed_function_29(0,global_13,param3,param4,param5);
param2 = uVar3 >> 8 & 0xff;
if (*(uint *)(param1 - 4) >> 1 == 0xc) {
    // snip...
    uVar3 = uVar3 & 0xff;
    iVar4 = unnamed_function_30(param1,param2,uVar3,param3,param4,param5);
    for (; iVar1 = global_14, uVar7 < 6; uVar7 = uVar7 + 1) {
        for (uVar2 = 0; uVar2 < 6; uVar2 = uVar2 + 1) {
            uVar5 = unnamed_function_29(uVar2 + param2,uVar7 + uVar3,
                                        param3,param4,param5);
            uVar6 = unnamed_function_29(param2 + 6 + uVar2,uVar7 + uVar3,
                                        param3,param4,param5);
            // snip...
        }
    }
}

param2 is the second least-significant pixel byte (green channel), while uVar3 becomes the least-significant pixel byte (blue channel). The subsequent unnamed_function_29() calls show that they are used as X and Y coordinate offsets into the pixel data!

From my previous analysis, 2 conditions for unnamed_function_31() to return true are (uVar5 & 0x100) == 0 and (uVar6 & 0x100) == 0. It is now clear that this means the least-significant bit of the green channel of 2 * 6 * 6 = 72 specific pixels must be 0. Another condition to satisfy is *(uint *)(param1 - 4) >> 1 == 0xc. Breakpointing at 0x1df0, which is where this comparison is being done, then entering a random flag, it can be seen that this condition means the inputted flag must be exactly 12 characters long.

So all that's left to analyze now is unnamed_function_30(). What does it do?

undefined4
unnamed_function_30(uint param1,int param2,undefined4 param3,undefined4 param4,
                    undefined4 param5, undefined4 param6)
{
    // snip...

    uVar10 = 0;
    if ( /* snip... */ ) {
        // snip...
        while( true ) {
            if (0xb < (int)uVar10) { return 1; }
            // snip...

            if ( /* snip... */ ) {
                plVar2 = (longlong *)export::__new(2,2);
                // snip...
                *(ushort *)plVar2 = *(ushort *)(param1 + uVar10 * 2);
            }
            else { /* snip... */ }
            // snip...

            uVar9 = 0;
            // snip...
            uVar3 = *(uint *)((int)plVar2 + -4) >> 1;
            if (uVar3 == 0) { /* snip... */ }
            else {
                // snip...
                if ( /* snip... */ ) {
code_r0x80001b49:
                    if ((int)uVar9 <= (int)((uRam0000108c >> 1) - uVar3)) {
                        plVar6 = (longlong *)
                                 (u_abcdefghijklmnopqrstuvwxyz012345_ram_00001090
                                  + uVar9);
                        // snip...
                        uVar5 = uVar3;
                        plVar1 = plVar2;
                        if (uVar4 == 0) {
                            do {
                                if (*plVar6 != *plVar1) break;
                                plVar6 = plVar6 + 1;
                                plVar1 = plVar1 + 1;
                                uVar5 = uVar5 - 4;
                            } while (3 < uVar5);
                        }
                        do {
                            if (uVar5 == 0) goto code_r0x80001bf0;
                            iVar7 = (uint)(ushort)*(wchar_t *)plVar6 -
                                             (uint)*(ushort *)plVar1;
                            if ((uint)*(ushort *)plVar1 !=
                                (uint)(ushort)*(wchar_t *)plVar6)
                                    goto code_r0x80001bf5;
                            plVar6 = (longlong *)((int)plVar6 + 2);
                            uVar5 = uVar5 - 1;
                            plVar1 = (longlong *)((int)plVar1 + 2);
                        } while( true );
                    }
                    uVar9 = 0xffffffff;
                    goto code_r0x80001c1b;
                }
                uVar9 = 0xffffffff;
            }
code_r0x80001c1b:
            if (uVar9 == 0xffffffff) {
                return 0;
            }
            iVar11 = (int)FLOOR((SQRT((double)(ulonglong)uVar9 * 8.0 + 1.0)
                                 + -1.0) * 0.5);
            iVar7 = uVar9 - ((uint)(iVar11 * (iVar11 + 1)) >> 1);
            iVar8 = (int)uVar10 % 2;
            param4_00 = (int)uVar10 / 2;
            uVar9 = iVar11 - iVar7;
            if ((uVar9 & 1) != 0) {
                unnamed_function_16(param2,param3,iVar8 * 3,
                                    param4_00,param4,param5,param6);
            }
            if ((uVar9 & 2) != 0) {
                unnamed_function_16(param2,param3,iVar8 * 3 + 1,
                                    param4_00,param4,param5,param6);
            }
            if ((uVar9 & 4) != 0) {
                unnamed_function_16(param2,param3,iVar8 * 3 + 2,
                                    param4_00,param4,param5,param6);
            }
            uVar9 = (uVar9 | iVar7 * 0x100) >> 8;
            if ((uVar9 & 1) != 0) {
                unnamed_function_16(param2 + 6,param3,iVar8 * 3,
                                    param4_00,param4,param5,param6);
            }
            if ((uVar9 & 2) != 0) {
                unnamed_function_16(param2 + 6,param3,iVar8 * 3 + 1,
                                    param4_00,param4,param5,param6);
            }
            if ((uVar9 & 4) != 0) {
                unnamed_function_16(param2 + 6,param3,iVar8 * 3 + 2,
                                    param4_00,param4,param5,param6);
            }
            uVar10 = uVar10 + 1;
        }
    }
    // snip...
code_r0x80001bf0:
    iVar7 = 0;
code_r0x80001bf5:
    if (iVar7 == 0) goto code_r0x80001c1b;
    uVar9 = uVar9 + 1;
    goto code_r0x80001b49;
}

There's a lot going on here, so I'll go through this slowly.

For unnamed_function_31() to return true, this function must return 1, and line 11 shows that uVar10 must be greater than 11. It is initialized to 0 in line 7 and gets incremented by 1 in line 97, so its value ranges from 0 to 12. In addition, line 17 assigns an unsigned short from param1 (the entered flag) to plVar2. So, uVar10 is an index for the flag string, and plVar2 is one of the flag characters.

u_abcdefghijklmnopqrstuvwxyz012345_ram_00001090 points to a Unicode string abcdefghijklmnopqrstuvwxyz0123456789. Every character in that string is 2 bytes long, which matches the width of an unsigned short. Lines 22 to 65 and 101 to 106 look intimidating, but looking closer, a few observations can be made:

1. In line 36, plVar1 is assigned one of the entered flag characters.
2. In lines 31 to 33, plVar6 is assigned something from the Unicode string. uVar9 is probably a character index for that string as line 105 shows it being incremented by 1, so plVar6 is one of the characters from that string.
3. In line 39, plVar1 and plVar6 are checked to see if they're equal.

It is very likely that these lines are checking if the entered flag character is inside the Unicode string. If it isn't, the function returns 0; otherwise, lines 66 to 97 are executed.

unnamed_function_16() can be called at most 6 times per flag character. Some numbers are calculated from uVar9 (the index of the flag character in the Unicode string) and uVar10 (the index of the character in the entered flag), which are used to determine which unnamed_function_16() calls to make and are passed into those calls as arguments. That function's decompilation is:

void unnamed_function_16(undefined4 param1,undefined4 param2,int param3,int param4,
                         undefined4 param5,undefined4 param6,undefined4 param7)
{
    unnamed_function_15(param1,param2,param3,param4,param5,param6,param7);
    if (param3 != 0) {
        unnamed_function_15(param1,param2,param3 + -1,param4,param5,param6,param7);
    }
    if (param3 != 5) {
        unnamed_function_15(param1,param2,param3 + 1,param4,param5,param6,param7);
    }
    if (param4 != 0) {
        unnamed_function_15(param1,param2,param3,param4 + -1,param5,param6,param7);
    }
    if (param4 != 5) {
        unnamed_function_15(param1,param2,param3,param4 + 1,param5,param6,param7);
    }
    return;
}

unnamed_function_15()'s decompilation looks like:

void unnamed_function_15(int param1,int param2,int param3,int param4,int param5,
                         int param6,int param7)
{
    // snip...
    
    uVar1 = unnamed_function_29(param1 + param3,param2 + param4,param5,param6,param7);
    if ( /* snip... */ ) {
        // snip...
        uVar2 = param7 + ((param6 + -1) - (param2 + param4)) * param5 * 4 +
                (param1 + param3) * 4;
        // snip...
        *(uint *)(uVar2 + *(int *)(global_12 + 4)) = uVar1 ^ 0x100;
        return;
    }

    // snip...
}

This function XORs a pixel value by 0x100. Since the second least-significant byte of a pixel is its green channel, this function is toggling the least significant bit of a pixel's green channel.

The pixel value being XOR'd is located at this byte offset in the BMP image:

param7 + ((param6 + -1) - (param2 + param4)) * param5 * 4 + (param1 + param3) * 4)

Tracing the values of these parameters, it's found that param1 and param2 are always the X and Y offsets obtained from the colour channels of the first pixel column, while param5 to param7 are always the image width, image height, and byte offset to pixel data respectively. Thus, param3 and param4 are offsets additional to the X and Y offsets from colour channels.

unnamed_function_16() calls unnamed_function_15() depending on the values of param3 and param4, and both range from 0 to 5 (inclusive). Given that both parameters are passed into unnamed_function_29() as additional X and Y offsets, it is now quite clear what kind of data is hidden inside the BMP image, and what problem this CTF challenge is actually about.

Bomb explosions a la Bomberman

Each of the 5 stages involves the least significant bits of the green channel from 72 specific pixels in the BMP image. The goal is to get all of these bits set to 0, and the inputted flag controls which bits get toggled. The pixels from each stage are arranged as 2 grids of 6 x 6 pixels.

unnamed_function_15() toggles the bit for 1 pixel. unnamed_function_16() toggles bits in a 'plus' shape (+). unnamed_function_30() controls where the 'bombs' are placed in the 2 grids, with the bomb explosions shaped like a 'plus'.

The way I viewed this is to imagine if a bomb from Bomberman were placed on the grid, the toggleable bits were flippable tiles, the grid's surrounded by indestructible walls, and the bomb's explosion flipped those tiles. Like the least powerful bomb in the game, the 'plus' explosion is 3 x 3 tiles and it only affects the tiles it covers, with the walls preventing the explosion from going outside of the grid.

To illustrate this, let's say 3 bombs were placed in a grid as follows:

The explosions will look like this:

5 + 4 + 3 = 12 tiles in that grid will be flipped.

Solving the challenge

All that remains at this point is for me to write a solver that does these:

1. Extract the X and Y offsets of the 2 grids for each stage by reading the green and blue channels of the first 5 pixels in the leftmost pixel column.
2. Extract the 2 grids for each stage using those offsets.
3. For each allowed character in the 12 positions of the flag, place bombs in both grids the same way as unnamed_function_30() and record all possible explosions for each tile.
4. Solve for the set of explosions that both results in all bits being set to 0 and is possible from the range of flags that can be entered.

This is a massively cleaned-up version of my solver:

from math import floor, sqrt
from PIL import Image
from z3 import *

total_num_stages = 5
flag_len = 12
charset = "abcdefghijklmnopqrstuvwxyz0123456789"
charset_len = len(charset)

bmp = Image.open("b3b6db05f1888b1603560e78a9564d9c.bmp", 'r')
offset_pairs = [bmp.getpixel((0, y))[1:3] for y in range(total_num_stages)]

nums0 = [floor((sqrt(c * 8 + 1) - 1) * 0.5) for c in range(charset_len)]
nums1 = [c - ((nums0[c] * (nums0[c] + 1)) >> 1) for c in range(len(nums0))]
nums2 = [nums0[i] - nums1[i] for i in range(len(nums0))]
nums3 = [(nums2[i] | (nums1[i] * 0x100)) >> 8 for i in range(len(nums0))]
num_pairs = [(num2, num3) for (num2, num3) in zip(nums2, nums3)]

for stage_num in range(total_num_stages):
    print(f"Solving for stage #{stage_num + 1}'s flag...")
    
    (x_offset, y_offset) = offset_pairs[stage_num]
    grid0 = [[bmp.getpixel((x_offset + x, y_offset + y))[1] & 1 for x in range(6)] for y in range(6)]
    grid1 = [[bmp.getpixel((x_offset + x + 6, y_offset + y))[1] & 1 for x in range(6)] for y in range(6)]
    grids = [grid0, grid1]
    
    # Every 36 variables in this array represents all possible characters for each flag character
    # Their values must be either 0 or 1, and since only 1 character can be chosen for each of the 12 positions,
    # their sum must be 1
    input_chars = [BitVec(f"f{i}", 8) for i in range(flag_len * charset_len)]
    solv = Solver()
    
    for i in range(len(input_chars)):
        solv.add(Or(input_chars[i] == 0, input_chars[i] == 1))
    
    for i in range(0, len(input_chars), charset_len):
        solv.add(sum(input_chars[i : i + charset_len]) == 1)
    
    # Record all possible explosions per tile
    for flag_char_position in range(flag_len):
        num4 = flag_char_position % 2
        num5 = flag_char_position // 2
        for i in range(charset_len):
            val = input_chars[charset_len * flag_char_position + i]
            for j in range(len(grids)):
                num6 = num_pairs[i][j]
                for power in range(3):
                    if (num6 & pow(2, power)) == 0:
                        continue
                    x, y = num4 * 3 + power, num5
                    grids[j][y][x] ^= val
                    if x != 0:
                        grids[j][y][x - 1] ^= val
                    if x != 5:
                        grids[j][y][x + 1] ^= val
                    if y != 0:
                        grids[j][y - 1][x] ^= val
                    if y != 5:
                        grids[j][y + 1][x] ^= val
    
    # Goal: set all bits to 0
    for i in range(len(grids)):
        for y in range(6):
            for x in range(6):
                grids[i][y][x] = simplify(grids[i][y][x])
                solv.add(grids[i][y][x] == 0)
    
    if solv.check() != sat:
        print("Failed...\n")
    else:
        m = solv.model()
        ans = ""
        for flag_char_position in range(flag_len):
            for i in range(len(charset)):
                val = input_chars[charset_len * flag_char_position + i]
                val2 = m.evaluate(val).as_long()
                if val2 == 1:
                    ans += charset[i]
                    break
        print(f"{ans}\n")

bmp.close()

Running this script, I got the following output:

Solving for stage #1's flag...
asparagus050

Solving for stage #2's flag...
squar3wave44

Solving for stage #3's flag...
tr4incars483

Solving for stage #4's flag...
s1lentbird01

Solving for stage #5's flag...
d3cayedb0nes

Let's go enter them on the website! After doing that, reloading the challenges in the reverse engineering category will show the flag: irisctf{asparagus050squar3wave44tr4incars483s1lentbird01d3cayedb0nes}

Bonus

After the CTF ended, I learnt a few more things about this challenge.

1. From top to bottom, the colours of the strips in the BMP image correspond to the 5 challenge categories you'll find the special challenge in. Pretty cool!
2. The game inside the BMP image is called Lights Out. No, it's not Bomberman-esque explosions like I thought. Since when did Bomberman games involve flipping tiles with bombs anyway?
3. The challenge author, nope, revealed that some numbers calculated in unnamed_function_30 (i.e. nums0 to nums3 in my solver) comes from inverting the Cantor pairing function. This involves turning a number z into a unique pair of numbers x, y. In fact, the x and y from that section of the Wikipedia article I linked matches nums2 and nums1 respectively. Moreover, nums3 is the same as nums1 due to the bitshift. So, the x and y obtained from inverting the Cantor pairing function for integers from 0 to 35 are the same as nums2 and nums3! However, neither x nor y are used as X or Y coordinate offsets into pixel data. They are instead used to determine where the bombs are placed in the grids.

There are some more things I have (partially) figured out while reversing the WASM but haven't written much about as understanding them isn't necessary. Two of those things are the following:

1. Why are there two calls to unnamed_function_30() in unnamed_function_31()? If the entered flag is correct, that function will be called just once, which sets the least significant bits of the green channels of the 72 pixels involved to 0. However, if the flag is wrong, that function will be called twice to reset those bits back to what they were originally. This ensures that the correct flag for each stage remains the same and doesn't change due to previous incorrect attempts.
2. Where do the descriptions of the 5 special challenges come from, and what does export::consciousness() return? The descriptions are made by the many function calls I didn't include in the decompilation of that function. To sum it up, there are 8 calls to unnamed_function_25() that create the question marks in the challenge title and description, the unnamed_function_12() calls seem to be placing the pointers to those question mark strings at a specific index of an array of string pointers, and the unnamed_function_26() call seems to create a joined string from that pointer array. Calling window.mind.consciousness() in the JS console with the correct challenge category index will return a JSON string for the special challenge that will appear for that category.

Lastly, I just wanted to know where exactly the 6 x 6 pixel grids from all 5 stages are in the BMP image. I'll write a script to turn those pixels red:

from PIL import Image

total_num_stages = 5

bmp = Image.open("b3b6db05f1888b1603560e78a9564d9c.bmp", 'r')
bmp2 = bmp.copy()
offset_pairs = [bmp.getpixel((0, y))[1:3] for y in range(total_num_stages)]

for stage_num in range(total_num_stages):
    (x_offset, y_offset) = offset_pairs[stage_num]
    for y in range(6):
        for x in range(6):
            bmp2.putpixel((x_offset + x, y_offset + y), (255, 0, 0, 255))
            bmp2.putpixel((x_offset + x + 6, y_offset + y), (255, 0, 0, 255))

bmp.close()

bmp2.save("highlighted_pixel_grids.bmp")
bmp2.close()

So what does that image look like?

Ahh, so they're at the start of each colour strip. But why is there a column of red pixels to the left of the image? It doesn't seem like it's used to mask the X and Y offsets hidden in the first 5 pixels of that column… I might never know.