PicoCTF asm3 challenge: Master the Art of Reverse Engineering

Welcome back to the exciting world of reverse engineering!
In this article, we will tackle the picoCTF asm3 challenge (here is our article on how to register there), a slightly more complex task than the previous ones (asm1, asm2). By analyzing this assembly code, we will uncover its secrets and discover the flag.

To start, head over to the picoCTF website and log in to your account. If you don’t have an account yet, follow the registration steps in our introductory article.
Once you’re logged in, navigate to the Practice section, select the Reverse Engineering category, and look for the asm3 challenge. Let’s begin our journey to master this exciting challenge together!

Registers Overview

Before starting with the real challenge, let’s look at something you may find useful about registers.

In a 32-bit processor, there are 8 general-purpose registers: EAX, EBX, ECX, EDX, ESI, EDI, EBP, and ESP. Each of these 32-bit registers can be further divided into smaller components for working with smaller data sizes.

Let’s consider the EAX register as an example. EAX can be divided into three components:

1. AX (16 bits): AX is the lower 16 bits of the EAX register. It allows for operations on 16-bit data.
2. AH (8 bits): AH represents the upper 8 bits of the AX register, which are bits 8 to 15 of the EAX register. It’s used for 8-bit operations on the upper half of AX.
3. AL (8 bits): AL stands for the lower 8 bits of the AX register, which are bits 0 to 7 of the EAX register. It’s used for 8-bit operations on the lower half of AX.

The same logic applies to other 32-bit registers as well (EBX, ECX, and EDX). They can be broken down into their 16-bit counterparts (BX, CX, and DX) and further into the 8-bit high and low byte registers (BH, BL; CH, CL; DH, DL).

These smaller registers offer flexibility when working with different data sizes, enabling programmers to optimize operations and memory usage.

The challenge Description

For this challenge, we will use Ghidra, a powerful reverse engineering tool, as it simplifies the analysis of complex assembly code.

Here’s the assembly code for the asm3 challenge:

asm3:
	<+0>:	push   ebp
	<+1>:	mov    ebp,esp
	<+3>:	xor    eax,eax
	<+5>:	mov    ah,BYTE PTR [ebp+0x9]
	<+8>:	shl    ax,0x10
	<+12>:	sub    al,BYTE PTR [ebp+0xe]
	<+15>:	add    ah,BYTE PTR [ebp+0xf]
	<+18>:	xor    ax,WORD PTR [ebp+0x12]
	<+22>:	nop
	<+23>:	pop    ebp
	<+24>:	ret  

Behaviour Intuition of picoCTF asm3

First, let’s understand the function’s behaviour:

1. The function begins by setting up the stack frame with ‘push ebp’ and ‘mov ebp, esp’.
2. It then initializes the EAX register to zero using the ‘xor eax, eax’ instruction.
3. The code moves the byte at the address [ebp+0x9] into the AH register.
4. It then shifts the AX register 16 bits to the left using ‘shl ax, 0x10’.
5. The byte at the address [ebp+0xe] is subtracted from the AL register.
6. Next, the byte at the address [ebp+0xf] is added to the AH register.
7. The code proceeds to xor the AX register with the word at the address [ebp+0x12].
8. The function ends with a ‘nop’ (no operation) instruction, followed by ‘pop ebp’ and ‘ret’ instructions.

We can attempt the computation ourselves, but a good reverse engineer uses powerful tools. Ghidra is a helpful tool for this purpose, as it provides a visual representation of the algorithm or a version on a higher-level language (decompiler), making it easier to replicate.

To use Ghidra, we need to install it on our Kali Linux machine. Unfortunately, it doesn’t come pre-installed. However, the installation process is quite simple.

Follow these two easy steps in the terminal:

1. Update the repository first.

sudo apt update

Then install Ghidra:

sudo apt install ghidra

Now we can run it by typing this

ghidra

Ghidra

With the new tool open, let’s import the script and see what it looks like!

1. Create a new project.

2. Follow the instructions, name the project, and click on the code browser.

To analyze the asm3 file (test.S), we need to clean it by removing line numbers and then compile it using GCC. Our code will look like this:

.intel_syntax noprefix
.globl asm3

asm3:
	push   ebp
	mov    ebp,esp
	xor    eax,eax
	mov    ah,BYTE PTR [ebp+0x9]
	shl    ax,0x10
	sub    al,BYTE PTR [ebp+0xe]
	add    ah,BYTE PTR [ebp+0xf]
	xor    ax,WORD PTR [ebp+0x12]
	nop
	pop    ebp
	ret  

To compile, type this command in the terminal: gcc -masm=intel -m32 -c test.S -o test.o

• The .intel_syntax noprefix directive switches to Intel syntax from the default AT&T syntax.
• The noprefix option stops the assembler from adding an underscore prefix to global symbols, common in Intel syntax.
• The .globl directive makes the asm3 symbol visible globally, allowing access by other object files or linking with other functions.

Now, drag and drop the test file into Ghidra! We could run it with given inputs, but our aim is to understand, so we’ll do static analysis instead!

Decompiler

To better understand the code, we are going to use Ghidra’s decompiler. But what is a decompiler?

It’s a tool that takes machine code and converts it back into a more human-readable programming language. This helps people understand how a program works, making it easier to analyze and modify the code.

At the project’s opening, Ghidra will ask you if you want to analyze it, so confirm, keep the default settings and go ahead.

Now you will have the compiler on your right, just click on the text section in the left side view and you will see this screen:

Ghidra has converted the assembly code into a more readable function.

ushort asm3(undefined4 param_1,undefined4 param_2,undefined4 param_3)

{
  return CONCAT11(param_2._3_1_,-param_2._2_1_) ^ param_3._2_2_;
}

takes three input parameters: param_1, param_2, and param_3. All these parameters are 4-byte values (undefined4).

The function returns a 2-byte unsigned short value (ushort) as the result. To calculate the result, it performs the following operations:

1. Extracts the third byte from param_2 (param_2.3_1) and makes it the most significant byte of the result.
2. Extracts the second byte from param_2 (param_2.2_1), negates it and makes it the least significant byte of the result.
3. Concatenates these two bytes to form a 2-byte value (CONCAT11 is the way for Ghidra to say that it concatenates two elements of size 1).
4. Extracts a 2-byte value from param_3, starting at the second byte (param_3.2_2).
5. Performs a bitwise exclusive OR (XOR) operation between the concatenated 2-byte value and the 2-byte value from param_3.

Finally, the function returns the result of this XOR operation.

Find the result of picoCTF asm3

Now that we have a clear understanding of the assembly code, let’s find out what asm3(0xd2c26416, 0xe6cf51f0, 0xe54409d5) returns.

We can do the computation manually or we can write an equivalent script and check the result. I would opt for the second one using Python:

def asm3(param_1, param_2, param_3):
    return ((param_2 >> 24) << 8 | (-(param_2 >> 16) & 0xFF)) ^ (param_3 >> 16)

def main():
    # Test input values
    param1 = 0xd2c26416
    param2 = 0xe6cf51f0
    param3 = 0xe54409d5

    # Call asm3 function with the test inputs
    result = asm3(input1, input2, input3)

    # Print the result
    print(f"The result of asm3 function is: 0x{result:04x}")

if __name__ == "__main__":
    main()

I just want to delve into some important aspects of the code, in particular, the bitwise operations:

1. param_2 >> 24: Shift the bits of param_2 24 positions to the right. This isolates the most significant byte of param_2.
2. ((param_2 >> 24) << 8): Shift the result from step 1 eight positions to the left. This moves the most significant byte of param_2 to the position of the second-most significant byte.
3. param_2 >> 16: Shift the bits of param_2 16 positions to the right. This isolates the second-most significant byte of param_2.
4. -(param_2 >> 16): Compute the negative value of the result from step 3.
5. (-(param_2 >> 16) & 0xFF): Perform a bitwise AND operation between the result from step 4 and 0xFF (255). This masks the result to keep only the least significant byte, effectively making sure the result is an 8-bit value.
6. ((param_2 >> 24) << 8) | (-(param_2 >> 16) & 0xFF): Perform a bitwise OR operation between the results from step 2 and step 5. This combines the two 8-bit values into a 16-bit value, where the most significant byte is from step 2 and the least significant byte is from step 5.
7. param_3 >> 16: Shift the bits of param_3 16 positions to the right. This isolates the two most significant bytes of param_3 as a 16-bit value.
8. ((param_2 >> 24) << 8 | (-(param_2 >> 16) & 0xFF)) ^ (param_3 >> 16): Perform a bitwise XOR operation between the results from step 6 and step 7. This computes the final 16-bit result of the asm3 function.

Now we can finally run the script by typing in our terminal:

python main.py

And it will print:

0x0375

And we are done, that’s our flag!

Now let’s insert 0x375 into the input in picoCTF and the challenge is complete!

Conclusion

In conclusion, our journey through the picoCTF asm3 challenge has been an exciting and enlightening adventure. We’ve delved into the world of bit manipulation, reverse engineering, and Python programming to decode this complex problem. This hands-on experience has not only expanded our understanding of low-level operations but also showcased the power of persistent curiosity and exploration.

If you’re as captivated by this challenge as we are, don’t miss out on more thrilling content related to cybersecurity, reverse engineering, and programming challenges. Follow the Stackzero blog, where we unravel the mysteries of the digital world and bring you cutting-edge insights to sharpen your skills and knowledge.

For real-time updates and engaging conversations with like-minded enthusiasts, be sure to connect with Stackzero on social media. Join our growing community, share your thoughts, and learn from the experiences of fellow tech aficionados. Together, we’ll continue to push the boundaries of learning and conquer new heights in the captivating realm of technology.

Don’t wait another moment! Follow Stackzero now and unlock a world of endless possibilities in the fascinating field of cybersecurity and beyond.