Sushant Dinesh

Radeco update

Mon, 21 Sep 2015 17:31:00 +0000

This post is to outline the work completed during the Google Summer of Code 2015 (GSoC) period and show you a glimpse of radeco and where we are heading with it.

For those who are not aware, radeco is a decompiler framework that is developed and maintained by the radare team. The entire framework is open source, flexible and reusable. The base of radeco is radeco-lib that implements the analysis and transformations that are needed for decompiler. radeco uses radeco-lib to perform decompiler. You can checkout these repositories here and here

Features

This section gives a brief overview of the internal working of radeco as of today. These may change over time as more features and analysis are added. Nonetheless this does form the basic foundations on which radeco is built.

Radare2 uses an internal IR, ESIL (Evaluable Strings Intermediate Language) for its emulation and analysis needs. Since radeco is built on top of radare2 it parses ESIL to an internal representation. The first step of building the decompiler framework is to develop a good intermediate language. During the start of GSoC period we investigated several intermediate languages(IL) like REIL, RREIL, Vine IL and BAP IL. We tried to incorporate the best features from all these ILs while trying to keep the language as simple as possible. The main reason ESIL is converted into another IL is that ESIL is not really suited for static analysis, it was built for emulation and dynamic analysis. Since there are already a lot of architectures that are supported by ESIL, it makes more sense to lift ESIL into our IL rather than writing a lifter for every architecture again. Initially, ESIL from radare2 is converted into a text based IL which is then transformed into a tree-like SSA representation that is used for all further analysis.

The second step is to transform ESIL that is emitted by radare2 into the IL. This is done in two stages. The first step is a basic transformation that transforms ESIL into radeco IL. This is then broken into basic blocks and a Control Flow Graph (CFG) is constructed. The Control Flow information is then used to transform the IL into a tree like SSA representation. This is used for further analysis and optimizations. Radeco supports a few analysis like constant propagation and dominator tree construction (this feature is not exposed using the API yet). Apart from analysis radeco also performs a basic level of dead code removal to declutter the code and remove unnecessary operations.

Currently, radeco can output it’s internal representation to dot file which can be converted into png/svg using utilities like graphviz. While this may not seem very useful, they do form the base of the entire decompiler and identifying bugs at this stage is important. Much of the work during the GSoC was laying the foundations and deciding an architecture that would allow the library to be as flexible as possible. Now that we have a solid foundation and a good architecture in place we can move at a faster pace and implement all the analysis needed to build a decompiler.

Usage

Radeco makes it easy for users to write their own custom analysis while leveraging the analysis that have already been implemented in radeco. One of the fundamental principles of radeco is allowing users to interact with the output produced from every stage of radeco’s analysis, make corrections or changes where necessary and then continue with the analysis. To make this happen radeco-lib introduces Pipeline. This allows users to decide which stages of the analysis is to be run, the order in which they are to be run and allow them to tap into the output that is generated after every stage. For more information about the usage please refer to the radeco’s README which explains this in detail. A pre-built version of radeco is to be released soon, but for now you will need a rust compiler (prefereably nightly) in order to compile and try out radeco.

And oh! We also have (some) documentation for radeco and radeco-lib.

Challenges

One of the main challenges we faced during the project phase was developing the internal representation for storing the SSA form. ESIL has features that are not present in other intermediate languages that make this process slightly tricky (this is also why ESIL is suited for emulation unlike other languages). ESIL has a concept of internal variables. These are basically variables that the ESIL VM maintains and hold a special meaning to the VM. All internal variables in ESIL are prefixed by $. The main use case for these variables are for computation of processor flags. To be able to compute the flags on-the-fly when needed the ESIL VM also holds three other values:

The value after the last operation that modifies or sets a flag (cur),
The value before the last instruction that modified the flag (old) and
The size of the last operand(s) (lastsz).

For example, let us consider the x86 zero flag. In ESIL, this is represented by $z. The VM on encountering a $z during its execution will perform the actual computations needed to decide the value of the flag, which in this case is checking if cur == 0. This means that the parser for radeco must also keep track of these values (cur, old and lastsz) and substitute the correct set of operations for each flag when it encounters an internal variable.

The other challenge that we faced was an architectural one. One of the principles that we kept in mind while developing radeco is reusability. We wanted all the analysis implemented in radeco to be reusable. This allows users to implement their own data structures for storage while still being able to use the analysis that radeco provides. To be able to achieve this we leverage the power of rust traits. Almost everything inside radeco is implemented in terms of traits to allow multiple implementations and reusability. Currently it is mandatory to use the radeco IL instruction set (while having the freedom to store the data in any manner) to be able to use the analysis that radeco provides. In the future, we aim to make this generic too, allowing users to use their own ILs and instruction sets. The main tradeoff for us during the summer was flexibility vs. time. The more flexible we needed the framework to be, the more time we needed to implement features in a way that would allow for this. Hence we decided to restrict ourselves and expand to support custom ILs later.

Future steps

Radeco is under heavy development and has a long way to go before being a full decompiler framework. Here is a quick and incomplete list of what to expect from radeco in the near future:

Type Inference
Type Propagation
More analysis
A full C-Writer module to output readable, C-like pseudo code
Control flow restructuring and reconstruction
Full integration with radare2
Text form for the internal IR allowing other tools to interact and use radeco for analysis
Maybe even a GUI for radeco!

Preview

Below is a small example to show constant propagation as implemented in radeco right now.

ASM of input binary:

It is clear from the input ASM that the jump is always executed and the instruction right after the jump is never executed. This aim is to check if radeco can infer this using constant propagation. This ASM was hand written as no optimizing compiler would produce this code (as they perform constant propagation too).

The binary generated is disassembled by radare2. Below is the ESIL output that radare2 generates and provides radeco with:

NOTE: Due to the large size of images, I’ve chosen not to embed them here. Please feel free to checkout the links for outputs.

This ESIL is parsed by radeco into an intermediate radeco IL: Non-SSA IL

The CFG is then transformed into the SSA representation: SSA

Constant propagation is run on the SSA form. Radeco implements Sparse Conditional Constant Propagation which performs constant propagation and unreachable dead code elimination simultaneously. Here is the output for reference.

As you can see from the above output, the unreachable branch is eliminated and radeco infers correctly the values of registers and flags after the function has been run.

Contributing

We encourage you to give radeco a try. Contribution in the form of bug reports, pull requests, improving documentation or even just community suggestions are welcome. If there is any feature that you miss and you would like to see file a report with a description of the same so that we can address this. Providing a backtrace and the sample binary (where possible) would help us track down the bug much faster. Please consider this while filing a report.

If you’re interested in using radeco for your own tools and have some questions, feel free to join #radare on Freenode. We would be more than happy to help you out!

GSoC 2015 Week 1-3

Mon, 15 Jun 2015 19:31:00 +0000

It’s been three weeks into GSoC and I’m having an amazing time. I am working along side dkreuter and been learning tons from him too!

Here is the repository where you can track our progress and also give us suggestions :)

We chose Rust as our language for implementation. Though at first I was a bit scared of this choice, I quickly realised how great the language is! Rust has allowed be to far more productive, after of course my initial battles with the borrow checker. The zero-cost abstractions allowed by Rust has been a great so far!

This is a quick roundup of what I’ve been upto:

Firstly, I got an ESIL parser up and running. We use this parser to convert to an IR which is easier to perform static analysis on. While the ESIL is amazing for emulation purposes, it’s not so much for static analysis as it has a very large number of supported opcodes. The RadecoIR (name subject to change) is much more simplified in terms of the number of opcodes. Also, ESIL is primarily just strings (as suggested by its name “Evaluable Strings Intermediate Language”) which could be pretty hard to work with.
The next step was to build a Control Flow Graph (CFG) out of the IR. Building a control flow graph will allow us to reason about the way the control flows in the program, and hence, helps us better understand the different programming constructs that go into making it. To make debugging and visualizations, I first we ahead and implemented a dot format emitter (ok, I admit it, I probably did this first as I felt it was more fun :P). Just a brief word about dot, dot is a graph description language which can be used an input to graphviz to obtain visual representations of a graph. The current implementation is just a minimal dot format emitter and all the features of dot format are not fully supported. At this point, it still cannot be used on any generic graphs. I plan to expand this in the future to allow more features and work on generic graphs.
The last part was making the CFG out of the emitted RadecoIR. This turned out to be pretty simple, however, there are still a ton of improvements to be made here :)

Apart from Radeco itself, I also helped in making r2pipe.rs which allows communication with radare2 over pipes. In the future, r2pipe.rs will be used to connect Radeco to radare2. R2Pipe.rs is great news if you’re a rust guy as you can now interact with radare2 and extend it to meet your needs!

Check out the documentation if you’re interested :)

This is just a glimpse of what’s in for the upcoming weeks:

Integration with radare2.
Write tests and documentation for all features. The limited documentation we currently have can be found here. This will be updated soon.
Convert the IR into the SSA form.
Write dataflow analysis on the SSA form and also perform optimizations like deadcode elimination.
Constant folding.
Type propagation.

Bonus: Here is a small example of the CFG graph that we currently generate

An Introduction to radare2

Sun, 31 May 2015 02:04:35 +0000

A couple of days back I conducted a session on “Introduction to radare2” over irc for a few people from my college. Here are the notes I made in case it might be helpful for someone else.

###Introduction radare2 is an opensource reverse engineering framework. Other reverse engineering tools include IDA and Hopper. These are extremely expensive and r2 aims at being the ultimate reverse engineering tool in the future and replace these.

Official repository of radare 2 is here, feel free to join in and contribute if you like to :)

A full(?) feature list of r2 and comparison of r2 vs Hopper vs IDA can be found here

Links to other cheatsheets and documentations (which you may like):

Cheat sheet
Official Radare2 Book
Using radare2 for Pwning
radare2 blog has some interesting articles to pwn ctf challenges using r2.
radare2 Wiki
#radare (official channel) on irc.freenode.net if you need any help from r2 folks anytime.

r2 has a ton of features which takes a lot of time to explore and understand. Think of r2 like vim/emacs. Unfortunately it lacks a robust GUI. Feel free to try out the web GUI or Bokken

It has a steep learning curve but we need only a few commands to do basic reversing (and for ctfs) and that is all we’ll be seeing for today :)

Disclamer: I started using radare2 recently too. These are just some of the commands that I felt were most useful and spent some time in learning. I hope it’ll be useful for you too to quickly get up and running and explore the world of r2. Think of this as a newbie’s guide for newbies!

Take your time to explore r2, it’s definitely worth it.

A (very) small tutorial for absolute newbies:

Most Important tip for today (and as long as you use r2!): What most people don’t realise is that r2 is self-documenting. Whenever you don’t know any command, its semantics, what it does etc. use ?.

Example: Just running ? will give you a list of all commands. Now look at a. The help menu says: “Perform analysis of code”. To get more information about commands starting from a, run a?. Use this to learn and discover r2. When in doubt feel free to consult wikis, guides and talk to people on #radare. q is usually used to exit menus and eventually radare2 itself.

Also usually all mneumonics are dervied from their longer form.

Usually this is the workflow you would follow:

Start up r2 by using: $ r2 ./hello_world
Run aa to “Analyze All”, or the newer aaa.
Enter V to enter “Visual Mode”. (Hint: You can use ? in Visual mode too)
To view the graph of a function hit V. If you don’t see a graph when you enter into graph mode, it usually means that you forgot to run the analysis (rarely it could be a bug in r2, in which case please do report).
Hit p to show the disassembly at the current location. Hit p again to go into debugger mode which shows all register states.
v to enter code analysis menu. This menu shows all the functions analysed. Selecting one and pressing g “seeks” to that function. So the first thing to do is seek to the main function. This will usually be shown as ‘main’ or ‘sym.main’. Normally you’ll want to begin analysing the binary from here.
In visual mode, if you want to run a r2 command, simple hit :. This brings up the same shell that you would have access to outside of the visual mode. All commands that work there work here too. To close the command line, just hit enter with a blank line.
use s <fn_name> (Example: s sym.main will take you to main directly) or s <offset> to “seek” to locations. s- to undo seek, and s+ to redo seek. This allows you to traverse the binary efficiently. Tab completion is available to help you out here :)
After some analysis, you might want to rename functions, to do so use afn <new_name> [offset].
To rename local variables, use afvn [identifier] [new_name]. This is the same for function arguments, but use afan instead.
Once you have done some analysis, you will want to save your work so that you can return to it later, use Ps [name] to save it as a project. Please check out P? for other project related commands. Most of them are self-explanatory.
To load a project, use Po [name]. Alternatively, you could also do this while starting up r2 buy using the -P option. (Example: $ r2 -P [name])

Additional:

To show all strings in a the data section of a binary, try: iz.
To show all strings in the entire binary try: izz.
Want to search for a string ‘Foo’ in the binary? Simple, do: / Foo.
This will return something like: > <offset> hit0_X "Foo". To quickly go to this location, s hit0_X. Again, tab-completion is available.
To help further with traversal, r2 offers vim style marks. To place a mark at an offset, use mK. Jump to a mark ‘K’ by using 'K (exactly how it works in vim!).
Don’t like a theme? Check out default themes using eco?. To select a theme run eco [name] .

TODO

o to seek.
u/U to undo/redo seek.
dr and d in general in visual mode.

This was just a basic introduction. radare2 offers many many more commands and is extremely powerful. Plus we’re constantly adding new commands, features and improvements, so stay tuned!

GSoC 2015!

Thu, 14 May 2015 14:00:35 +0000

Late post. But I got selected for GSoC 2015 under openwall/radare! :D

So for the next three months I get to contribute to radare2 without any interruptions and also get paid! This is awesome! I would like to thank my mentors Anton Kochkov and Jeffrey Crowell for giving me this opportunity. Looking forward to an codeful summer ahead!

Coding period starts 25th May. So stay tuned for more awesome posts about radare2!

GiTS 2015 Teaser Writeup - Lost To Time

Mon, 15 Dec 2014 05:11:35 +0000

Well this is my first blog post about a CTF, feel free to let me know what you think.

Challenge:

So I lost my computer. And also my compiler and most of the documentation. Can you help me, please? file

The challenge presented us a XZ Compressed data file. Decompress it with

xz -dc < losttotime > lost

The instructions in the file belong to COMPASS Assembly Language. The first thing we did was to find some documentation which gives us a reference of the instruction set to understand the instructions. We also need to understand the architecture of CDC6000 system. Finding documentation proved tricky as COMPASS is very old and the only documentation available were pictures of actual documentations (so no search in pdfs). Links to documentations we used are at the bottom.

We also observed that each VFD (lines 54 - 77) are 60 bits long each. ‘B’ means that the given numeric constant is in octal representation. We went ahead and converted each of them into their 60-bit long binary representations as we would need this in our code. data file.

Using the documentation we decoded the given assembly as:

This is fairly easy to convert to a program. Ruby is my choice of language. Here is the program that will give us the flag:

Running the same gave us the flag:

FLAGiCYBERCONTROLCYBERDATACYBERCORPORATIONj

This was a great challenge and we enjoyed sifting through those ancient documentations and getting a glimpse of how computing has changed over the years. Thanks to the organizers for a great challenge!

Here are links to some documentations we used:

Compass Sample Code - Wikipedia
Instruction Set - Partial
Instruction Set - This in particular proved most useful.

First Post

Fri, 05 Dec 2014 15:32:46 +0000

I guess this is Introduction?

You can find out more about me in the About section of this blog.

I’ll mainly be blogging about security and CTF related activities. I also contribute to Mozilla, so I may also occassionally blog about my amazing experiences with that. I believe that web and opensource are the future. I like meeting and learning from other developers (I’m still a newbie though!). So feel free to get in touch with me if you have suggestions, or even for random discussions!

On to the Hacks!