I'm always on the lookout for a good hex editor. I've tried ImHex a few times and it doesn't quite fit the need I have and I prefer HxD for the simple things. The Pattern Language is pretty neat, I hadn't looked into that before. For the complicated things, it's not always a linear process like in the case of the blog article. Mainly, there's often lots of gaps in my structure until I figure them out. Some tools like ImHex or Kaitai, you can put in placeholder fields between two known objects, but that's kind of annoying. 010 Editor lets you highlight and put colored bookmarks, which kind of works. The workflow I've ended up using for reversing in-memory data structures is to get a hexdump of the memory, take a screenshot and then put it in OneNote. Then I can draw colored boxes or highlight and put annotations off on the side with what things are or notes. If a field is a memory address to another related and unknown structure, I often will then get a short dump of that memory and put a screenshot of it on the same page with an arrow to it. It's pretty kludgy, but I like the freeform nature of it. It's basically a RE notebook, just without any useful RE built-ins. If anyone has ideas or recommendations on better discovery workflows, I'd love to hear them.

Story time: I worked the night shift at a motel during college. "Bored to tears" was an understatement. I was poking around the front desk computer system one night, as one does, and found its data file. Lacking anything else to do, I wrote a little hex dumper in BASIC so I could explore the file. The first thing I noted was that customer names were spaced exactly N bytes apart. Oooh, fixed records! Then I spent the next week or so reverse engineering the DB file format. I'd twiddle a field in a random customer record then look at their record in the hex dump to see what changed, then update my notes.

Eventually I wrote another little BASIC program to run analytics, like which customers spent the most with us, and auto-fix mistyped names ("This says 'Bb Jones'. Did you mean 'Bob Jones'?") by writing directly to the file.

I got a pretty sweet little bonus for my hackery.

I finally dipped into using ImHex when doing some reversing of the PS2 memory card format for https://ps2iodb.com.

The program is so good, and has enough of a reputation that I knew to reach for it even though I know nothing about reversing, and that was 3-4 years ago now! Also shoutout to the awesome GUI library is uses; https://github.com/ocornut/imgui

My few rules are: 1) What's the entropy? This helps with encrypted/compressed. You probably need to overcome this? 2) What's the context? Apps are developed in context and context suggests formats. Is it C? Then expect structs. Does it need to go over the wire? Expect run-length encodings. Python/JS - JSON/pickle. 3) There are broadly speaking only a few ways you can read data back into memory: fixed layout, run-length encoding, terminators. A great case-study is ASN.1, once you've built an ASN.1 parser by hand no file format will ever daunt you.

I tried to do a similar thing last year to datamine a gacha game's assets and somehow did it without knowing the existence of tools like IDA/Ghidra. They're using a custom db format that was XORed with a key. Iirc it was a bizarre format, it's actually an archive where each column is a file containing all the rows of that column. So a table is just a folder full of these files. Maybe it's a common thing idk. I got a partially working XOR key so I used ImHex to look at the partially decrypted db file and manually looked at the decrypted hex values to "correct" them.

I wish I read something like this article at that time!

Spoiler: half the time, that unknown file format is a zip containing a sqlite db

I’ve had a lot of luck with ImHex looking at various binary formats and even using it to aid in writing some file magic.

It would be really nice to be able to roundtrip C headers with it. The syntax is close, but often I older formats there is a 1:1 mapping with C structs that doesn’t quite match the imhex syntax.

I've been reverse engineering some old Minecraft Legacy Console Edition save files with ImHex and the pattern templates work pretty well, though debugging is sometimes painful (e.g. reading runtime-sized arrays can just overflow the file and error without showing previously decoded structs). Great tool though. LLMs are, of course, scarily good at reverse engineering and spotting patterns that take me way too much effort to spot.

I've always wondered how something like an Autodesk Inventor sldprt file was formatted so that I could write a program to manipulate those files.

I assumed it was proprietary and left it at that.

A few years later, I saw that Onshape allows importing sldprt files into their system and I wonder how they were able to provide that ability without some proprietary knowledge transferred

I’m going to try that on the newest outlook binary format used to store the emails locally... why couldn’t they stick to the plain simple sqlite?

ImHex makes tackling proprietary formats so much less painful. Seriously a lifesaver for quickly mapping out structures.

If only it were BlackIce, it would be perfect.

[dead]

Damn a video game spoiler warning, guess I can't read the rest of the article.

I may feed it into an LLM to strip out the spoilers actually