Software Security


What's the current 'state of the art' as way as software security only because I don't want my own software to be broken.

CSCI 2951U Topics in Software Security

From the overview of the course:

"CSCI 2951U investigates the state-of-the-art in software/systems exploitation and defense. More specifically, the course is structured as a seminar where students present (along with the instructor) research papers to their peers. We will begin with a summary of prevalent software and hardware related defects, which are typically found in applications written in memory unsafe languages, like C/C++, and/or contemporary architectures, such as x86 and ARM, and proceed to surveying what we are up against: traditional and modern exploitation techniques, ranging from classical code injection and code reuse up to the newest goodies, like JIT-ROP, Blind ROP, and software-based microarchitectural attacks. For the bulk part, we will be focusing on the latest advances in protection mechanisms, mitigation techniques, and tools against modern vulnerability classes and exploitation methods."

How the course works:

"Every week we will be discussing (a set of) research papers. Students are expected to read the assigned papers and write a short review (critique) before each class…..(Your paper review) should discuss the pros and cons of the proposed idea, protection mechanism, or bypass technique"

Exactly what I am looking for, a survey of modern attacks and mitigations to stop them not a bunch of how-to tutorials, we can figure out the how-to ourselves after reading about their implementation and experiment ourselves. All the prereqs needed we'll just figure out ourselves backtracking and filling in gaps as we go.

Lecture 1

The lecture slides. There's prereqs but they indicate we are going to review the most important concepts anyway in all the papers and assigned reading, great.

We just need to know:

  • basic C/x86 asm
  • code injection
  • code reuse (return oriented programming)
  • linking and loading
  • virtual memory

The first paper they want us to read is this SoK: Eternal War in Memory. Get the DOI number from that page and paste it into Sci-Hub or click on PDF.

Reading the abstract they setup a general model for memory corruption attacks, pointing out performance is a problem for using better memory policies so nobody uses them. We read this with the course advice in mind to 'discuss the pros and cons of the proposed idea, protection mechanism, or bypass technique'. Let's read the paper.


This paper talks about pwn2own which you can watch day 1 here and around the 6hr mark there's a good interview with an Egyptian hacker who says it takes 3 months to understand how a thing works, then not very long after to find a vuln using IDA Pro to reverse engineer said thing. He says in the beginning he knew nothing and was just persistent to get to his level. There is a moment of cringe when the interviewer implies that web hacking is somehow not as 'hardcore' as blowing up junk VMs or reversing some junk proprietary binary. Egyptian hacker guy seems unimpressed with this question and repeats that you just have to put in the work to understand whatever it is you're trying to exploit. He ends with saying the biggest problem is nobody is looking at solutions which is something Egor Homakov another security researcher has written about many times: actual investment into real security is a waste of time to most software companies when the current market for software is like a slot machine you add as little investment as possible and pull that handle until you cash out meaning somebody bigger buys your software and now security is their problem. I'm also amazed anybody wastes their time with these competitions, if you look at their 2021 live results, some team chained together 6 bugs to execute code but because a single bug was declared to be already known (and of course not yet patched) they were screwed out of full payment.

Shadiest of all hacking competitions

Cytaka is a real global competition held in Dubai or Israel every month engineered by what looks like Israeli-Emirates joint gov actors for $1m to $250k prize money. It's pretty obvious if you attend such a thing you'll likely be recruited even if you don't win for state sponsored hacking, or recruited in the app. I have no moral high ground to say being a mercenary security developer for the Emirates is right or wrong since my own country is doing the exact same thing just has better image marketing.

Looking at the Cytaka app, it's gamified training so you need to win in-app money in order to open later levels. Actually not a bad idea, in fact I was going to gamify the entire learnai site eventually esp the math content. The top 10 each month get sent to compete, so everything resets monthly. Playing around in it, there's a lot of weird semantic mistakes in the multiple choice answers like 'all of the above' or 'everything above', and only one answer is correct. If you want try the app as you finish this workshop, maybe eventually you'll get into the top 10 that month and can compete for $, just be aware anyone hiring you there or in the app is surely from some kind of gov agency.

Basic machine arithmetic crash course

Watch the second lecture from CMU's 15-213 playlist to understand what a bit is, what a byte is, what two's complement is, TMax and Tmin, hexadecimal encoding, basic shift operations, you don't have to watch the whole lecture where it later goes into type casting. You can also get the latest version of CS:APP from library genesis if you don't want to watch a lecture, the lecturers wrote that book. It goes into much more detail than we will ever need here and can be looked at later.

Basic C crash course

Next these brief notes to learn the C memory model. They are a great resource, getting right to the point and explaining just what we need aimed at a reader who has functional programming experience in Racket, and I assume you've done CS19 in the other workshop and Pyret was originally built from Racket so you will have no trouble understanding this.

Start reading at 2.1 Basics looking at the C memory model diagram, each row is 32bits (one word) which the author notes that 32bits is used for the purposes of making it easier to read this guide as it's similar to how a 64bit model works anyway. This is a recurring theme in courses I have taken, for example CMU's operating system course is only taught with 32bit because conceptually the same thing is going on and 64bit is a nightmare to work with for assignments they do like building your own tiny kernel. If you read the course syllabus the prof does not consider fighting with x86-64 complexity to give you any more insight into how a kernel works.

Back to 2.1, he goes through a sample C program, these are all sequentially executed statements that return nothing (void) they produce side effects and the return statement is an exit back to whatever program called it, usually a shell, and the return status numbers indicate error codes. Moving on to 2.2 and 2.3 you can see an example of prototypes or external global variables here in MIT's implementation of xv6 which is a V6 unix clone they use to teach their operating systems class. Some of the header files have external globals some don't. At the end of 2.3 there's questions: what happens if we exclude x file? The compiler can't link, if gcd-driver.c forgets to include gcd.h then the global variable being used in gcd-driver will fail, the compiler will also give warnings no prototypes when it compiles gcd-driver.c and doesn't know the types of the parameters of the gcd function inside gcd-func. This will all be covered when we do linking.

2.4 Iteration if you've taken CS19 in the compsci/software workshop then you already know what structural recursion is, it means taking apart the structure the same way you built it up. Of course you can translate all the Racket programs in this book to Pyret (if you've done this workshop)

fun sum1(n :: Number) -> Number:
  doc: "Structural Recursion, if the structure is a natural number n++ then disassemble by n--"
  if n == 0:
  else: n + sum1(n - 1)
  sum1(5) is 5 + 4 + 3 + 2 + 1

fun sum2(n :: Number) -> Number:
  doc: "Accumulator recursion, keeps a running tally in a variable and returns it"
  fun sum2-helper(counter, acc):
    if counter == 0:
      sum2-helper((counter - 1), acc + counter)
  sum2-helper(n, 0)
  sum2(5) is 5 + 4 + 3 + 2 + 1

2.5 Computational Model the standard I/O library opened with the #include <stdio.h> statement translates to: open the namespace of this library filled with standard input/output programs so you can call them by name and they're all in scope like printf() function. Since we are taking a security course, you'll see later that the more refined scope you can have the better so your programs don't have access to things like the network if they're not supposed to. In other language idioms, you would prefer to only extract from libraries the single functionality you need like 'import printf from (library)', create a let block where that namespace is contained and then you aren't carrying around a gigantic attack space in the entire program.

The stack frame writeup here is excellent, no silly diagrams just memory examples. You can also watch this lecture describing what it is from a x86-64 point of view. If a program main() calls a new function y(), a stack frame is allocated where y() will store all it's local variables in memory. When y() returns that memory is deallocated and can't be relied on anymore. Here is plenty of opportunites for hacks, what if you change the return address of a stack frame to something else, you can jump somewhere different in the program or access memory you shouldn't be able to. What if you return the stack out of order like move the stack pointer to whatever frames you want on the stack? You can then like using lego blocks rearrange the all the returns to do your evil bidding which we'll learn soon. Skim through the next chapter to read about C loop idioms if you've never seen them, I didn't know they were originally to help signal to the compiler where to optimize.

Chapter 3 pointers. This is the clearest and most efficient description of pointers I've ever read. 'Word size' means the largest size of memory that machine offers which is typically 64bit.

Why do pointers exist? Because if you consider a machine's memory model to be a gigantic flat array of bytes, and you want to be able to create linked lists and queues or trees and whatever with that flat array, you can use a memory address abstraction to merely move around pointers that define the structure and everything you do is in constant time. Example: adding a new value on to the head of a linked list. Instead of having to copy the old list to a new resized memory location including the new head you merely move the list head pointer and leave everything else where it is. It doesn't have to be all side by side in memory, just needs a pointer to the next memory location to find the next linked list element.

The semantic demo here of pointers requires careful following until you get it, simply remember the rules of lvalue and rvalue:

  • *p = 5 (lvalue, store 5 in memory location of p or &p)
  • x = *p (rvalue, return contents of &p which is 5)

Important quote: C programs can expose enough of the machine they are running on that even perfectly correct programs can exhibit different behaviour when compiled and run on different machines. More opportunites for hacks.

3.4 Heap allocation in C another great and brief chapter telling you just what's needed to know in order to understand how malloc works conceptually generating a pointer to a chunk of heap memory. Heap memory has different definitions ie: data structure heap memory, here it solely means the portion of memory that has been dynamically allocated by malloc. For example if main program calls a function, we know that function returns and the stack frame is lost and it's memory contents unreliable. If you want to keep that memory around for whatever reasons, have malloc manually initialze and free it later when not needed. More opportunities for hacks of course, what happens if if you free it twice accidentally or intentionally? These notes point out the true functionality of sizeof which is an instruction to the compiler for portability substituting the actual machine size.

Skip to 3.7 and 3.8 about arrays though this entire set of notes is worth working through like seeing the heavy use of mutation for O(1) queue/dequeue running time. Now you know C array indexing. Reallocate or realloc is described here to dynamically resize arrays, briefly skim all this noting where to return if later needed. Finally Intermediate C skim over pointer arithmetic and strings. You're done.

Paper: Eternal War in Memory

Finishing this paper. Figure 1 was in the lecture slides, now we know what a dangling pointer is: a pointer to freed malloc created heap memory or the other example in the If You Must Learn C flanerie where he showed what happens when you have a dangling pointer to a stack frame, like trying to return the memory address of a variable created in the frame back to the calling program. Amazing how those 2 simple errors can result in 4 categories of disaster. The green part of the diagram is the common mitigations, like data space randomization to prevent information leak. In fact notice how much randomization is being used now in this figure to try and stop these attacks from instruction set randomization to making the memory addresses non deterministic so you can't easily predict where things are on a stack or in memory to jump to, and yet they still don't work.

Reading II Attacks everything begins with triggering a memory error. Kernel-space means the protected memory space reserved for your operation system functions as compared to user space which is low privilege run programs.