You can simplify !(a|a) to ¬a. It is the set Q2 whose predecessors you want to compute.
((q' → p) <-> (o&q → a)) & p'
is the symbolic formula representing your relation R.
Let's simplify this formula a bit:
((¬q' | p) <-> (¬o|¬q|a)) & p' = ((¬q' | p) & (¬o|¬q|a) & p') | (¬(¬q' | p) & ¬(¬o|¬q|a) & p')
=
¬q' & ¬o & p' |
¬q' & ¬q & p' |
¬q' & a & p' |
p & ¬o & p' |
p & ¬q & p' |
p & a & p' |
q' & ¬p & o & q & ¬a & p'
Next, we plug that in to formula on slide 59 of the chapter on state transition systems, and simplify.