You can simplify !(a|a) to ¬a. It is the set Q2 whose predecessors you want to compute.
((q' → p) <-> (o&q → a)) & p'
is the symbolic formula representing your relation R.
Let's simplify this formula a bit:
((¬q'  | p) <-> (¬o|¬q|a)) & p' = ((¬q'  | p) & (¬o|¬q|a) & p') | (¬(¬q'  | p) & ¬(¬o|¬q|a) & p')
=
¬q' & ¬o & p' |
¬q' & ¬q & p' |
¬q' & a & p' |
p & ¬o & p' |
p & ¬q & p' |
p & a & p' |
q'  & ¬p & o & q & ¬a & p'
Next, we plug that in to formula on slide 59 of the chapter on state transition systems, and simplify.