You can simplify !(a|a) to ¬a. It is the set Q_{2} whose predecessors you want to compute.

((q' → p) <-> (o&q → a)) & p'

is the symbolic formula representing your relation R.

Let's simplify this formula a bit:

((¬q' | p) <-> (¬o|¬q|a)) & p' = ((¬q' | p) & (¬o|¬q|a) & p') | (¬(¬q' | p) & ¬(¬o|¬q|a) & p')

=

¬q' & ¬o & p' |

¬q' & ¬q & p' |

¬q' & a & p' |

p & ¬o & p' |

p & ¬q & p' |

p & a & p' |

q' & ¬p & o & q & ¬a & p'

Next, we plug that in to formula on slide 59 of the chapter on state transition systems, and simplify.