The syntax of the binary temporal operators always requires square brackets (see https://es.cs.uni-kl.de/tools/teaching/QuartzRef.pdf).
Hence, using the correct syntax your thoughts are as follows with psi := [b SB a] | G !a
S = [c WB [b WB a]]
= [c WB [b SB a] | G !a]
= [c SB [b SB a] | G !a] | G!([b SB a] | G !a)
= [c SB psi] | G!psi
Now you claim that the following is valid (which is true):
(G(!c&psi)) -> !([c SB psi] | G!psi)
which leads to your first formula G(!c&([b SB a] | G !a)).
Second, you use [c SB psi] = [!psi SU c & !psi] (which is right) and you claim that the following holds:
[psi SU c] -> !([c SB psi] | G!psi)
which is not correct (I don't see why you think so). However, you may argue that
!([c SB psi] | G!psi)
= ![c SB psi] & !G!psi
= [(!c) WU psi] & F psi
= [(!c) SU psi]
and hence, you could use [(!c) WU psi] to prove
[(!c) SU psi] -> !([c SB psi] | G!psi)
I guess that was what you wanted.
However, also that will not be accepted by our clever tool, since even the following holds:
[(!c) SU psi] <-> !([c SB psi] | G!psi)
Note that the implication must be a strict one (that was the main reason why the requirements were stated in terms of sets where we have a strict inclusion operator). Hence, be careful with just doing equivalence transformations of the formulas.