Hmm... thinking further on it...

It could be work on in two steps.

1st) Adapt the first NN to support proper sentence learning.

AND

2nd) Adapt a second NN to seperately learn how to properly address the question properly.

Both layers being independant, except for the obvious linking of the decisions to give back peoper information...