This paper identifies the culprits of naively com- bining learning-based controllers and state estimators for robotic in-hand manipulation. Specifically, we tackle the chal- lenging task of purely tactile, goal-conditioned dextrous in- hand reorientation with the hand pointing downwards. Here, we observe that due to the limited sensing available, many control strategies that are feasible in simulation do not allow for accurate state estimation. Hence, separately training the controller and the estimator, and combining the two at test time, leads to poor performance. Our proposed solution to this problem involves training a control policy by reinforcement learning coupled with the state estimator in simulation. We show that this approach leads to more robust state estimation and overall higher performance on the task while maintaining an interpretability advantage over fully end-to-end learning approaches. Due to our unified learning scheme and an end- to-end gpu-accalerated implementation, learning only takes 5h to 8h on a single GPU. In simulation experiments with the DLR-Hand II and for four significantly different object shapes, we provide an in-depth analysis of the performance of our approach. Finally, we show the successful sim2real transfer with rotating the objects to all 24 possible π/2-orientations.