People are better able to empathize with others when they are given information concerning the context driving that person’s experiences. This suggests that people draw on prior memories when empathizing, but the mechanisms underlying this connection remain largely unexplored. The present study investigates how variations in episodic information shape the emotional response towards a movie character. Episodic information is either absent or provided by a written context preceding empathic film clips. It was shown that sad context information increases empathic concern for a movie character. This was tracked by neural activity in the temporal pole (TP) and anterior hippocampus (aHP). Dynamic causal modeling with Bayesian Model Selection has shown that context changes the effective connectivity from left aHP to the right TP. The same crossed-hemispheric coupling was found during rest, when people are left to their own thoughts. We conclude that (i) that the integration of episodic memory also supports the specific case of integrating context into empathic judgments, (ii) the right TP supports emotion processing by integrating episodic memory into empathic inferences, and (iii) lateral integration is a key process for episodic simulation during rest and during task. We propose that a disruption of the mechanism may underlie empathy deficits in clinical conditions, such as autism spectrum disorder.