Transitive inference (TI) that uses known relationships to deduce unknown ones (using A > B and B > C to infer A > C given no direct interactions between A and C) to assess the opponent’s strength, or resource-holding potential (RHP), is widely reported in animals living in a group. This sounds counter-intuitive because the mechanism of TI seems to require social cognition and large memory capacity; individuals, in the TI mechanism, need abilities to identify others, observe contests among others and keep the results in memory. We examine the coevolution of memory and transitive inference by the evolutionary simulations, using the asymmetric hawk-dove game when a cost for losers is higher than a reward for winners. We found that the immediate inference strategy (II), which estimates the opponent's strength based on the past history of the direct fights, evolves with the large memory capacity, while the TI strategy, which estimates the unknown opponent's strength by transitive inference, evolves with the limited memory capacity. When a cost for losers is slightly higher than a reward for winners, the II strategy with the large memory capacity has an evolutionary advantage over the TI strategy with the limited memory capacity. It is because the direct fights are not so costly that more information about the fights leads to more accurate estimation of the opponent's strength and results in the accurate rank of the RHPs. When a cost for losers is much higher than a reward for winners, the TI strategy with the limited memory capacity has an evolutionary advantage. It is because a good way to avoid the costly fights is the prompt formation of the dominance hierarchy which does not necessarily reflect the actual rank of the RHPs; the TI strategy builds the dominance hierarchy much faster than the II strategy regardless of memory capacity, and the large amounts of information are not required for the TI strategy to form the dominance hierarchy promptly. Our study suggests that even smaller memory capacity is evolutionarily favoured in TI. The TI strategy tends to reinforce the hierarchy once it is built, regardless of whether it is consistent with RHP or not, because results of direct fights are always counted. Smaller memory capacity allows players to adjust the hierarchy well if it does not represent RHP. These results prove that TI can evolve in animals, which do not have the large memory capacity.