This is why modern criminal justice theory took several thousand years to develop...
Modern, yes, but I always find it amazing how the beginnings of it were already in place back in Classical Greece. Ever read Aeschylus' Eumenedies? It's the culmination of the Oresteia trilogy, thoughout which people have been killing each other in the name of justice (indeed, their causes were just, and at times god-ordered), but each dead only led to another, until finally Orestes, the latest to have killed someone (his mother in this case), is hounded by the Furies because of the blood-guilt that is on him. Vengeance and vindictive justice, the ways of the 'old world' (as Aeschylus portrays it, personified in the Eumenidies), keep going eye for an eye without end. But end it does, because the new gods, the Olympians, who deal not in violence (okay, leap of faith there - this is how they are portrayed in this particular play) and are enlightened, end it, and they do so in this manner. Apollo sends Orestes to Athens, and there the first trial for murder is arranged. Orestes is defended by Apollo, and prosecuted by the Furies. Athene is the judge, and the elders of Athens the jury. They speak their points, the jury has an even vote, and Athene votes on the side of Orestes. The cycle of bloodshed is over, because imparital third-party justice has allowed it.
Now, of course, this myth likely is one of explanation, specifically, how the Areopagus (the council that judged murder trials), came to hold this power. But the idea that violence cannot solve violence, and that reason and order must conquer it, even as the Olympians did the older generation, was one that the Greeks of that era were well aware of. So, while our system has taken a long time to get where it is, the basic premise has existed for at least 2500 years.
You mention in a later post tribalism and its bloodshed. It is interesting, but the Greeks, specifically Athenians, had their various tribes (can't remember how many.) However, during the democratic reforms they were essentially removed, replaced by an artifical group to which one belonged. Especially, it removed the patronymic 'I am this person, son of this person', connection to a clan or large family, which could cause problems within a democracy. Also, it makes it rather difficult to have inter-tribal feuds when the tribes have been annulled. Tribalism and imparital justice are very difficult to reconcile.
(Certainly, there were other forms of this as well. Laws were codified, and I'm absolutely certain that even in 3000BC councils of elders decided matters such as this. But this is the earliest examination of the uselessness of vengeance that I can think of.)