There is a bias in primes, typically known as Chebyshev's Bias or the "Prime Race". Basically, this bias says that if you start going along the number line and sort the primes into two bins as you go along, one bin for prime that are 1 mod 4 and the other for those that are 3 mod 4 (we can ignore2), then for 99% of this "race" there will be more primes in the 3 mod 4 bin. That is, most of the time there are more primes that are 3 mod 4 than there are primes 1 mod 4.

This works in greater generality. If we sort primes mod N then there will (usually) be more primes in buckets x mod N where x is not a square mod N than in the buckets y mod N where y is a square mod N. It seems as though the primes try to avoid being quadratic residues as much as they can.

This is a bit odd in light of Dirichlet's Theorem on Primes in Arithmetic Progressions, which says that the primes are all equidistributed in appropriate residue classes as a whole. Though, it isn't inconsistent with this result, as it merely says that at the end - even if non-residues are in the lead for most of the race - it will end in a tie. But this bias still exists and there should be a meaningful reason for it.

To dig into these results a bit we, naturally, end up working with the Zeta function and its friends. These do, after all, hold information about how primes are distributed within its analytic properties. But with these functions (and some strong Riemann Hypothesis assumptions) we can prove this bias and quantify it pretty well (see here for more). But with this analysis we can actually see how we are, in a way, creating this bias.

The zeta functions do not tell us about primes. They tell us about prime powers. A lot of results about primes which use the zeta function actually prove a related result for prime powers, and then get rid of the contributions to higher prime powers through some kind of approximation or correction. You can see this happen in the Divergence of the Sum of Reciprocal Primes and in the Prime Number Theorem when we pass to the Chebyshev Function. Overall, the zeta functions tell us that prime powers are the natural objects of study when it comes to the distribution of primes.

The proof of Chebyshev's bias is along the same lines: Get rid of the contributions of higher powers through some asymptotic bound. But through this process is when we end up creating the bias itself. There is no Chebyshev's Bias for the prime powers and it is through the exclusion of higher powers that we create the bias. To see how this works, we can consider the case of sorting the prime powers mod 4. When we do this, every prime-square will fall into the 1 mod 4 bucket. That's a lot of numbers that automatically go to one of the buckets and to balance it out we will need the primes to act as counterweights which means favoring the 3 mod 4 bucket. And so Chebyshev's bias exists to prevent a bias in the distribution of prime powers.

I'm not the best at making graphics, but you can track how often one bucket is favored over the other as a percent and when you compare primes to prime powers, you can see the bias disappear in this graph). There are some scaling factors in here, but these arise from how Chebyshev's function tells us we should weight prime powers when counting them. Overall we can see that, in the end, Chebyshev's bias is an anti-biasing mechanism.

But, in the end, Terry Tao can explain things much better than I, though he didn't include a poorly made Python graph to illustrate things.