I was having lag even though all gold/iron/crystals were smelted and gems in a bucket, there was nothing visible on the conveyor belts.

I then noticed that there was a crystal glow coming from inside one of the conveyor sorters and realized there was a crystal stuck.

I ran a magnet slowly through it and pulled out a bunch of gold/iron/crystals/gems but the lag was still fairly bad.

I then decided to just pick the sorter up and a whole crap load of everything came out of it.

Did some additional testing, allowing the game to get to the same lag each time, and determined that it seems to happen only with the first sorter in a line of sorters.

If you setup multiple mine lines and each line has its own sorters, it will happen to the first sorter of each line.

It appears that enough can get stuck in the first sorter to cause lag. Do not appear to see the same issue with additional sorters.

Setting a conveyor shredder before the first sorter, to make sure everything goes down center, does NOT appear to prevent the issue.

So if you are lagging, have no gold/iron/crystals visible that are causing it, try lifting your first sorter and see how much falls out.