Is there a way to stop a driver from wearing down?

Don't use it, store it in a safe place within a protective container unassembled. That's about it.

If you use it, it will wear. That's how contact surfaces work.