The reason is that the WCF WS-Policy contains this section:
This means that your WCF service uses X.509 certificates negotiation. In such a scenario, clients use the server X.509 certificate for encryption. The unique here is that clients are not required to have this certificate out of band (as in most cases), rather they get this certificate using a SOAP-level negotiation. This is implemented as an extension over WS-Trust. While this is not strictly a proprietary Microsoft solution, Microsoft was the only one to implement it so far. In short - X.509 negotiation (SslContextToken) is not interoperable. Unfortunately(?) the default configuration of a WCF service is to use this negotiation. To turn it off either update your WsHttpBinding configuration:
Or choose the correct scenario in your CustomBinding:
You can also use the equivalent cases for username authentication. Note that any clients (including WCF ones) will now need to have the service certificates defined out of band.