Prometheus mysqld_exporter and “Access denied” errors

This is rather specific case, but I couldn’t find solution online, so, here it is.

After installing mysqld_exporter (it was done via Ansible role cloudalchemy/prometheus, but that should not matter) to get some real-time info about my MySQL 5.7 server, I was not able to gather metrics beyond few items like mysqld_exporter_build_info and mysql_up, with the latter always being zero. To be clear, exporter was running, it was accessible, but it was reporting MySQL server to be down.

I have encountered following error in system logs:

Sep 05 20:49:02 host mysqld_exporter[2411]: time="2020-09-05T20:49:02+03:00" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'exporter'@'localhost' (using password: YES)" source="exporter.go:119"

As you can see, connection attempt was successful on network level, but password check did not pass.

Did some googling, and here is incomplete list of possible reasons:

• quotes around values in config files (this is probably specifc to Ansible)
• funny characters in password: %, #, etc.
• invalid user host

None of that applied in my case. So, here is another possible reason:

• by default, clearstext passwords are not allowed because client-side plug-inmysql_clear_password is not enabled (some more info from Oracle)

To fix that, set it to true by modifying DSN:

DATA_SOURCE_NAME=exporter:password@unix(/var/run/mysqld/mysqld.sock)/?allowCleartextPasswords=true

Relevant part is ?allowCleartextPasswords=true at the end (or &allowCleartextPasswords=true if there are other options). The option is parsed by Go MySQL library, docs are here.

There are some other options which might help in your particular case: allowNativePasswords, allowOldPasswords. In my case enabling cleartext passwords was the only missing piece.