Samba Debugging =============== ## Always run `testparm` Maybe one of the most important tools we have. The `testparm` utility checks if the the `smb.conf` is valid. Run # testparm -s If somone provided you with a smb.conf file use: # testparm -s /path/to/smb.conf If modifications are done to a smb.conf file you should always run testparm to verify it is correct! Also if you're repoting a bug please always share the output of `testpam -s` **How to collect logs** We use winbind as an example here: # systemd stop winbind.service Add the following options to the smb.conf: log level = 10 debug pid = true max log size = 0 Log level 10 gives a lot of information but if we want to see network packet dumps, we can use log level 50. This also dumps request/response packets to /tmp. To only increase the logs for certain debug classes you can do the following: log level = 1 winbind:10 This only sets log level 10 for winbind processes. Better get full logs if you're not sure what you're doing. After rasing the debug level remove old logs and start winbind again: # rm -f /var/log/samba/log.* # systemd start winbind.service Reproduce the issue, for example: # date; wbinfo --name-to-sid <name>; date **NOTE**: The `date` command is important because it will print the timestamp when the command got executed. It makes it a lot easier for a developer to find the related lines in the log files! This is an extreme time saver. Collect log files from `/var/log/samba/log.*` Winbind in an AD Environment ---------------------------- **Check DNS** For Active Directory, a correctly set up DNS infrastructure is essential! Active Directory uses Kerberos and has a fallback to NTLM. If the DNS setup doesn't work it will always fallback to NTLM increasing the load on DCs. Often customers don't notice that DNS is not set up probably. A client normally should be able to reach its DC server via DNS and be able to lookup SRV records to find the Kerberos KDC. **Get information about current domain** Winbind talks and gets all the information it has been joined to. That's the domain or realm the machine account has been created. You can get some information about the DC it is joined to with the following command: # net primarytrust dumpinfo **Get information about trusts** This the following command will list the trusted domain and the type of trust and direction the domain has winbind is joined to: # wbinfo --trusted-domains --verbose **How to find out in which groups a user is member of** This will not be able to produce a correct membership information: root:~ # id <user> Log in as the domain user and run: DOMAIN\user:~ # id **The above is what you should teach a customer!** For debugging you can authenticate a user also as root with either NTLM (`wbinfo -a`) or Kerberos (`wbinfo -K`): Example: root:~ # wbfino -a DOMAIN\user The information are now cached in the samlogon cache. You can inspect the cache using: net cache samlogon list net cache samlogon show <sid> Another way is using S4U2Self impersonation using the machine account credentials. net ads kerberos pac dump impersonate=user@REALM -P S4U2Self is a privileged information. Often machine accounts lack permissions on the AD DC side to perform it, so the impersonation might fail. Samba File Server ----------------- **Enable debug logs on the fly** smbcontrol <pid> debug 10 Samba Pool Usage ---------------- Samba use talloc as it's memory allocater. This is a hierarchy-based allocator with destructors. As of this we are able to dump the memory hierachy with the following command: smbcontrol smbd pool-usage If you experience a memory leak in a daemon you can use the following to track it down: smbcontrol <pid> pool-usage > start-pool-usage.log wait till the memory increases and then do: smbcontrol <pid> pool-usage > current-pool-usage.log diff -u start-pool-usage.log current-pool-usage.log This works wit smbd and winbindd! Valgrind -------- valgrind --tool=memcheck -v --num-callers=20 --track-origins=yes --leak-check=full --log-file=valgrind.winbind.log /usr/sbin/winbindd --foreground Network traces =============== If you're requrested to create network traces make sure they match to the Samba log files you provide. Always make sure to capture only one problem per network trace file. This makes it easier to understand the problem. You can use tcpdump to create the network traces using the following command: tcpdump -n -i eth0 -s 0 -w samba-problem-description.pcap ### Limiting to an IP address Network traces should just capture which is going on between client(s) and server. So limiting the captures makes it easier to process the huge amount of information. You can do that using the ip of the host you're connecting too or which is connecting to the server: tcpdump -n -i eth0 -s 0 -w samba-problem-description.pcap host <ip> ### SMB client traffic To limit the traffic only to smb client traffic, use the following command: tcpdump -n -i eth0 -s 0 -w smb.pcap host <ip> and port 445 ###### tags: `samba` `debug`