Here are my notes taken on a training I’ve had recently on Linux security analysis topic. These notes are specifically for Ubuntu-based Linux distros, especially Linux Mint.
Table of contents:
Logs
How logs are created?
- Directly – a process is writing to a file in the /var/log directory. This is the way Apache server or Nginx works.
- Using kernel – a process is passing log messages to the Linux kernel, then some demon receive them from the kernel and stores them somehow according to its configuration.
- In the past this mechanism was called “syslog”. The chain was: process => kernel => syslog => file…
- Nowadays this is called “journald”. The chain is: process => kernel => journald => some demon (if any, it can be syslog) => …
- Journald is writing log data to its own binary files. Storage defined in file /etc/systemd/journald.conf (property “Storage”). These binary files are stored in subdirectories of the /var/log/journal directory.
- Then journald passes log data to some defined demon (if any), which does whatever its configuration says.
Viewing logs
To view logs stored by journald use command: journalctl. Interacting with this program is similar to using less. Among others:
- Press “/” or “?” to search. “/” is for searching forward and “?” is for searching backward. Enter a text to search and press Enter.
- Arrows, Page Up, Page Down buttons work as expected. The End button moves you to the very latest log entry.
- Press “q” to exit.
Please note that running this command as a plain user may give limited results. All logs are available for sure to the root user.
The journalctl command can be used with grep. Example: journalctl | grep sshd
The journalctl command can accept command line paremeters. Examples:
- selecting logs belonging to a single service:
journalctl -u ssh- A service name is the same as used with
systemctlcommand. This is “systemd unit” name.
- A service name is the same as used with
- selecting logs since some time, for example for last hour:
journalctl --since -1h
Viewing ssh login attempts
To see successful ssh login attempts:
journalctl -u ssh | grep Accepted
To see failed ssh login attempts:
journalctl -u ssh | grep “Invalid\|Disconnected”
Some interesting log files
- /var/log/auth.log – authentication
- /var/log/kern.log – kernel (diagnostics)
As on Ubuntu/Linux Mint rsyslog demon is used. Its configuration is in the file /etc/rsyslog.conf and in configuration files in the /etc/rsyslog.d directory. On Linux Mint 20.2 the file /etc/rsyslog.conf contains directive to include all *.conf files from the mentioned directory. This configuration defines (among other things) which extra files collect given categories of log messages.
Extra logs of login events
Login events are recorded in additional binary log files: /var/log/wtmp and /var/log/btmp. You use following commands to view them:
last– shows the list of successful login events and reboot events.lastb– shows the list of unsuccessful login attempts
Auditd
Audit demon. By default present on RedHat-based Linuxes.
If present in the system, its log file is: /var/log/auditd/auditd.log
ausearch – a program to view auditd logs
SELinux
NOTE: phrase avc: denied denotes violation of some SELinux policy.
AppArmour
This is installed on default on Debian/Ubuntu-based Linuxes. It is logging into dmesg and Kernel logs.
NOTE: the phrase apparmour=”DENIED” denotes AppArmour rule violation.
Other logs
Some applications are writing logs on its own and these log files doesn’t have to be located in /var/log directory.
Some option to find them: find / -iname '*log' 2>&1
Some logs files doesn’t contain the word “log” in its file name, so it is worth to check “err” and “out” as well. Example: catalina.out.
User history
In general it is worth to review dot-files in user home directory if there is a suspicion this user done something (or rather someone done something under this user account): ls -la
User history files are written in files in user’a home directory, so it can be affected by a user. If unaffected, it can be helpful to analyse what was done. Example: .bash_history
NOTE: for user accounts created for services like PostgreSQL, MySQL, Nginx, Apache, etc. home directories are not within /home directory. The system file /etc/passwd contains location of home directories for local accounts.
Vim
File .viminfo shows which files were opened. It may contain history of commands (like replacing) user executed in Vim.
Less
File .lesshst shows what terms a user used for searching.
Others
Directories .cache and .config under user’s home directory may contain traces of running some programs. Event a time of the last modification of some file can be helpful.
Access
Accounts with passwords
Look into /etc/shadow. The second column contains ‘*’ or ‘!’, if password is not set. Service accounts should not have a password.
Who is “root”?
Root priviliges are verified based on “uid”, not on user name. Uid=0 denotes a root user account. Any user account may have set uid=0.
SSH authorized keys
NOTE: there are 2 files relevant in user’s home directory: .ssh/authorized_keys and .ssh/authorized_keys2!
Sudo
The file /etc/sudoers contains configuration which tells which user can use sudo command and how it can be used. Some of its configuration can exists as files under /etc/sudoers.d directory.
NOTE: sudo-access to any reading program is dangerous, as it allows to review fileas like /etc/shadow. Moreover the less command allows file edition, and some editors, like nano allow to execute any command!
File access mode
Don’t use file access mode “777” to solve problems.
Executables belonging to root should not have mode “u+s” or “g+s”. This is so called “setuid/setgid”. Such executables are executed with file owner (or group) privileges, no matter who run it.
NOTE: many executables correctly have this mode set. Examples: passwd, sudo.
You can find such files using command:
find / -perm /u+s,g+s
ACL
Access control list – extention of standard Unix file access model. ACLs allows to assign more groups to a file or to add access to individual users. When security breach is considered, one should check for which files in the system ACLs are configured. When files are listed with the ls -l command then files with ACL have “+” sign at the end. Example: -rw-rw-r--+
getfacl FILE – shows ACL summary for given file.
System state
Sessions
Commands to check who is logged in the system:
wwho– very similar to the above, with less detailsloginctl– more powerful than above. Without params it just lists sessions (live w or who). With params user-status UID it shows processes run by the given user.
Processes
Commands to view processes of all users:
ps auxps -ef– shows PID and parent process PID, which is not shown by the above.ps axjf– shows simple process treespstree– shows process trees, but without process details
NOTE: some of these commands are redundant, but it may be relevant on a system that was compromised as some tools may have been replaced.
pgrep
This command lists all processes of a given user with names/descriptions: pgrep -u UID -a
pkill
The pkill command allows to kill all processes of a given user.
/proc filesystem
Subdirectories and files in the /proc directory are representing system state. Linux kernel presents here information about running processes and information about system parameters/settings. These are not real files.
Among others, subdirectories with names being numbers are representing processes – these numbers are PIDs. Inside each such directory /proc/NNN there are files representing certain information about a process:
exe– link to an executable file this process is executingcwd– link to a current directory of this processcmdline– should hold a complete command line, but processes have the freedom to override the memory region and break assumptions about the contents or format of this file!fd– a directory with links representing open file descriptors. Links to files named like/dev/pts/2reference just console.
NOTE: vim doesn’t hold a file descriptor open all the time if it doesn’t have to. So a file “opened” for reading in vim may not be listed here.- and many others…
NOTE: every process in Linux can set its own name/description and it is a fully legal operation. An attacker can use it to “hide” on a process list, for example by naming a process like [kworker]…. So watch carefully process owners.
List of all open files
lsof – the command shows all open files, by which process and which user. You may need to install it from a repository. Always run the command with “-n” parameter to skip resolving domain names for files representing open network connections.
This command can list opened connections as well!
Not-existing files
A process in Linux can write a file that seems to not exist any more. A file opened by some process may be deleted by another process or user. This doesn’t break using file’s descriptor by the first process. The lsof command shows such cases with annotation “(deleted)”.
An attacker can try to hide his files that way. Below command lists all such cases:
lsof -n | grep deleted
In such cases the df command correctly reports disk space taken by such files (deleted, but still used by processes that opened it before deletion).
Affecting commands execution
Aliases
Run command alias to see what aliases are defined. An attacker can potentially defined aliases to alter commands like ps to hide his presence. Do we really run ps program after executing ps command?
Environment variables
PATH
Very relevant variable. Can affect what we run when executing commands. The variable can depend on:
- global configuration files in Linux
/etc/environment,/etc/bashrc(this is for bash shell, other shells can have their own global configuration files),/etc/profile - configuration files in user’s home directory:
.profile, .bashrc, .bash_profile(last two relevant for bash only)
NOTE: the configuration file /etc/environment defines environment variables present for every process.
LD_PRELOAD
Even more relevant environment variable! This variable tells programs where they should look for libraries. Attacker can replace important system libraries to hide his activity.
Similar meaning have configuration files: /etc/ld.so.conf and files inside directory /etc/ld.so.conf.d. These files contain paths, where programs are looking for libraries.
There is also a system cache in Linux that is used to load libraries. You can preview the case with the command:
ldconfig -p
How to verify if system programs are legitimate?
If in doubt if some of system executables is original, not replaced by an attacker:
- Locate the executable file:
which ps
In my case it gave /bin/ps, but it could be also /usr/bin/ps on some other Linux distro - Find a package that contains it:
dpkg-query -S /bin/ps
If this gives nothing retry with /usr/bin/ps instead of /bin/ps, as some Linux distros are dropping the old Unix tradition of putting some executables in /bin and some in /usr/bin.
In my case the command gave package name procps - Finally verify consistency of all executables comming from that package:
dpkg --verify procps
If nothing is printed it means it’s OK.
Verification of consistency of all packages installed in the system
dpkg --verify
Files
Directories to check
Look into these directories for some unusual stuff:
- /tmp, /var/tmp – temporary folders
- /run, /var/run – another kind of temporary folders
- /dev/shm – this is a RAM-disk! Usually empty. You can see it’s usage with the command
df -h
As a root, I cannot remove some file
Check attributes of such file with command: lsattr FILE
The letter “i” in the output denotes immutable attribute. If a file has immutable attribute set, than it cannot be deleted. To do so, you need to remove immutable attribute first: chattr -i FILE
File timestamps
The command ls -l shows timestamp of file’s last modification. But depending on a file system, a single file can have many timestamps attached. To show all such timestamps use command:
stat FILE
This can show following timestamps:
- Access – last read
- Modify – last change of file’s content
- Change – last change of file’s attributes, like owner, access mode
- Birth – file’s creation
WARNING: you cannot assume all these timestamps to be valid! An attacker could modify it. Or updating these timestamps could be turned off for a given filesystem!
This can be checked be reviewing output of the mount command. If a filesystem was mounted with “noatime” parameter then it means last access timestamps are not updated. The parameter “nomtime” means last modification timestamps are not updated. The parameter “relatime” means that last access timestamps are not updated in some cases. Refer to “mount” manual (man mount) for details.
NOTE: for some time (since 2009?) in Linux “relatime” is a default mount parameter due to efficiency/power saving reasons.
TBC…
The Art of Code 