Event monitoring git clone and git push on the local server Networks

Sometimes there is a desire to monitor the local GIT server on the subject who (first and last name from LDAP), what project and from where(ip address) tends or pushes.

After reviewing documentation, it became clear that this functionality out of the box no, actually it is, but in the paid version Networks. Under the cut is my experience of monitoring implementation.

My recipe does not claim universality, I hope many will be useful as a starting point.

Description of my configuration:

we Have installed "Community Edition 10.0.3 Networks", user authentication happens over LDAP, clone and push they do using a single SSH uchetku 'git'. In fact it is the standard configuration for a more or less large company.

Every git clone and git push, in file '/var/log/auth.log' a message appears stating that the user "git" logged in to the system with such 'fingerprint'

cat auth.log

Oct 17 12:41:11 Networks-test sshd[25931]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.111.24 user=git
Oct 17 12:41:13 Networks-test sshd[25931]: Accepted publickey for git from 192.168.111.24 port 55268 ssh2: RSA 63:3b:ca:8d:23:2f:d2:0c:40:ce:4d:2e:b1:2e:5f:7c
Oct 17 12:41:13 Networks-test sshd[25931]: pam_unix(sshd:session): session opened for user git by (uid=0)

Then, in file '/var/log/networks/networks-shell networks shell.log' the message indicates that the user with that 'key' cloned or spousal such projects.

cat networks-shell.log

I, [2017-10-19T16:17:32.006429 #1115] INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.02417
I, [2017-10-19T16:17:32.006954 #1115] INFO -- : networks-shell: executing git command < git-upload-pack /var/opt/networks/git-data/repositories/tech/ansible-server.git> for user with key key-1030.

Finally, in file '/var/log/auth.log' a message appears stating that the user git is logged off:

cat auth.log

Oct 17 12:41:13 Networks-test sshd[25944]: Received disconnect from 192.168.111.24: 11: disconnected by user
Oct 17 12:41:13 Networks-test sshd[25931]: pam_unix(sshd:session): session closed for user git

After examining all of the tables in the database, it became clear that the mythical 'key', which is in the logs GitLab'a, it is possible to find a sane user name and 'fingerprint'.

Because in the log lines appear in a strict order, the last entry in '/var/log/auth.log' we 'fingerprint' will contain the IP address of the user. Even if messages from different users are not recorded strictly in order, will not fail because according to 'fingerprint' — 'IP address' is searched from the end.

the user Name is in table 'identities', 'user_id', which can be found in the table 'keys' 'key' which we see in the log file 'networks-shell.log'.

SELECT extern_uid FROM identities WHERE user_id = (SELECT user_id FROM keys WHERE id = 1030);

In the table 'keys' 'key' is the 'fingerprint'

SELECT fingerprint FROM keys WHERE id = 1030;

the SQL query to find the user's real name:

SELECT extern_uid FROM identities WHERE user_id = (SELECT user_id FROM keys WHERE id = 1030);

Ooriginal these data were invented algorithm how to find the user name, ip address, project and action that he made.

1. parsim using tail-f the new lines in the log Gita,
2. once you come across the line sootvetstvuyushie the regular season we go into this database and are looking for poluchenia 'key' username and its 'fingerprint',
3. 'fingerprint' in the 'auth.log' looking for ip address of the user and take the most recent entry.
The script is written in Python(Python, so forgive me fans).
This is my first experience of writing anything in Python. I would appreciate constructive criticism and recommendations.

It requires the following libraries and modules:
python-tail To track the new lines in the file 'networks-shell.log'
psycopg To work with PostgreSQL
gelfHandler To send messages to the GrayLog server
The script is quite big, he is able to pull from the configuration file 'networks.rb' this to connect to a PostgreSQL database, checks the log files of SSH and 'networks-shell.log' can write to a file and send it to GreyLog the server writes logs the error to a file and to the console. Any of these options can be enabled or disabled.

Script

#!/usr/bin/python
# -*- coding: utf-8 -*-

"""
Brief description:
1. In '/var/log/networks/networks-shell networks shell.log' looking for the name of the project, action and 'key' of the user.
2. In the database looking for real name of the user and his 'fingerprint' previously disposed of along with the 'key'.
3. In "/var/log/auth.log" looking for user IP in the last line, we 'fingerprint'.

Line of networks-shell.log
I, [2017-10-17T12:19:56.526131 #21521] INFO -- : networks-shell: executing git command < git-receive-pack /var/opt/networks/git-data/repositories/web/markets.git> for user with key key-11.

SQL query to find 'fingerprint'
SELECT fingerprint FROM keys WHERE id = 11;

SQL query to find the user's real name:
SELECT extern_uid FROM identities WHERE user_id = (SELECT user_id FROM keys WHERE id = 11);
"""

# Import modules and libraries
from gelfHandler import GelfHandler
import logging
import psycopg2
import re
import os
import tail
import subprocess
import sys
import datetime

time=str(datetime.datetime.now())

# Enable/turn on debug-on or off - off
debug = 'on'
#debug = "


# Enable/enable display the full path to the file GIT. The file name matches the project name.
#pach_git_all = 'on'
pach_git_all = 'off'


# Path to your log files:
log_file_GuiLab = '/var/log/networks/networks-shell networks shell.log'
log_file_SSH = '/var/log/auth.log'


# Parameters for connecting to SQL. For the automatic determination of the parameters, you must specify the path to the configuration file of Galaba, or you can specify its parameters.
gitlab_rb = '/etc/networks networks.rb'
#gitlab_rb = "

sql_db = 'networks'
sql_user = 'python_user'
sql_password = 'qwer123'
sql_host = '127.0.0.1'
sql_port = '5432'


# Parameters Greylag server:
out_GrayLog='yes'
#out_GrayLog="
logger = logging.getLogger()
gelfHandler = GelfHandler(
host='192.168.250.145',
port=6514,
protocol='UDP',
facility='Python_parsing_log_file_GitLab'
)
logger.addHandler(gelfHandler)

#options record the results in the logfile
out_log_file_name='/home/viktor/pars_log_GitLab.log'
#out_log_file_name="

# Function to handle the error if no log files
def funk_error_file(log_file):
print "File ", log_file, "not found!!!"
out_log_file = open(out_log_file_name, 'a')
out_log_file.write(time);
out_log_file.write(" ");
out_log_file.write("File ");
out_log_file.write(log_file);
out_log_file.write(" not found!!!");
out_log_file.close()
sys.exit()


# The whole script is one function tail, she starts to run when the file "networks-shell.log" there is a new line
def funk_pars_gitlab_shell(string_gitlab_shell):

# Local variables:
action = 'action'
project = 'project'
user_key = 0
username = 'username'
time_ssh = 'time_ssh'
host_name = 'host_name'
id_ssh_log_message = 0
usr_name_git = 'usr_name_git'
ip_address = 'ip_address'
fingerprint_log_ssh = 'fingerprint_log_ssh'
fingerprint = 'fingerprint'
time_git = 'time_git'
id_git_log_message = 'id_git_log_message'


# Check the regular season, so a new line match the template, the result is an array with three elements, including the empty(pre-check for full and short file path)
if pach_git_all == 'on':
regexp_string_gitlab_shell = re.findall(r'^.*\[([^ ]+)\.[\d]+\s\#([^ ]+)\]\s.*<([^ ]*)\s ([^ ]*) > \s{1,}for user with key\s{1,}key-([^ ]*)\.$', string_gitlab_shell)
elif pach_git_all == 'off':
regexp_string_gitlab_shell = re.findall(r'^.*\[([^ ]+)\.[\d]+\s\#([^ ]+)\]\s.*<([^ ]*)\s.+repositories/([^ ]*)\.git>\s{1,}for user with key\s{1,}key-([^ ]*)\.$', string_gitlab_shell)


# Pull out of the resulting array
for arr_string__networks_in shell regexp_string_gitlab_shell:
time_git = arr_string__networks_shell[0] # Time of recording messages in a log file GIT
id_git_log_message = arr_string__networks_shell[1] # message ID in the log file GIT
action = arr_string__networks_shell[2] # Action ie clone, push or something else.
project = arr_string__networks_shell[3] # Project in which you were thinking or pushil
user_key = arr_string__networks_shell[4] # a user's misappropriation indentificator him gitlerom.

# Check whether there is new data in the variable in order not to pull once again the database and not send the void
if action != 'action':
# Connect to the database
connect = psycopg2.connect(database=sql_db, user=sql_user, password=sql_password, host=sql_host, port=sql_port)

# Open the cursor (i.e. podklyuchaem to the database)
curs = connect.cursor()

# Generated by variable substitution in the SQL query. In the query for user ID it will look for user_id by which the hedgehog will find a sane user name
sql_string_find_username="""SELECT extern_uid FROM identities WHERE user_id = (SELECT user_id FROM keys WHERE id = %s);""" %user_key
curs.execute(sql_string_find_username) # the query itself


# Generated by variable substitution in the SQL query. In the query by user ID will look for his fingerprint.
sql_string_find_fingerprint="""SELECT fingerprint FROM keys WHERE id = %s;""" %user_key
curs.execute(sql_string_find_fingerprint) # the query itself
sql_string_fingerprint = curs.fetchall() # Array with resultam request


# Close database connection
connect.close()

# The following two loops invoked to parse the returned arrays from SQL
for fingerprint_array in sql_string_fingerprint:
fingerprint = fingerprint_array[0]

for username_array in string_external_uid:
username = re.findall(r'^.*uid=([-\w\._]+)', username_array[0])
username = username[0]

# Generated variable and do a system  call  to retrieve the last lines of the log file auth.with the received log of SQL fingerprint.
command = """grep '%s' %s | tail-n 1""" %(fingerprint, log_file_SSH)

p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
str_line = str(line)
retval = p.wait()

# Parse found strochku of auth.log with the help of the regular season on time, IP address and other not very useful data.(server name, message ID, SSH git user)
arr_reg_exp_ssh_info = re.findall(r'^([\w*\s\d\:\s]+)\s([^ ]+)\ssshd\[(\d+)\]:\sAccepted publickey for\s([^ ]+)\sfrom\s([^ ]+)\sport\s[\s\w]+:\sRSA\s([\w\:]+)$', str_line)

# Put the array in a variable
for data_ssh_info in arr_reg_exp_ssh_info:
time_ssh = data_ssh_info[0]
host_name = data_ssh_info[1]
id_ssh_log_message = data_ssh_info[2]
usr_name_git = data_ssh_info[3]
ip_address = data_ssh_info[4]
fingerprint_log_ssh = data_ssh_info[5]

# Change the value of the variable 'action' to a more traditional us values.

if action == 'git-receive-pack':
action = 'push'
elif action == 'git-upload-pack':
action = 'clone'

# Print variables for debugging
if debug:
print'----'
print ' ', 'time_ssh', '\t\t', time_ssh
print ' ', 'time_git', '\t\t', time_git
print ' the ', 'username','\t\t', username
print ' the ', 'action', '\t\t', action
print ' ', 'project', '\t\t', project
print ' ', 'ip_address', '\t\t', ip_address
print ' ', 'user_key', '\t\t', user_key
print ' ', 'host_name', '\t\t', host_name
print ' ', 'id_ssh_log_message','\t', id_ssh_log_message
print ' ', 'id_git_log_message','\t', id_git_log_message
print ' ', 'usr_name_git', '\t', usr_name_git 
print ' ', 'fingerprint_ssh', '\t', fingerprint_log_ssh
print ' ', 'fingerprint_sql', '\t', fingerprint
print'----'
print '\n'


# Compose and send the message to Greylag
if out_GrayLog != 'no':
logger.warning(
'Now message', extra={'gelf_props': {
'title_time_ssh':time_ssh,
'title_time_git':time_git,
'title_username':username,
'title_action':action,
'title_project':project,
'title_ip_address':ip_address,
'title_user_key':user_key,
'title_host_name':host_name,
'title_id_ssh_log_message':id_ssh_log_message,
'title_id_git_log_message':id_git_log_message,
'title_fingerprint':fingerprint
}})


# Creating and sending a message to a file
if out_log_file_name:
out_log_file = open(out_log_file_name, 'a')

out_log_file.write("{time_ssh:");
out_log_file.write(time_ssh);

out_log_file.write("}{time_git:");
out_log_file.write(time_git);

out_log_file.write("}{username:");
out_log_file.write(username);

out_log_file.write("}{action:");
out_log_file.write(action);

out_log_file.write("}{project:");
out_log_file.write(project);

out_log_file.write("}{ip_address:");
out_log_file.write(ip_address);

out_log_file.write("}{user_key:");
out_log_file.write(user_key);

out_log_file.write("}{host_name:");
out_log_file.write(host_name);

out_log_file.write("}{id_ssh_log_message:");
out_log_file.write(id_ssh_log_message);

out_log_file.write("}{id_git_log_message:");
out_log_file.write(id_git_log_message);

out_log_file.write("}{fingerprint:");
out_log_file.write(fingerprint);
out_log_file.write("}");

out_log_file.write("\n");
out_log_file.close()



# Perform the check for the log files, and gracefully  close  the script if they are not(by calling the appropriate function)
if os.path.exists(log_file_GuiLab):
if os.path.exists(log_file_SSH):

if gitlab_rb:
command_searh_db = """grep "gitlab_rails\['db_" %s|tr-s '\\n' '\ '""" %gitlab_rb
p = subprocess.Popen(command_searh_db, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
str_db_line = str(line)
retval = p.wait()
arr_db_con = re.findall(r'^.*db_database\'\]\s\=\s\"([^ ]+)\".*db_username\'\]\s\=\s\"([^ ]+)\".*db_password\'\]\s\=\s\"([^ ]+)\".*db_host\'\]\s\=\s\"([^ ]+)\".*db_port\'\]\s\=\s([^ ]+)\s.*$', str_db_line)
# Put the array in a variable
for data_db_info in arr_db_con:
sql_db = data_db_info[0]
sql_user = data_db_info[1]
sql_password = data_db_info[2]
sql_host = data_db_info[3]
sql_port = data_db_info[4]

#  If  debug is enabled, print a header at start
if debug:
print '\n'
print 'debug ==> on'
print 'Start', (os.path.basename(__file__))
print '\n'
print 'sql_db ==> ', sql_db
print 'sql_user ==> ', sql_user
print 'sql_password ==> ', sql_password
print 'sql_host ==> ', sql_host
print 'sql_port ==> ', sql_port
print'----'

# Run the main function
t = tail.Tail(log_file_GuiLab) # 'log_file_GuiLab' File which will parse the tile
t.register_callback(funk_pars_gitlab_shell) # Call the soma function 'funk_pars_gitlab_shell'
t.follow(s=1) # the Frequency of parsing the file
funk_error_file(log_file_SSH)
funk_error_file(log_file_GuiLab)

As a logical conclusion service for systemd:

cat /lib/systemd/system/pars_log_GitLab.service

[Unit]
Description=Python parsing_log_file_GitLab

# start after the launch of the following services
#After=network.target postgresql.service

# Required services
#Requires=postgresql.service

# Necessary services
#Wants=postgresql.service

[Service]
# Startup
Type=simple

# Restart if it fails
Restart=always

# location of the PID file
PIDFile=/var/run/appname/appname.pid

# Working directory
#WorkingDirectory=/home/username/appname

# User and group under which to run
User=root
Group=root

# This parameter is needed to give the right to the following
#PermissionsStartOnly=true
# ExecStartPre to run BEFORE the application is launched
#ExecStartPre=-/usr/bin/mkdir -p /var/run/appname
#ExecStartPre=/usr/bin/chown-R app_user:app_user_group /var/run/appname

# Run application
ExecStart=/usr/bin/python2 /usr/bin/pars_log_GitLab.py &

# Pause if necessary
TimeoutSec=300

[Install]
WantedBy=multi-user.target

Now the script can be controlled with the following commands:
the

systemctl start pars_log_GitLab.service
systemctl status pars_log_GitLab.service
systemctl stop pars_log_GitLab.service

Don't forget to disable debugging before running.

Thank you for your attention!

Article based on information from habrahabr.ru