Audit Logging

Audit Logging is a feature that allows an audit log of events on selected jBASE data file(s). It was introduced in jBASE 5.5.

The output format is flexible with the audit logs being written in either Multi Value format, Comma Separated Value format, Tab Separated Format, or as a JSON string.

The output can be directed to a number of different location types. The output location can be a directory, in which case a flat OS file is created, one line being added to the file per audit log entry, one file per day. The output location can be a simple OS file, in which case we write one line to the file per audit log entry, with no limitation. The output location can also be a named pipe, which involves having a user-supplied reader process.

1. Installation and configuration.

Audit Logging is an integral feature of jBASE and needs no installation as such. However there are two steps required in order to audit log your first file.

1.1. Step 1. Create an @AUDIT file.

The @AUDIT file is any regular jBASE data file. For the moment, we will create it as an ordinary J4 file, but as you will see later, there is a new file type which provides more functionality.

Create the @AUDIT file. This should be somewhere in your file path where every user can open the file.

jsh jbasedev /tmp -->create-file @AUDIT 1,1 97,1
[ 417 ] File @AUDIT]D created , type = J4
[ 417 ]File @AUDIT created , type = J4

As with most regular jBASE files, you can perform read and write operations (subject to OS permissions). So creating the @AUDIT file as a J4 file means you can do operations like clear-file on it. The new special file type described later, TYPE=AUDIT, only allows read operations.

1.2. Step 2. Mark a file as requiring auditing.

jsh jbasedev /tmp -->jchmod +M ACCOUNTS

Example #1 of audit logging.

You are now ready to commence auditing on the ACCOUNTS file, and each event will be stored in the @AUDIT file. Below is an example.

jsh jbasedev /tmp -->ED ACCOUNTS GREG
GREG
TOP
001 COOPER
. r/COOPER/SMITH
001 SMITH
. FI 
Record 'GREG' written to file 'ACCOUNTS'
jsh jbasedev /tmp --> ED @AUDIT \* 
63975_08C0AE7C2*1
TOP
. p
TOP
001
002 2
003 63975_08C0AE7C2
004 1458343438
005 1
006 6
007 63975
008 jbasedev
009 jbasedev
010
011 GREG
012 /home/jbasedev/apps/ACCOUNTS
013 edit.b
014 239
BOTTOM
. EX
Record '63975_08C0AE7C2*1' exited from file '@AUDIT'

Although 2 records will be written to @AUDIT, only 1 is shown. The layout of the @AUDIT record is given in Appendix A. The item id comprises a unique ID which uniquely identifies that process followed by an incrementing integer.

Because the @AUDIT file is a hashed file, then “ED @AUDIT *” will return the items in hashed order. If you want to edit the items in any particular order, you need to SELECT or SORT first.

You can now mark as many files as you like with audit logging and each event to each file will get logged to the @AUDIT file.

2. The AUDIT type file

We have shown how a simple audit logging process can be initiated using a standard J4 file as the @AUDIT file. For jBASE 5.5 there is a new file type designed especially for audit logging and we will look at that now. This new type gives greater capabilities than using other file types.

Create a @AUDIT file like this

jsh jbasedev /tmp -->DELETE-FILE @AUDIT
jsh jbasedev /tmp -->CREATE-FILE @AUDIT TYPE=AUDIT OUTPUT=/home/audit FORMAT=MV
[ 417 ] File @AUDIT]D created , type = J4
[ 417 ] File @AUDIT created , type = AUDIT

The DICT section is created as a regular J4 file, but the DATA section is of the new type AUDIT. Note that the DICT is filled with suitable entries.

jsh jbasedev /tmp -->COUNT DICT @AUDIT

 19 Records counted

What we have specified here is is that the output of audit logging will be to flat OS files in the /home/audit directory. You need to be careful here of OS permissions, the CREATE-FILE should be allowed to create /home/audit if it doesn't already exist, and application processes need permission to update files in the directory.

Example #2 of audit logging.

jsh jbasedev /tmp -->ED ACCOUNTS GREG
GREG
TOP
001 SMITH
. R/SMITH/JONES
001 JONES
. FI
Record 'GREG' written to file 'ACCOUNTS'
jsh jbasedev /tmp -->ED @AUDIT \* 
audit_2016_03_18_00_00_00*0*103
TOP
 . p
TOP
001
002 1
003 62585_08B4845A4
004 1458330682
005 1
006 6
007 62585
008 jbasedev
009 jbasedev
010
011 GREG
012 /home/jbasedev/apps/ACCOUNTS
013 edit.b
014 612
BOTTOM
EX
Record 'audit_2016_03_18_00_00_00*0*103' exited from file '@AUDIT'

Items in the AUDIT file are stored sequentially and are guaranteed to be in the same order that they were written to the audit log. The item id is an internal representation of the item and should have no connotations associated with it.

Even though these items are simply lines in an OS file, you can still use jBASE commands to view them. For example

jsh jbasedev /tmp -->LIST @AUDIT

PAGE    1                                        09:15:51  29 FEB 2016
Event.....    Date.......    Time....    Port...    jBASE.....
                                         Number     login

READ          29 FEB 2016    09:11:28          1    jbasedev
WRITE         29 FEB 2016    09:11:29          1    jbasedev

2 Records Listed

This output uses the default DICT item “1”. There is also an “ALL” dictionary item created when the file is created

Output is to a flat OS file in the directory specified during CREATE-FILE. Below is an example of viewing the file via OS commands

trillian-~/tmp: cd /home/audit
trillian-~/home/audit: ls -l
total 4
-rwxrwxrwx 1 jbasedev users 168 Feb 29 09:11 audit_2016_03_22_00_00_00
trillian-~/home/audit: cat audit_2016_03_22_00_00_00
^1^62585_08B4845A4^1458330682^1^6^62585^jbasedev^jbasedevGREG^/home/jbasedev/apps/ACCOUNTS^edit.b^612 
^2^62585_08B4845A4^1458330683^2^6^62585^jbasedev^jbasedevGREG^/home/jbasedev/apps/ACCOUNTS^edit.b^239

The file name is in the format audit_yyyy_mm_dd_hh_mm_ss with the date/time in UTC representing the rollover time. Each OS file will contain by default a single day of updates and when the day rolls to a new day, a new file will be created. As you will see later, that rollover interval can be changed.

Limitations of the AUDIT file type.

Because of the sensitive nature of the audit information, not all database operations will work when using a file of type AUDIT. It is basically a read-only file from a user point of view. Hence SORT, EDIT (without FI), SELECT, LIST , OPEN, READ, READNEXT will all work, DELETE-FILE, CLEAR-FILE and DELETE will not work.

The CREATE-FILE command

While the @AUDIT file can be any support jBASE file, the new file type of AUDIT will work best. The CREATE-FILE command supports some options for this file type. The full format is

CREATE-FILE filename TYPE=AUDIT

OUTPUT=directory_name shows where we place the output log files. It can also define a named pipe (see later). The default is $JBCGLOBALDIR/audit

FORMAT=MV|CSV|TAB|JSON. Defines what the output format is. The default is MV

INTERVAL=nnn. This defines the interval in minutes between switching output log files. The default is 1440, which is one full day. The minimum value is 5.

VERBOSE=TRUE|FALSE|YES|NO. Defines if the audit log is created in verbose mode. In effect this means that a jQL or jSQL statement will have each individual READ logged.

0.1. Changing output options.

You can change the auditing options by editing the item @AUDIT_CONFIG in the DICT @AUDIT file. Below is an example of changing the output type from MV to CSV.

Note: These changes will not take effect for running processes. To pick up the changes, a process needs to exit and re-start.

trillian-~/tmp: ED DICT @AUDIT @AUDIT_CONFIG
@AUDIT_CONFIG 
TOP
. p
TOP
001 /home/audit
002 MV
003
004 1440
BOTTOM
. 2
002 MV
. r/MV/CSV
002 CSV
. FI
Record '@AUDIT_CONFIG' written to file 'DICT @AUDIT'

The format of the definition record is as follows.

· Attribute <1> defines the output directory or output named pipe

· Attribute <2> defines the output format, MV, CSV, TAB or JSON

· Attribute <3> defines any options such as V for verbose logging during jQL or jSQL.

· Attribute <4> defines the output file interval in minutes (default 1 day)

If we now edit another audited file, we can see the difference in output.

trillian-~/tmp: ED ACCOUNTS GREG
GREG 
TOP
 FI 
Record 'GREG' written to file 'ACCOUNTS'
trillian-~/tmp: cat /home/audit/* 
^1^62585_08B4845A4^1458330682^1^6^62585^jbasedev^jbasedevGREG^/home/jbasedev/apps/ACCOUNTS^edit.b^612 
^2^62585_08B4845A4^1458330683^2^6^62585^jbasedev^jbasedevGREG^/home/jbasedev/apps/ACCOUNTS^edit.b^239 
,1,62585_08B4845A4,1458330704,1,6,62704,jbasedev,jbasedev,,GREG,/home/jbasedev/apps/ACCOUNTS,edit.b,612 
,2,62585_08B4845A4,1458330705,2,6,62704,jbasedev,jbasedev,,GREG,/home/jbasedev/apps/ACCOUNTS,edit.b,239

0.2. Output formats.

The default output format is MV, which means the output line has its values delimited by an attribute mark, making it a Multi Value format. Other output formats are shown below. These formats can be speficied during the creation of the @AUDIT file with the FORMAT=xxx option, or by manually editing the DICT as shown above.

0.2.1. MV.

The output format is Multi Value, i.e. each value is separated by an attribute mark. This is the only format that will allow some jBASE commands to understand the format. For example, you can SORT or LIST the file when the output format is MV. If you change the format, then SORT and LIST will only see a single value comprising a single text string.

0.2.2. CSV

The output format is Comma Separated Value. Each value is separated by a comma.

0.2.3. TAB

The output format is TAB. Each value is separated by a TAB.

0.2.4. JSON

The output format is JSON. Each log entry will create a JSON string. For example.

trillian-~/tmp: ED DICT @AUDIT @AUDIT_CONFIG
@AUDIT_CONFIG
TOP
001 /home/audit
002 CSV
003
004 1440
BOTTOM
. 2
002 CSV
.r/CSV/JSON
002 JSON
. FI
Record '@AUDIT_CONFIG' written to file 'DICT @AUDIT'
trillian-~/tmp: ED ACCOUNTS GREG
GREG
TOP
. FI
Record 'GREG' written to file 'ACCOUNTS'
trillian-~/tmp: cat /home/audit/*
^1^62585_08B4845A4^1458330682^1^6^62585^jbasedev^jbasedevGREG^/home/jbasedev/apps/ACCOUNTS^edit.b^612 
^2^62585_08B4845A4^1458330683^2^6^62585^jbasedev^jbasedevGREG^/home/jbasedev/apps/ACCOUNTS^edit.b^239 
,1,62585_08B4845A4,1458330704,1,6,62704,jbasedev,jbasedev,,GREG,/home/jbasedev/apps/ACCOUNTS,edit.b,612 
,2,62585_08B4845A4,1458330705,2,6,62704,jbasedev,jbasedev,,GREG,/home/jbasedev/apps/ACCOUNTS,edit.b,239 {"EVENT":1,"UNIQUEID":"63675_08BE52A51","UTC":1458340964,"SEQID":1,"PORT":6,"PID":63675,"OSNAME":"jbasedev","JBNAME":"jbasedev","IP":"","EVENTOP":"GREG","FILEPATH":"\/home\/jbasedev\/apps\/ACCOUNTS","SOURCE":"edit.b","LINENO":612} {"EVENT":2,"UNIQUEID":"63675_08BE52A51","UTC":1458340967,"SEQID":2,"PORT":6,"PID":63675,"OSNAME":"jbasedev","JBNAME":"jbasedev","IP":"","EVENTOP":"GREG","FILEPATH":"\/home\/jbasedev\/apps\/ACCOUNTS","SOURCE":"edit.b","LINENO":239}

The JSON is output in compact format i.e. no white space. If we “beautify” one line of JSON, we get a clearer picture of the output.

{
   "EVENT": 1,
   "UNIQUEID": "63675_08BE52A51",
   "UTC": 1458340964,
   "SEQID": 1,
   "PORT": 6,
   "PID": 63675,
   "OSNAME": "jbasedev",
   "JBNAME": "jbasedev",
   "IP": "",
   "EVENTOP": "GREG",
   "FILEPATH": "\/home\/jbasedev\/apps\/ACCOUNTS",
   "SOURCE": "edit.b",
   "LINENO": 612
}

1. Appendix A. The Audit record format.

Attr No

DICT name

Description

2

EVENT

The event command as follows 1 = A record was READ from the file 2 = A record was WRITTEN to a file 3 = A record was DELETED from a file 4 = The file was CLEAR-FILE'd 5 = The file was DELETE-FILE'd 6 = A jQL or jSQL query was made on the file.

3

UNIQUE

A string that uniquely identifies the process. This allows you to separate updates from other processes that might share the same process ID or Port Number.

4

UTC

The time of the operation in UTC (Universal Co-ordinated Time) format. You can convert this to a local TIME using OCONV(UTC,”U0FF0”) and to a local DATE using OCONV(UTC,”U0FF1”). There are also DICT entries for TIME and DATE that do this.

5

SEQID

An integer that is an incrementing value which defines the relative audit log update for that process.

6

PORT

The jBASE port number.

7

PID

The operating system process ID.

8

OSNAME

The account name that the process is logged on to as supplied by the operating syts

9

JBNAME

The jBASE account name

10

IP

The internet address or host name from where the user has connected.

11

EVENTOP

The operand to the event. This can be.. (a) The item-id of the update in the file that caused the audit log record. (b) The jQL or jSQL query made against the file.

12

FILEPATH

The full file path of the file where the update occurred. This can be split into its component parts of DIRECTORY using OCONV(FILEPATH,”U0FF2”) and FILE using OCONV(FILEPATH,”U0FF3”). There are DICT entries for DIRECTORY and FILE that do this.

13

SOURCE

The name of the jBC source where the update occurred.

14

LINENO

The line number within the jBC source where the update occurred.

2. Appendix B. Output to pipes.

When the @AUDIT file is created of type AUDIT, the default option for OUTPUT=xxx is to specify a directory name at xxx. In this directory, a series of files will be created, one file per day, containing the audit logs , one line per log.

Instead of providing the name of a directory, you can also provide the name of a FIFO pipe. Under this scheme, the audit log will be written to a named pipe, and a different process will read the information from the pipe.

Using named pipes allows a huge amount of flexibility. The reading process can be third party applications, it could encrypt the data and pipe it elsewhere, it can provide a security layer between the jBASE process and the rest of the system.

With named pipes, jBASE can only write the audit log to the named pipe. You cannot read from the named pipe, so other jBASE commands such as LIST @AUDIT or ED @AUDIT * will not work.

This is an example of using named pipes on Unix. Note that on some variants of Unix, you need to use the 'mkpipe <filename>' command instead of 'mknod <filename> p'

2.1. 1) Create a named pipe

jbase-~/tmp: mknod mypipe p
jbase-~/tmp: ls -l
total 0 prw-rw-r-- 1 fb fb 0 Mar  4 15:04 mypipe

2.2. 2) Create the @AUDIT file to reference this pipe. (Alternatively you could edit the DICT @AUDIT @AUDIT_CONFIG attribute 1 to point to the pipe).

jbase-~/tmp: CREATE-FILE @AUDIT TYPE=AUDIT OUTPUT=`pwd`/mypipe FORMAT=JSON
[ 417 ] File @AUDIT]D created , type = J4
[ 417 ] File @AUDIT created , type = AUDIT

2.3. 3) Start a reading process.

jbase-~/tmp: tail -f mypipe

2.4. 4) From a different shell, create some audit log entries.

jbase-~/tmp: create-file DUMMY 1,1 23,1
[ 417 ] File DUMMY]D created , type = J4
[ 417 ] File DUMMY created , type = J4
jbase-~/tmp: jchmod +M DUMMY
jbase-~/tmp: ED DUMMY ITEMID
New record
ITEMID
TOP
. FI
Record 'ITEMID' written to file 'DUMMY'

2.5. 5) Going back to the original process where we started the reading process in stage (3), we can see a record has been written to the named pipe and read by the reading process like this.

jbase-~/tmp: tail -f mypipe

{"EVENT":1,"UNIQUEID":"63675_08BE52A51","UTC":1458340964,"SEQID":1,"PORT":6,"PID":63675,"OSNAME":"jbasedev","JBNAME":"jbasedev","IP":"","EVENTOP":"GREG","FILEPATH":"\/home\/jbasedev\/apps\/ACCOUNTS","SOURCE":"edit.b","LINENO":239}