Data Leak Forensics

A complete walkthrough of AngelTrack's security features for tracking a data leak, how to use them, and some helpful tips

If you discover that HIPAA-encumbered electronic data has leaked from your organization, AngelTrack has built-in forensics tools to help identify the responsible employee.

Forensics

Tracing a Leaked Paper Run Report

All run reports printed out from AngelTrack show the name of the employee who originally printed the document:

RunReport.PrintedBy

If the printout is found to have escaped from HIPAA-regulated custody, begin your investigation by interviewing the employee whose name appears on the printout. That employee is legally answerable for their usage of HIPAA-protected data, including:

  1. Why did you need to access this data?
  2. Why was it necessary to create a paper copy?
  3. To whom did you transfer the paper copy?
  4. Why did that person need to access this data?
  5. Did that person understand the legal obligations of receiving and handling HIPAA-protected data?

Tracing a Leaked .PDF Run Report or CMS-1500

If you discover a leaked .PDF in the wild, you can quickly identify the person who originally downloaded it from AngelTrack. Simply open the .PDF and look for the "Printed by" field as seen in the screenshot above.

Keep in mind, the .PDF only shows the employee who originally download it. It could've thereafter made its way to other people for any of the following legitimate reasons:

  • ER asks the attending to email over the run report
  • Patient asks for their run history to be emailed over
  • Outside biller asks for the week's run reports to be transferred over for billing
  • Medical director asks for electronic copies of recent critical transports
  • Biller burns many run reports to a data CD in order to respond to an ADR from Medicare
  • Supervisor bulk downloads run reports for offline archiving on data DVDs, stored in a file cabinet at headquarters

You can also inspect the PDF's properties (i.e. its metadata). AngelTrack places the requesting employee's name and IP address, as well as the date requested, into each PDF's metadata fields.

Tracing a Leaked .CSV Export

AngelTrack's many data grids offer to export themselves as .CSV documents, for viewing in Microsoft Excel™ or Google Sheets™. If one of these exported documents goes astray, and later turns up some place it doesn't belong, you can trace it back to its source by means of the forensic signature that AngelTrack placed into the document.

In the final data row of the .CSV document (or of any spreadsheet into which the .CSV was converted), beyond the rightmost column, is the forensic signature. The signature shows who downloaded the data from AngelTrack, and when, and from what IP address.

Tracing a Leaked NEMSIS XML Document

AngelTrack's NEMSIS XML exports contain forensic engravitures which will assist an investigator in tracing a leak back to its originator.

In any NEMSIS XML document, find the following section. It will be repeated at the bottom of every PCR included in the XML. You can locate the data by searching for the word "eRecord.90".

<eCustomResults.ResultsGroup>
     <eCustomResults.01>
          Produced 2021-07-01 17:35:16-00:00 by acme.angeltrack.com for user Jane Doe (ID 16) at 192.168.1.42.
     </eCustomResults.01>
     <eCustomResults.02>eRecord.900</eCustomResults.02>
</eCustomResults.ResultsGroup>
<eCustomResults.ResultsGroup>
     <eCustomResults.01>
          UHJvZHVjZWQgMjAyMS0wNy0wMSAxNzozNToxNi0wMDowMCBieSBhY21lLmFuZ2VsdH
          JhY2suY29tIGZvciB1c2VyIEphbmUgRG9lIChJRCAxNikgYXQgMTkyLjE2OC4xLjQyLg==
     </eCustomResults.01>
     <eCustomResults.02>eRecord.901</eCustomResults.02>
</eCustomResults.ResultsGroup>

There are two pieces of data included. The first, in the "eRecord.900" section, contains a plain-text statement about who requested the data, and when, and from what IP address. The second section, in "eRecord.901", is the very same data, except encoded with base64, in case a very clever wrongdoer has found their own name in the first section and so deleted it from the document.

If the NEMSIS XML data has passed into and then back out of a foreign software app, that software app is responsible for updating these forensic engravitures to indicate which of their users has re-exported the data... but they probably will not, in which case they will still indicate who originally exported the data from AngelTrack.

Forensic engravitures and trauma registries

When AngelTrack uploads data to a state trauma registry, or to the ET3 program, the engravitures will read like this (for New York in this example):

Produced 2021-07-01 17:35:16-00:00 by acme.angeltrack.com for newyork.imagetrendelite.com/Elite/Organizationnewyork/NEMSIS3Ws.

When the trauma registry forwards the data onward to a statewide or to a national repository, it is responsible for updating the engravitures to that effect.

Tracing a Leak of Other Patient Data

PatientAccessJournalIf you must trace a leak of patient data other than run reports (which are conveniently engraved with the name of the employee who produced them), then begin at the Patient Records Journal, available under Settings.

  1. The journal has filters for date range and for specific patient. Use the filters to zoom in as best you can.
  2. Peruse the data using the grid, or Export-Sep-21-2022-12-03-20-42-AM export it to Microsoft Excel™ or Google Sheets™. Build a list of all the dates where the patient record in question was accessed, and by whom.
  3. Visit the actual patient record in AngelTrack, using the Patients List or the Genie. Select the "History" tab to show all the times the patient was serviced by EMS. You should be able to correlate each date of service with the data accesses you found in the journal. Remember that patient records are accessed not just by crew members, but also by dispatchers, QA reviewers, billers, and supervisors.
  4. If there are data accesses in the journal which are not correlated with a date of service nor with the subsequent billing chores, then interview the employees who made those accesses.
  5. If you still don't have a smoking gun, then follow the instructions below for "Tracing a Leak of Other Data".

To learn more about the Patient Records Journal, refer to the Patient Records Journal Guide.

Tracing Employee Logons

If you need to review an employee's logon history, AngelTrack's logon records can be reviewed using the Employee Logon History report, available under HR Home.

The report shows the local timestamp, IP address, and user-agent (i.e. the web browser's identification text) for every logon attempt, including failed attempts.

You can paste each IP address into an IP address geolocation service (many are available for free on the internet) to get an idea of where that IP address is located, and the name of its ISP. Bear in mind that older records may show an IP address which is no longer geolocated where it was at the time of logon.

Tracing a Leak of Other Data / Reading a Cloud Server Log

If you must trace a leak of other data -- something not engraved with the name of the employee who downloaded it -- then you'll have to get your hands dirty in AngelTrack's webserver logs.

The webserver logs form a record of all requests made to AngelTrack by anyone, for any purpose. The logs show the UTC date, UTC time, IP address, user account, and the name of the requested AngelTrack page. You can download them from the Support Home page, but you must be an AngelTrack Administrator to do so.

The webserver logs are organized by date and time. Each log represents a 24-hour timespan, beginning and ending at midnight GMT; therefore you will see each log beginning and ending at 17:00 or 18:00 or 19:00 your local time.

Download all logs from the time period in question. Hopefully you have been able to narrow down the time period by reviewing the Patient Records Journal (described above), or by looking at the file creation time of the leaked data.

Here is a typical webserver log, showing a dispatcher beginning a new shift:

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2016-10-02 00:00:21
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2016-10-02 00:25:03
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2016-10-02 00:25:21 GET /Shifts.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/Dispatch.aspx 200 0 0 292
2016-10-02 00:25:21 GET /images/icons/Export.png - - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/Shifts.aspx 200 0 0 61
2016-10-02 00:25:21 GET /images/icons/Add.png - - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/Shifts.aspx 200 0 0 75
2016-10-02 00:25:23 GET /ShiftBegin.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/Shifts.aspx 200 0 0 461
2016-10-02 00:25:23 GET /scripts/FuelLevels.js - - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 92
2016-10-02 00:25:31 POST /ShiftBegin.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 172
2016-10-02 00:25:40 POST /ShiftBegin.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 198
2016-10-02 00:25:42 POST /ShiftBegin.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 145
2016-10-02 00:25:44 POST /ShiftBegin.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 160
2016-10-02 00:25:45 POST /ShiftBegin.aspx - jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 209
2016-10-02 00:25:45 GET /Shifts.aspx ATSaveRefer=0 jdoe 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/53.0.2785.116+Safari/537.36 https://sandbox2.angeltrack.com/ShiftBegin.aspx 200 0 0 139
[...]

 

Loading a webserver log into Microsoft Excel™

It is tricky to get Microsoft Excel to properly render a webserver log. The webserver log is space-delimited, but the extra header-lines in the file will throw off Excel's auto-detection attempt.

Follow these steps to force Excel to load the file properly:

  1. Do not double-click the logfile you wish to view. Instead, open the Excel program directly by clicking its icon on the Start menu.
  2. Click the File | Open menu item, and browse to the folder containing the webserver logs. You must switch the file-open dialog to "All Files (*.*)" to make the logfiles appear:

    Forensics.Excel.Open.1

  3. In page 1 of Excel's import wizard, switch the "Original data type" to "Delimited", set the starting import row to 4, and then click "Next":

    Forensics.Excel.Open.2

  4. In page 2 of the import wizard, change the delimiter character to space, then click "Finish":

    Forensics.Excel.Open.3

  5. When the completed worksheet appears, you must correct the title row. The word "#Fields:" occupies row 1 column A, but it is not a proper column header. The column headers begin at column B, so you must shift them all back one column. Thus, your worksheet initially looks like this:

    Forensics.Excel.Open.4

    ...but should look like this after you move all the column headers one column to the left:

    Forensics.Excel.Open.5

Now your webserver log is loaded and divided into its proper columns. You can thereafter use Excel's tools to hide the unimportant columns (particularly s-ip and s-port) and search and filter.

Incidentally, AngelTrack's webserver logs are in Microsoft IIS version 7.5 format. There is a great deal of help available on the internet for those wishing to analyze and interpret IIS 7.5 logfiles.

Interpreting a webserver log

Each entry in the log contains the following data items, in order:

Column Contents
date Date the request was received by the cloud server, in GMT, so you must convert it to your local time by adding 4, 5, 6, 7, or 8 hours. Remember you must also take account of Daylight Savings Time as it was at the time the logfile was written.
time Time the request was received by the cloud server, in GMT, not in your local time.
cs-method Contains the word "GET" or "POST", indicating whether the request was the first fetch of a page (GET) or was a postback of information to the cloud server (POST). All interactions with an AngelTrack page -- including button clicks, grid sorts, and filter actions -- are POSTs.
cs-uri-stem Path of the request, relative to your cloud server's URL. The page's name is a strong hint about what the requester was doing. Most pages in AngelTrack do not have data export facilities. For any page you see in the webserver log, you can visit it on your live AngelTrack cloud server to see what it does.
cs-uri-query Parameters passed to the AngelTrack page, such as "ID=42" or "ReturnTo=Dispatches". If you are replaying a page request you saw in the webserver log, then you must combine the cs-uri-stem with cs-uri-query by adding a question-mark between them, like this: DispatchEdit.aspx?ID=42
cs-username The login-name of the employee who made the request. No login-name will be reported for GETs of secondary files such as images and javascript modules.
c-ip The IP address of the computer making the request. You can look these up online to see which internet service provider owns the address; this will give you additional information about where the computer was. IP addresses owned by Verizon, T-Mobile, and AT&T Mobility indicate that the request came from a mobile device that was not using a wifi access point.
cs(User-Agent) This string of characters identifies the brand and version of web browser that issued the request. User-agent strings are misleading: they will mention several different browser names to indicate their compatibility. The only way to conclusively identify the user-agent string is to look it up on the internet.
cs-host The host name of the AngelTrack server, which indicates the customer name.
sc-status HTTP status code returned by the cloud server, where 200 indicates success, and 302 indicates a redirect as occurs after a Save button click.
sc-substatus HTTP sub-status code, where 0 indicates success.
sc-win32-status Internal windows error code if any, where 0 indicates success.
time-taken Time in milliseconds to process the request and transmit the reply back to the user. This includes travel time back crossing the internet.

These logs can be used to conclusively show which internet connection an employee was using at the time they accessed the system, and which browser and device they used.

Tracking a bulk export or other file download

If the leak began with a bulk export of data from AngelTrack, then you will be looking for a file download in the webserver log.

Here is a webserver log snippet showing an employee using the Closed Dispatches page to filter down to a set of records and then bulk export them:

2017-01-19 16:58:00 GET /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/Dispatch.aspx 200 0 0 470
2017-01-19 16:58:07 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 995 717
2017-01-19 16:58:08 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 995 845
2017-01-19 16:58:08 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 829
2017-01-19 16:58:33 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 813
2017-01-19 16:59:04 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 895
2017-01-19 17:15:52 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 370
2017-01-19 17:16:17 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 15126
2017-01-19 17:16:17 GET /images/icons/Suitcase.loaded.zip.gif - - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 63
2017-01-19 17:16:30 GET /temp/acme/AcmeAmbulance.jsmith.2017-01-19-09-16-16.zip - - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 6182
2017-01-19 17:16:39 POST /DispatchesClosed.aspx - jsmith 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_12_2)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/55.0.2883.95+Safari/537.36 https://acme.angeltrack.com/DispatchesClosed.aspx 200 0 0 424

 

Notice the initial GET for DispatchesClosed.aspx, which was referred by Dispatch.aspx. Then we see several POSTs, as the employee adjusts the grid's filters to narrow down the record-set. Then, finally, we see the GET for the completed bulk export at /temp/AcmeAmbulance.jsmith.2017-01-19-09-16-16.zip.

Simple grid exports -- in which the contents of an AngelTrack grid are Export-Sep-21-2022-12-05-36-83-AM exported as a .CSV to be opened in Microsoft Excel -- appear the same way in the webserver log. However, their file extension is .csv rather than .zip.

To quickly search a webserver log for file downloads, simply search for "GET /temp/".

Bear in mind, the logfile snippet above shows only those requests from user jsmith. There will normally be many other requests mixed into the log, from other (innocent) employees. It can take a while to screen out the many unrelated requests in order to zoom in on the activity from just one employee.

Replaying a prior download

If you see a file export in the log, you may be able to replay the download from your own web browser, if not too much time has passed. AngelTrack retains exported files in /temp for between two and seven days, depending on file size, in order that the download link remains valid on the original requester's computer.

To replay a download, find the path in the webserver log, and then type it into your web browser's address window like this:

https://acme.angeltrack.com/temp/acme/AcmeAmbulance.jsmith.2017-01-19-09-16-16.zip

If the exported data is still there (i.e. if the original export was performed only a couple of days ago), you will be able to download it again for inspection.

Help From AngelTrack LLC

If you aren't making any headway in your investigation, AngelTrack LLC will make available to you a data specialist on an hourly basis. The specialist can sign a Non-Disclosure Agreement with you as you see fit. Simply contact AngelTrack Headquarters Support at (800) 946-1808.