Servers
Recording Server
| The server that has access to
recorded channels data and database.
The application (ActFtpUpload)
will run on this server, called by ActControl service.
|
Central Storage
| Standard FTP server
|
Server where all chunks are
stored
|
Upload queue
“upload queue” is a concurrent queue that is accessible from
each thread and which contains an ordered list, by creating time, of mp4 files
and subtitle folders to be uploaded.
When one files are taken from the list to upload, this file
will be removed from list. If the connection will be interrupted while this
file is uploading, this will be saved already in database as “uploading” and
when connection will be stabilized, the upload will be resumed.
When the queue comes to be updated and it still contains
files, the new files will be added at the beginning of the queue, ordered.
Object structure
video
| VideoFile object
|
subtitles
| Dictionary<String (language;
ex. eng), List<SubtitleFile> (list of subtitle objects)>;
|
sublitleDirCreatedTime
| Craetion date of subtitle
directory
|
subtitlePath
| Path for subtitle directory
|
rootDirectory
| Root directcory of chunk (ex.
D:\\Media\Record)
|
subtitleDirSize
| Subtitle directory size
|
videoStatus
| Video file status
|
subtitleStatus
| Subtitle status
|
isLastProcessed
| Flag, to know if this is the
newest file of channel
|
ActFtpUpload Configuration
ActusSettings.xml file.
<appSettings>
<add
key="machine" value="BaiaMare"/>
</appSettings>
machine – to name status files on ftp server
This part will be included in
ActusSettings.xml between <configuration> … </configuration>
<ActFTPUpload>
<Settings>
<add key="ftpport"
value="14147" />
<add key="ftpuser"
value="user" />
<add key="ftppassword"
value="123456" />
<add key="destfolder"
value="ftp_server/test_upload/Media/Record" />
<add key="FTPStatus"
value="ftp_server/test_upload/Media/status_files" />
<add
key="checkThreadSleepTime" value="60000" />
<add key="uploadThreadSleepTime"
value="10000" />
</Settings>
<recordingroots>
<server
root="D:\\Media\Record"/>
<server
root="D:\\Media\Record2"/>
</recordingroots>
</ActFTPUpload>
ftpserver – ftp server address
ftpport – ftp server port
fptuser – ftp server user
ftppassword – ftp server password
destfolder – root of files destination on ftp server
FTPStatus – path for status file on ftp server
checkThreadSleepTime – the time in milliseconds to wait for
checking files on recording server
uploadThreadSleepTime – the time in milliseconds to wait for
uploading files if the upload queue is empty
recording roots – the service will check all of these directories
for channels to upload files
Status file
The status file is an xml file that contains information on
central storage about all files from recording server (uploaded, not uploaded,
uploading).
Each recording server will work with a separate file of its
and on the central storage will be a status file for every recording server to
avoid errors.
On Central Storage, all status files will be in one folder (ex. \Media\Record\status_files\). Status files will be
named as: RecordingServerName_DateOfFile(yyyy_mm_dd).
Ex. London_2018_02_22.xml.
The status files will be overridden during the day, but at
every new day will be created a new status file and status file of the last day
will remain intact on server. In this way we can keep a history.
Status file structure
| File size
| Status file size, to be able to
check if it was sent correctly (bytes)
|
Date of creation
| Status file creation date
|
Content
| Files status
| File / folder path
Type:
· video
· subtitle
|
Size (bytes)
|
Status:
· waiting
· completed
· uploading
|
Created on
Uploaded on
Upload duration
|
|
Status file example
<?xml version="1.0"
encoding="utf-8"?>
<StatusFile>
<Header size="4469"
date="2018-04-30 11:25:56 AM" />
<Content>
<Item
path="channel01\2018_04_30\channel01_2018_04_30_10_45_00.mp4"
type="video" size="24610346" status="completed"
createdOn="2018-04-30 10:45:00 AM" uploadStarted="2018-04-30
10:50:13 AM" uploadedOn="2018-04-30 10:50:14 AM"
uploadDuration="0.48" />
<Item
path="channel01\2018_04_30\channel01_2018_04_30_10_45_00"
type="subtitle" size="4763645" status="completed"
createdOn="2018-04-30 10:45:00 AM" uploadStarted="2018-04-30
10:52:27 AM" uploadedOn="2018-04-30 10:52:27 AM"
uploadDuration="0.14" />
<Item
path="channel01\2018_04_30\channel01_2018_04_30_10_40_00.mp4"
type="video" size="24579362" status="completed"
createdOn="2018-04-30 10:40:00 AM" uploadStarted="2018-04-30
10:52:27 AM" uploadedOn="2018-04-30 10:52:28 AM"
uploadDuration="0.38" />
</Content>
</StatusFile>
Workflow
Thread 1
In every 5 minutes
| Check on the recording server
if there are new chunks ready.
(The subtitles will be checked
only if mp4 file are created)
|
The new files are ordered and added
to upload queue.
|
The ordered list is ordered by
chunk time from newest to oldest.
The last created file will be
the first on this list, so the new files will always be uploaded firstly.
|
Thread 2
In a loop
| Get object (video file /
subtitle folder) to upload from upload queue.
|
Remove object from upload
queue.
|
Add files to database
|
Update status file from central
server to know which object is being uploading and as case which object was
uploaded.
|
Start to send object to central storage.
|
Update data base on recording
server with status information.
|
Wait until current object is
uploaded
|
The object has been uploaded.
|
Database will be updated.
|
**If the connection is
interrupted, the central server will know which files/folders were completely
uploaded, which file/folder is not fully uploaded and which folders are not uploaded
at all
|
|
|
Troubleshooting
The video files are sending one by one, for better control,
from a Recording Server to Central Storage.
The subtitles are sending all folder at once, folder by
folder.
After each video file and subtitle folder, status file will
be sent. In this way we know on the central storage the status of each
recording server.
In case of internet down for long time, the created chunks
are not uploading all at a time to not block the connection.
The recent chunks are sent first, so in case of a slow
connection the newest files will be on central storage.
When the application starts, Thread 2 will check (in
database) if there are file which started to upload, but did not uploaded
totally. If there are such a file, this will be uploaded firstly.
Database on Recording Server
Two table:
1. act_ftp_upload
· id (int, autoinc.) – record
id
· path – file path on
recording server
· date_of_creation – creation
date of file
· status – file status: 0 -
wait to upload, 1 – uploaded, 2 – uploading
· date_of_upload – upload
date of file
· total_size – file size
· type: 1-video, 2-subtitle
· upload_started: datetime
when upload started
· upload_duration: duration
of upload in seconds
2. channel_status
· id (int, autoinc.)
· channel_name – channel name
· root_directory – channel
root directory (ex. D:\\Media\Record)
· last_preocessed_file_date –
creation date of last processed file, the last file found up to that point
Others
If directories do not exist on
ftp server the service will create them.
Only the files that were created
after the date saved in the database (channel_status table,
last_processed_file_date) will be checked, so if you want to adjust the date
from where you want to upload the files, will need to change this date. If the
date is not set then it will check and upload all existing files