* To update file handling for the new DR host linking mechanism, file_locations -> file_location_anvil_uuid was changed to file_location_host_uuid.
This required a fair number of changes elsewhere to handle this, with a particular noted change to Database->get_anvils() to look at host_uuid's for the subnodes in an Anvil! and, if either is marked as needing a file, make sure the peer is as well. Similarly, any linked DRs are set to have the file as well.
* Created a new Network->find_access that simply takes a target host name or UUID, and it returns a list of networks and IPs that the target can be accessed by.
* Updated Network->load_ips() to find the network interface being used for traffic so that things like the interface speed can be recorded, even when an IP is on a bridge or bond.
Unrelated, but in this commit, is a restoration of calling scan agents with a timeout now that the virsh hang issue has been resolved.
Signed-off-by: digimer <mkelly@alteeve.ca>
my$query="SELECT network_interface_uuid FROM network_interfaces WHERE network_interface_host_uuid = ".$anvil->Database->quote($host_uuid)." AND network_interface_name = ".$anvil->Database->quote($active_interface).";";
-- This tracks which files on Strikers should be on Anvils!
CREATETABLEfile_locations(
file_location_uuiduuidnotnullprimarykey,
file_location_file_uuiduuidnotnull,-- This is file to be moved to (or restored to) this machine.
file_location_anvil_uuiduuidnotnull,-- This is the sum as calculated when the file_location is first uploaded. Once recorded, it can't change.
file_location_activebooleannotnulldefaultTRUE,-- This is set to true when the file should be on Anvil! machines, triggering rsyncs when needed. When set to false, the file will be deleted from members, if they exist.
modified_datetimestampwithtimezonenotnull,
file_location_uuiduuidnotnullprimarykey,
file_location_file_uuiduuidnotnull,-- This is file to be moved to (or restored to) this machine.
file_location_host_uuiduuidnotnull,-- This is the sum as calculated when the file_location is first uploaded. Once recorded, it can't change.
file_location_activebooleannotnulldefaultTRUE,-- This is set to true when the file should be on Anvil! machines, triggering rsyncs when needed. When set to false, the file will be deleted from members, if they exist.
@ -586,6 +586,7 @@ The definition data passed in was:
<keyname="error_0411"><![CDATA[The server: [#!variable!server!#] was not found. Exiting.]]></key>
<keyname="error_0412"><![CDATA[The server: [#!variable!server!#] is on the Anvil! node: [#!variable!on_anvil!#], and this is: [#!variable!this_anvil!#]. Exiting.]]></key>
<keyname="error_0413"><![CDATA[You need to specify the server with '--server <name or uuid>'. Available servers are;]]></key>
<keyname="error_0414"><![CDATA[The target host: [#!variable!target!#] was not found in the database.]]></key>
<!-- Files templates -->
<!-- NOTE: Translating these files requires an understanding of which lines are translatable -->
@ -2236,7 +2237,7 @@ The file: [#!variable!file!#] needs to be updated. The difference is:
<keyname="log_0610">We're online as: [#!variable!node_name!#], but we're not quorate yet. Continuing to wait.</key>
<keyname="log_0611">We're online as: [#!variable!node_name!#] and quorate!</key>
<keyname="log_0612">We're not online yet. Waiting for 'in_ccm/crmd/join': [#!variable!in_ccm!#/#!variable!crmd!#/#!variable!join!#]. ('in_ccm' = consensus cluster member, communication layer. 'crmd' = cluster resource manager daemon is up, 'join' = allowed to host resources).</key>
<keyname="log_0613">The file: [#!variable!file_name!#] is not recorded for the Anvil! [#!variable!anvil_name!#] yet. Registering it now as not sync'ed to this Anvil! system.</key>
<keyname="log_0613">The file: [#!variable!file_name!#] is not recorded for the host [#!variable!host_name!#] yet. Registering it now as not sync'ed to this system.</key>
<keyname="log_0614">Asking 'anvil-boot-server' to boot the servers now.</key>
<keyname="log_0615">We were asked to delete the file: [#!variable!file!#], but it doesn't exist, so nothing to do.</key>
<keyname="log_0616">The file: [#!variable!file!#] has been successfully removed.</key>
@ -2657,8 +2658,11 @@ About to try to download aproximately: [#!variable!packages!#] packages needed t
<keyname="message_0192">Moving the file: [#!variable!file!#] to: [#!data!path::directories::shared::files!#].</key>
<keyname="message_0193">Calculating the md5sum. This can take a little while for large files, please be patient.</key>
<keyname="message_0194">The md5sum is: [#!variable!md5sum!#]. Storing details in the database.</key>
<keyname="message_0195">Copying the file over to: [#!variable!host!#]. Please be patient, this could take a bit for large files.</key>
<keyname="message_0196">Registering the file to be downloaded to the Anvil!: [#!variable!anvil_name!#]. Anvil! members will sync this file shortly. Member machines that are not online will sync the file when they do return.</key>
<keyname="message_0195">
Copying the file: [#!variable!source_file!#] over to: [#!variable!host!#:#!variable!target_directory!#] using: [#!variable!ip_address!# (#!variable!network!#)].
The file size is: [#!variable!size!#], and the link speed appears to be: [#!variable!link_speed!#], for an estimated copy time of about: [#!variable!eta_copy_time!#].
Please be patient, this could take a bit for large files.</key>
<keyname="message_0196">Registering the file to be downloaded to host: [#!variable!host!#]. Hosts that are not online will sync the file when they do return.</key>
<keyname="message_0197">Upload is complete!</key>
<keyname="message_0198">Processing the pull of a file from Striker.</key>
<keyname="message_0199">We're a DR host and there are: [#!variable!strikers!#] dashboards, so we will wait to pull the file until after the nodes are done. We're currently waiting on; Node 1? [#!variable!node1_waiting!#], Node 2? [#!variable!node2_waiting!#]. We'll check again at: [#!variable!wait_until!#].</key>
@ -2860,6 +2864,8 @@ Proceed? [y/N]</key>
<keyname="message_0306">This is a test alert message sent at alert level: [#!variable!level!#].</key>
<keyname="message_0307">Failed to send a test alert at level: [#!variable!level!#]. Is anyone listening at that level? Is the mail server configured?</key>
<keyname="message_0308">The DRBD config file was not found. A protect job needs to be run from the Anvil! node hosting the server to be protected.</key>
<keyname="message_0309">Waiting a bit to make sure the file: [#!variable!file!#] is done uploading...</key>
<keyname="message_0310">Upload complete.</key>
<!-- Translate names (protocols, etc) -->
<keyname="name_0001">Normal Password</key><!-- none in mail-server -->
@ -3264,6 +3270,15 @@ If you are comfortable that the target has changed for a known reason, you can s
<keyname="suffix_0055">Zbps</key><!-- Short suffix for 'zettabits per second' (1,000,000,000,000,000,000,000 bits per second). -->
<keyname="suffix_0056">Ybps</key><!-- Short suffix for 'yotabits per second' (1,000,000,000,000,000,000,000,000 bits per second). -->
<keyname="suffix_0057">Bytes</key>
<keyname="suffix_0058">Bytes/sec</key>
<keyname="suffix_0059">KiB/sec</key><!-- Short suffix for 'kibibyte/sec' transfer rate. -->
<keyname="suffix_0060">MiB/sec</key><!-- Short suffix for 'mebibyte/sec' transfer rate. -->
<keyname="suffix_0061">GiB/sec</key><!-- Short suffix for 'gibibyte/sec' transfer rate. -->
<keyname="suffix_0062">TiB/sec</key><!-- Short suffix for 'tebibyte/sec' transfer rate. -->
<keyname="suffix_0063">PiB/sec</key><!-- Short suffix for 'pebibyte/sec' transfer rate. -->
<keyname="suffix_0064">EiB/sec</key><!-- Short suffix for 'exbibyte/sec' transfer rate. -->
<keyname="suffix_0065">ZiB/sec</key><!-- Short suffix for 'zebibyte/sec' transfer rate. -->
<keyname="suffix_0066">YiB/sec</key><!-- Short suffix for 'yobibyte/sec' transfer rate. -->
<!-- Test words. Do NOT change unless you update 't/Words.t' or tests will needlessly fail. -->
# This replaces anvil_uuid with host_uuid to support more granular location info to support the new
# multi-target DR system
sub update_file_locations
{
my ($anvil) = @_;
my $updated = 0;
foreach my $uuid (sort {$a cmp $b} keys %{$anvil->data->{cache}{database_handle}})
{
my $query = "SELECT COUNT(*) FROM information_schema.columns WHERE table_schema = 'public' AND table_name = 'file_locations' AND column_name = 'file_location_anvil_uuid';";
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { query => $query }});
my $count = $anvil->Database->query({query => $query, uuid => $uuid, source => $THIS_FILE, line => __LINE__})->[0]->[0];
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
uuid => $uuid,
count => $count,
}});
# Wipe it out, then re-add
if ($count)
{
$updated = 1;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { updated => $updated }});
push @{$queries}, "DROP FUNCTION history_file_locations() CASCADE;";
push @{$queries}, q|CREATE TABLE file_locations (
file_location_uuid uuid not null primary key,
file_location_file_uuid uuid not null, -- This is file to be moved to (or restored to) this machine.
file_location_host_uuid uuid not null, -- This is the sum as calculated when the file_location is first uploaded. Once recorded, it can't change.
file_location_active boolean not null default TRUE, -- This is set to true when the file should be on Anvil! machines, triggering rsyncs when needed. When set to false, the file will be deleted from members, if they exist.
# Make sure all files are linked to all nodes and dr_hosts
if ($updated)
{
$anvil->Database->get_hosts();
$anvil->Database->get_files();
$anvil->Database->get_file_locations();
foreach my $file_name (sort {$a cmp $b} keys %{$anvil->data->{files}{file_name}})
{
my $file_uuid = $anvil->data->{files}{file_name}{$file_name}{file_uuid};
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
's1:file_name' => $file_name,
's2:file_uuid' => $file_uuid,
}});
foreach my $host_name (sort {$a cmp $b} keys %{$anvil->data->{sys}{hosts}{by_name}})
{
my $host_uuid = $anvil->data->{sys}{hosts}{by_name}{$host_name};
my $host_type = $anvil->data->{hosts}{host_uuid}{$host_uuid}{host_type};
my $host_key = $anvil->data->{hosts}{host_uuid}{$host_uuid}{host_key};
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
's1:host_name' => $host_name,
's2:host_uuid' => $host_uuid,
's3:host_type' => $host_type,
's4:host_key' => $host_key,
}});
next if $host_type eq "striker";
next if $host_key eq "DELETED";
my $file_location_uuid = $anvil->Database->insert_or_update_file_locations({
debug => 2,
file_location_file_uuid => $file_uuid,
file_location_host_uuid => $host_uuid,
file_location_active => 1,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { file_location_uuid => $file_location_uuid }});
}
}
}
return(0);
}
# This converts the old/broken 'notifications' tables with the more appropriately named 'alert-override'
sub update_notifications
{
@ -373,9 +504,9 @@ sub update_notifications
foreach my $uuid (sort {$a cmp $b} keys %{$anvil->data->{cache}{database_handle}})
{
my $query = "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public' AND table_catalog = 'anvil' AND table_name = 'notifications';";
$anvil->Log->variables({source => $THIS_FILE, uuid => $uuid, line => __LINE__, level => 2, list => { query => $query }});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { query => $query }});
my $count = $anvil->Database->query({query => $query, source => $THIS_FILE, line => __LINE__})->[0]->[0];
my $count = $anvil->Database->query({query => $query, uuid => $uuid, source => $THIS_FILE, line => __LINE__})->[0]->[0];
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { count => $count }});
digimer
2 years ago