Commit Graph

137 Commits

Author SHA1 Message Date
Digimer
13b0f5bdcc Bumped 'Exhaust Temp' jump threshold to 30c in scan-ipmitool.
Adjusted some logging.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-10-08 20:34:09 -04:00
Digimer
a4ef93404c * Fixed a bug in DRBD->gather_data() to remove trailing commas for existing TCP ports.
* Added the missing 'clear-mapping' switch to Get->switches in anvil-daemon.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-10-05 20:15:32 -04:00
Digimer
ac8135709a Fixed a bug where scan-server faulted with a divide by zero error when the host had no swap.
Signed-off-by: Digimer <digimer@alteeve.ca>
2022-09-27 00:40:30 -04:00
Digimer
2fab7bc1b7 This adds support (testing needed) for "Long-Throw" DR; which is a wrapper for using 'drbd-proxy' to provide larger transmit buffers so slow/high-latency DR hosts.
* Created DRBD->check_proxy_license() to do (some level of) sanity checks on the DRBD proxy license file.
* Updated DRBD->gather_data() to parse out the inside and outside ports for resource configs using proxy.
* Reworked DRBD->get_next_resource() to return 1, 3 or 7 TCP ports depending, with the new long_throw_ports parameter triggering the 7 ports.
* Added 'tcpdump' to the anvil-core requires list.
* Reworked scan-drbd to record the ports used in proxy configs. This required adding a check to change the 'scan_drbd_peer_tcp_port' column type to 'text' to support CSVs.
* Reworked anvil-manage-dr (needs testing!) to support "long-throw" DR configs.
* Updated anvil-safe-stop to check if the nodes are in the cluster before trying to migrate.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-09-21 23:35:06 -04:00
Digimer
89121a2b3b * Fixed a bug in Alert->check_condition_age() where not setting a host_uuid caused the returned age to always be 0.
* Updated scan_apc_pdu to not report a lost PDU unless it's been gone for ten minutes.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-08-29 17:30:52 -04:00
Digimer
4ecc6097d3 * Cleaned up some old 'die' calls with better nice_exit() calls to help avoid dangling db_in_use flags.
* Reworked Network->bridge_info() to use 'ip' to get the list of bridges, and 'bridge' to find interfaces connected to the bridge.
* Added 'test' messages to Words->string().
* Fixed a bug in scan-lvm where mdadm based PVs didn't read the sector size properly.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-08-12 16:32:20 -04:00
Digimer
ef3ac86162 * Fixed a bug where setting the db_in_use flag without a valid $ENV{_}.
* Added a nice_exit call to tools/striker-access-database

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-08-09 15:45:10 -04:00
Fabio M. Di Nitto
7decdb2887 scan-network: fix path to script
Signed-off-by: Fabio M. Di Nitto <fabbione@fabbione.net>
2022-08-04 19:52:46 +02:00
Digimer
15aadc3a4e * Updated scan-network to check for inactive or activating interfaces and manually bring them up, if the uptime is less than 10 minutes.
* Fixed a bug in scancore-agents/Makefile.am where scan-network was missing.
* Started work on anvil-delete-server.8. Incomplete at this time.
* Updated Network->get_ips() to record the interface status.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-08-03 23:38:56 -04:00
Digimer
e025f5b927 Fixed line wraps
Signed-off-by: Digimer <digimer@alteeve.ca>
2022-07-09 19:53:58 -04:00
Digimer
7fd6185445 * Disabled firewalling for now. There appears to be an issue starting up with DRBD.
* Updated Convert->time() to return whatever was passed in instead of '#!error!#'.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-07-09 19:46:38 -04:00
Digimer
bce9e2caaf This is the first attempt at enabling firewalld completely. There is a decent chance that problems exist, so it won't be a surprise if a few more commits are needed to this branch before things work.
* Added multiple new private methods to Network that help in managing the firewall.
* Updated Server->boot_server to manage the firewall after the server boots. Updated ->migrate_server to create a job, if a database connection exists, for the migration target to update it's firewall as soon after the server appears as possible.
* Updated ocf:server:alteeve to manage the firewall when called post-migration, in case there was no DB connection and the job above didn't run. Fixed a bug where the disk state wasn't being evaluated properly.
* Updated scan-server to check that the firewall is managed when a server state has changed.
* Updated anvil-daemon to run Network->manage_firewall on startup.
* Heavily reworked 'anvil-manage-server' to either just run 'Network->manage_firewall', or if passed '--server X', to wait for the server to appear for up to 1 minute, then to check that the firewall is managed (to capture servers being migrated to the host.)
* Removed firewall management from striker-prep-database.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-07-02 17:06:04 -04:00
Digimer
b2ea4f9adc * Moved System->manage_firewall() to Network->manage_firewall(). Started working on actually implementing it, which involves basically fully rewritting it.
* Updated tools/Makefile.am and scancore-agents/Makefile.am to add missing files.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-06-30 00:01:50 -04:00
Digimer
ab9b00a2f7 * Updated anvil-daemon, in its daily checks, to disable ksm and ksmtuned daemons.
* Updated scan-drbd to purge peer records that no longer have corresponding LVM data.
* Updated System->{en,dis}able-service to take the 'now' paramter which, when passed, causes the action to take immediate effect.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-06-21 22:25:07 -04:00
Digimer
24f5d39dff This is a set of changes all stemming from trying to debug frequent resyncs. More bugs still to be fixed.
* Updated Database->get_host_from_uuid() to cache results.
* Fixed a bug in Database->get_storage_group_data where a DELETE wasn't deleting from the history schema as well.
* In Database->resync_databases(), references to the old 'host_uuid' that we used to use to resync just the local host's data was removed. Added also a check where two or more entries in a given history schema had the same modified_date and, when found, the newest entry is preserved and the rest are deleted. Before this, a resync where two+ records had the same modified_time would only sync the last record, leaving a mismatch in history schema entries triggering repeated resyncs.
* Fixed a bug in Email->send_alerts() where the 'alerts' table was being updated without a modified_date being set.
* Fixed a bug in System->test_ipmi() where the 'hosts' table was being updated without a modified_date being set.
* Updated scan-network to clear up old deleted ip_addresses, bonds and bridges. Also fixed bugs where public schema records were being deleted without history records being deleted.
* Updated anvil-update-states to fix bugs where DELETEs were happening without setting the modified_date.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-06-12 23:14:49 -04:00
Digimer
e6dcff1cf1 * Added a missing modified_date to ip_addresses in Database->get_ip_addresses().
* Updated scan-network to purge old historical ip_addresses when clearing duplicates now.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-05-21 15:52:25 -04:00
Digimer
1b70b49cf8 * Updated Network->find_matches() to try to populate the first and second parameters if they're not passed in.
* Updated Network->load_ips() to load extra information about the interfaces.
* Updated ocf:alteeve:server to not check libvirtd daemon state on server start.
* Updated scan-hardware to check for duplicate entries and purge if found.
* Updated scan-network to check for the 'default' virbr0 interface by checking if the config file exists instead of calling virsh.
* Updated scan-server to have better logging.
* Created the new (and incomplete) anvil-test-alerts tool
* Updated scancore to support --purge to pass to all agents and then exit.
* Updated ScanCore->call_scan_agents() to no longer use 'timeout' as it was causing issues with virsh calls.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-05-20 10:28:21 -04:00
Digimer
d26a16e711 * Updated anvil-provision-server to handle human-readable sizes for disk and ram.
* Updated Database->get_anvils() to make it possible to translate a file name to a file UUID.
* Updated System->test_ipmi() to quote passwords properly. Also dropped the timeouts to 2 seconds.
* Updated anvil-provision-server to support pure CLI switch server provisioning using the --ci-test (and optional --options {--machine}) to allow CI tests.
* Continued work of anvil-manage-server.
* Fixed a bug in striker-prep-database to fix a bug in writing the pg_hba.conf file.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-05-10 00:42:40 -04:00
Digimer
1dbca79dde * Created Network->get_ip_from_mac() which takes a MAC address and returns an IP address.
* Updated ocf:alteeve:server to always try to bring up the peer's DRBD resource, even when the local resource is up.
* Fixed a bug in scan-network where purging duplicate bridges failed in some cases.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-04-08 23:09:34 -04:00
Digimer
3ad1793c23 Made scan-network more robust at determining when an interface is a virtio device.
Signed-off-by: Digimer <digimer@alteeve.ca>
2022-04-04 18:51:28 -04:00
Digimer
c9633aa3b0 * Updated Database->_find_behind_databases() to not run unless it's on a Striker.
* Updated scan_network to properly mark virtio network interfaces as being full duplex. Also updated it to purge interfaces flagged as 'DELETED'.
* Updated the VM OS list.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-04-04 16:38:14 -04:00
Digimer
f77f486775 Fixed a typo in scan-network
Fixed a missing 'next' to prevent the first DB from disconnecting when down'ing excess DBs.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-02-09 15:52:21 -05:00
Digimer
a886653af1 * Updated scan-network to purge duplicate bridges and bonds.
Signed-off-by: Digimer <digimer@alteeve.ca>
2022-02-01 20:15:01 -05:00
Digimer
2c76103a96 Fixed a bug where, if the host IPMI BMC wouldn't allow spaces in the password and the user had a space, IPMI would never configure or get used as a fence method.
Signed-off-by: Digimer <digimer@alteeve.ca>
2022-01-17 17:54:46 -05:00
Digimer
796814531e Fixed a bug in Alert->check_condition_age() where, when the 'clear' parameter was set and the value was already 'clear', it would flip to 'set' erroniously.
Signed-off-by: Digimer <digimer@alteeve.ca>
2022-01-13 21:07:25 -05:00
Digimer
652f87ec74 * Updated scan-network to also clean up the media type.
* Updated anvil-daemon to check for files in /mnt/shared/incoming on striker dashboards and add them to the media library if needed.

Signed-off-by: Digimer <digimer@alteeve.ca>
2022-01-12 23:27:44 -05:00
Digimer
257a998743 * Updated Database->configure_pgsql() to use 'postgresql-setup --initdb --unit postgresql' instead of the deprecaded 'initdb' switch.
* Updated Database->insert_or_update_states() to switch to an active UUID if the passed in UUID is not an active handle.
* Updated Database->query() to swutch to 'sys::database::read_uuid' if the passed in 'uuid' is not an active handle.
* Updated Database->_test_access() to return immediately if the passed in uuid is not an active handle.
* Started working on a Storage->get_storage_group_from_path() bug where the storage group isn't being returned.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-28 12:07:36 -04:00
Digimer
32d47f70f1 * Fixed bugs around ScanCore->check_power() so that it now returns time on batteries and highest charge are returned properly.
* Created Network->is_our_interface() which returns '1' if an interface is one managed by an Anvil!. Also updated scan-network to use this to determine when an interface alert should be a warning or notice level alert.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-11 21:57:35 -04:00
Digimer
5c85cee49a Made estimated hold up time alerts 'notice' level instead if 'warning'.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-11 16:44:50 -04:00
Digimer
226d1de6b5 Updated anvil-update-states to use the permanent MAC addresses, as done in scan-network. Updated Network->get_ips() to do the same.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-11 02:57:45 -04:00
Digimer
7f1f71dcf0 Updated scan-apc-pdu to remove PDUs that are associated with fence devices that have beed deleted.
Updated scan-network to set a health score against an interface that has been missing for 1 minute, instead of waiting for 5 minutes.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-11 01:49:30 -04:00
Digimer
63c45430bb * Updated scan-network to clear duplicate IP addresses.
* Fixed a bug in anvil-daemon where striker-prep-database was always being called, when it shouldn't in some cases.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-10 22:27:54 -04:00
Digimer
75abeb3b9a * Updated scab-storcli to not trigger a health alarm when the alarm state is "absent".
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-09 21:24:36 -04:00
Digimer
eeafefbe08 * Updated scan-network to (hopefully, needs testing) not set a health score against unused/unconfigured network interfaces. Also silenced the purging of old RX/TX variables now that it appears to be working well.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-09 01:08:19 -04:00
Digimer
2557fa454a * Updated scan-network to delete old TX/RX records.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-08 00:56:13 -04:00
Digimer
a1604344c7 * Fixed a bug in Convert->round where numbers with exponents were not handled properly.
* Fixed a bug in scan-hardware where the raw bytes free for swap was used to see if the high / cleared thresholds were passed, instead of the percentage as it should have been.
* Fixed a bug in scan-network where a new-line wasn't be cleared off the MAC address.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-07 22:08:34 -04:00
Digimer
47c832bc0e * Updated Network->get_ips() to check for 'permaddr' when processing 'ip addr list' to ensure the partmanent MAC is used.
* Updated scan-filesystems to set swap usage alerts to notice level only.
* Updated scan-network to pull the permanent MAC address from an 'ethtool -P <iface>' call to deal with the fact that wireless interfaces don't have their real MAC in the sysfs address file.
* Updated anvil-provision-server to set the rtc_tickpolicy to catchup.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-10-07 17:10:25 -04:00
Digimer
6664c5b77f * Fixed a bug where scan-drbd, with DR configured, was not recording TCP ports assigned to connections properly.
* More bugs fixed in anvil-manage-dr, tested repeatedly as a job and so far, so good. Other functionality still to come.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-09-12 23:34:25 -04:00
Digimer
213babaaf2 Trying to fix a bug where vnet devices keep reporting as having returned.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-08-30 14:09:45 -04:00
Digimer
8abb5b46e0 * Added support for setting per-agent log-level and log secure values in amvil.conf.
* Moved the check for an agent being disabled into ScanCore->agent_startup()

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-08-18 23:07:15 -04:00
Digimer
38f95870bb Changed the agent runtime timeout to 60 seconds.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-08-11 22:39:23 -04:00
Digimer
15d8309095 This commit adds scan agent DB connection info caching to help minimize the number of unnecessary DB resync checks that happen.
* Created ScanCore->agent_shutdown() that writes out the time the scan agent last ran, and how many databases were available when it last ran.
* Updated ScanCore->agent_startup() to read the the last run data created above.
* Updated Database->connect() to set 'sys::database::last_db_count' to the scan agent's recorded last DB count.
* Updated all agents to call ScanCore->agent_shutdown() at the end of their run.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-08-11 21:14:35 -04:00
Digimer
aec22bb79c Added a check in scan-network that finds/removes duplicate network interface names.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-08-11 12:17:01 -04:00
Digimer
e1aa4442ad Updated scan-server to not check libvirtd status as it now auto stop/starts as needed. Also fixed a pair of bugs that was preventing VNC checks from happening, or for stopped server definition checks from happening.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-08-03 13:36:32 -04:00
Digimer
8274aa8f79 Updated (but needs testing) scan-server to look for changes in the on-disk XML definition changes or database XML data for servers that are NOT running on a host or node.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-07-31 12:22:48 -04:00
Digimer
ccd89f923b Fixed two small bugs that were preventing proactive live migration from working.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-07-20 01:56:50 -04:00
Digimer
ef10765d8a Fixed a bug with registering health scores against down links in scan-network.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-07-18 23:25:46 -04:00
Digimer
ad4609b39c Fixed a bug where interfaces in bonds weren't recording their parent bond.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-07-18 16:30:45 -04:00
Digimer
d62900d712 Fixed a bug where the wrong string key was used when a network interface comes, goes or changes it's bond_uuid reference. Also changed the alert level of vnetX interfaces that disappear to notice level.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-07-15 23:04:23 -04:00
Digimer
0c77736dc8 * Fixed a bug in Cluster->manage_fence_delay() where removing the 'delay="15"' attribute was failing, now set it to 0 instead.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-07-15 19:39:05 -04:00