Commit Graph

575 Commits

Author SHA1 Message Date
digimer
fd880e2fdf Finished anvil-watch-servers
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-29 11:37:00 -05:00
digimer
26f4446bf9 Continued work on anvil-watch-servers; Parsed server data now.
* Updated Cluster->parse_cib() to store DRBD fence node restrictions by
  server/node. Also updated to make it easier to get the server's
  preferred node.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-29 01:27:21 -05:00
digimer
207a014ae0 Got anvil-watch-servers showing the status of subnodes.
* Updated System->maintenance_mode() to take 'host_uuid' so that the
  maintenance mode of remote machines can be checked/set.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-27 23:47:29 -05:00
digimer
8ce1f04335 Finished CPU support in anvil-manage-server-system!
* Updated Get->available_resources() to record the maximum cores that
  can be allocated to a server. This is N-1 for hosts with 4 or less
  cores, or N-2 cores otherwise.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-24 16:51:15 -05:00
digimer
b4037fade5 Added RAM change support to anvil-manage-server-system
* Updated Database->insert_or_update_servers() to error if the RAM being
  recorded is less than 640 KiB. This is because, somewhere yet
  undiscovered, the RAM is being recorded in KiB which breaks things.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-23 22:53:36 -05:00
digimer
6bc2601d34 Updated anvil-manage-server-system to change boot device ordering.
* Updated Server->parse_definition() to store a hash to translate a
  device target to a device type.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-23 19:22:46 -05:00
digimer
1e376fc06b Updated anvil-manage-dr to change --list to --show for consistency
* Updated anvil-manage-dr to handle DR hosts without a VG in a given SG
* Fixed up minor display issues in anvil-manage-storage-groups

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-17 23:52:58 -05:00
digimer
ef0f04117f Added '--list' to anvil-manage-dr
* Updated Database->get_hosts() to store hosts in a host_type hash.
* Updated Database->get_servers() to store servers by name, regardless
  of host Anvil! node.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-14 17:50:48 -05:00
digimer
d9aa8aee74 Updated anvil-manage-keys to work from the command line
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-13 18:59:24 -05:00
digimer
9bd98951b5 Added backup/restore of partition table in Storage->auto_grow_pv
* Auto-growing PVs (and the backing partition) is now supported by
  anvil-manage-host '--auto-grow-pv'.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-08 20:59:40 -05:00
digimer
edc544255e Rebased with main and resolved conflicts.
This branch resolves issue #462; Auto growing PVs. Specifically, it looks at the LVM PVs on the host and checks to see if there is unused free space after the backing partition. If there is, it auto-grows the partition and then resizes the PV. This featu
re is designed to make life easier for users who deleted the auto-created '/home' partition during the anaconda disk partitioning tool.
* Created Storage->auto_grow_pv() that does the above.
* Added the missing hidden method name _create_rsync_wrapper in the Storage module POD.
* Added a call to Storage->auto_grow_pv() in anvil-configure-host and anvil-version-changes for nodes and DR.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-08 12:00:48 -05:00
digimer
e5e958a03b Updated anvil-configure-host to clear "config::map_network"
Also updated anvil-manage-host to properly display the network mapping
status.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-03 17:35:01 -04:00
digimer
8c97f478a8 Updated Server->update_definition() to undefine a server when needed.
* Boosted logging to debug anvil-delete-server hang in jenkins.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-01 16:06:59 -04:00
digimer
9b55504872 Updated anvil-manage-server-system to update defined servers.
* Updated Server->locate() to take the new 'anvil' parameter to speed up
  searches.
* Updated Server->update_definition() to use Server->locate() to find
  where updates are needed. It now also defines the server with the new
  config.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-11-01 00:15:13 -04:00
digimer
85f86cda8a Updated Storage->read_file() to test if the target file exists.
This allows for better logging if the file to be copied doesn't exist.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-25 21:21:52 -04:00
digimer
ed269aa450 Fixed a bug where duplicate variables were being created in some cases.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-20 22:05:33 -04:00
digimer
a35e790a4d Fixed a minor striker-boot-machine bug.
Divide by zero error when no hosts with IPMI found.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-20 11:38:15 -04:00
digimer
b1f89c2723 Finished initial version of striker-show-jobs
* Updated Database->get_jobs() to take 'job_host_uuid = all' to allow
  loading jobs from all cluster machines. Also updated it to record the
  'job_host_uuid' and the unix timestamp version of 'modified_date'.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-18 20:51:20 -04:00
digimer
4398ffe70c Updated striker-boot-machine to support booting all machines.
* Wrote the man page for striker-boot-machine, changing --host-name to
  --host, and adding the '--host all' support.
* Updated anvil-manage-host to support checking/enabling/disabling
  network mapping mode.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-12 22:18:07 -04:00
digimer
55b1380031 Finished (but need more testing) of Server->locate().
This includes the changes in PR#492.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-11 17:22:06 -04:00
digimer
f12e001ac2 Finished Server->connect_to_virsh().
* Now, connecting to virsh can detect when still-open connections
  already exist.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-11 17:22:06 -04:00
digimer
245f75de9b Added Server->update_definition()
* This takes a server and new definition XML and updated the database and any available hosts. Does not yet update defined or running servers.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-11 17:22:06 -04:00
digimer
62fe62a44b * Continued work on anvil-manage-server-system. It now displays the boot devices, CPU and RAM info.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-10-11 17:22:06 -04:00
digimer
74ddb7f3a9 Updated Database-get_files() to detect/remove duplicate file entries.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-29 21:28:49 -04:00
digimer
fcbace6713 Updated anvil-join-anvil to hold if either node is still running anvil-configure-host
* Fixed a minor bug and added logging of maintenance_mode calls in anvil-configure-host.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-28 16:01:32 -04:00
digimer
582a8b292c Added more job updates to anvil-manage-power.
* This is a test to see if the job waiting for the uptime to be 300s,
  leaving the job_progress as 0, was causing the job to be repeatedly
  called.
* This is related to issue #479

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-28 01:19:52 -04:00
digimer
ef042eef25 Cleaned up logging while waiting for subnodes.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-28 00:15:14 -04:00
digimer
5d5270486e Added a wait loop when forming node clusters.
* This adds a check where anvil-join-anvil waits until both subnodes are
  marked as configured and not in maintenance mode.
* Should address issue #479 (maybe, this shouldn't trigger reboots, but
  it was certainly a race condition found while investigating).

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-27 22:38:07 -04:00
Digimer
745081b649
Merge branch 'main' into patch-screenshot 2023-09-22 23:28:12 -04:00
digimer
c039c58128 * This commit moves taking screenshots of hosted servers onto the strikers using the Sys::Virt module. This was needed because the screenshots were being taken by scan-server, and that was causing it to take a long time to run. It should never have been handled by the scan agent anyway. This update requires a WebUI fix to use the new screenshot tool. This tool also adds holding multiple screenshots to allow users to "scrub" through screenshots up to 10 hours in the past.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-22 17:15:09 -04:00
digimer
8925dabb9d * Updated anvil-shutdown-server to take the new '--immediate' switch which forces a server to shut down immediately (akin to pulling the power on a traditional machine). This is needed to allow a user to recover a crash or hung server.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-21 18:56:12 -04:00
digimer
580980717d This commit covers the convertion of 'virsh' shell calls to using 'Sys::Virt' module, and fixes several small bugs related to scan-server;
* Switched all calls to virsh to use Sys::Virt to deal with contention of simultaneous virsh calls.
* Removed collecting screenshots from scan-server.
* Fixed a bad variable substitution in an alert.
* Fixed a bug where a server's boot time wasn't being recorded properly.
* Reworked how we determine which server definition was most recently updated and propogated.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-21 15:59:43 -04:00
digimer
3c9086d1f3 Fixed bugs related to running jobs.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-07 20:06:00 -04:00
digimer
e8a84e1c97 Added job handling to anvil-manage-server-storage (needs more testing though).
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-07 15:37:31 -04:00
digimer
2f429d2bc7 Fixed bugs related to adding drives and extending drives to servers.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-05 22:53:52 -04:00
digimer
e895e1f264 * Finished writting the anvil-manage-server-storage.
* Fixed handling --eject and --insert to work without a device target specified when only one exists, or to find the file path when only the file name is given.
* Updated anvil-manage-server-storage to show files when processing an optical devices without a file being passed.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-05 16:53:08 -04:00
digimer
17078347ee Reworked anvil-manage-server-storage to use the translation system.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-09-03 16:02:47 -04:00
digimer
02de75a6ab * Improved log messaging to not log of a potential boot failure when the local DRBD volume(s) are all UpToDate and the peer is offline.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-08-14 12:58:00 -04:00
digimer
3ee30e6e24 * Updated DRBD->allow_two_primaries() to gracefully fail if the peer isn't connected.
* Updated DRBD->manage_resource() to check if the host is StandAlone when asked to 'up' a resource and, if so, connect first. Also updated this to error out gracefully if the call to allow_two_primaries() returns non-zero.
* Update Server->migrate_virsh() to error out gracefully if the DRBD->allow_two_primaries() returns non-zero.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-08-08 22:52:16 -04:00
digimer
88af919142 * Fixed bugs in ocf:alteeve:server
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-08-08 11:52:36 -04:00
digimer
6ee2ad75db * Updated anvil-delete-server to actively check for and delete any drbd-fenced attributes left over in the CIB after a server is deleted. This addresses issue #374.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-25 21:45:34 -04:00
digimer
be290bf561 This commit fixes a bug where the drbd kernel module build was being killed mid-compile, leaving DBRD unusable.
* Created System->wait_on_dnf() which was plucked from anvil-daemon, and now also called in scancore and anvil-safe-start.
* Updated scancore and anvil-safe-start to check on start that DRBD's kernel module is available (and build if not).

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-24 22:32:41 -04:00
digimer
d68adb5b4e * Updated anvil-manage-power to not reboot if anvil-version-changes is running (which, if it's taking time, is generating new kmods).
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-24 20:44:40 -04:00
digimer
66c82e5e22 * Fixed a bug in anvil-update-system where updating a single package with --reboot wouldn't request a reboot. Finished reworking it so that a check is made to see if the kernel or DRBD kmod will be updated and, if so, removes the kmod-drbd RPMs prior to doing the update (as opposed to the sloppier check-on-error method).
* Fixed a bug in System->reboot_needed() where the cache file path had a typo in the hash key.
* Updated anvil-daemon to use the full path to dnf when determining if a dnf process was running.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-23 21:43:26 -04:00
digimer
e278de4b5a The main change in this commit deals with anvil-daemon startup. During OS updates, it would pick up the queued update job and run it while the other --no-db one was still running. This could become an issue for other tasks in the future, so updated anvil-daemon to not run any jobs for the first minute after startup. Also updated it to see if an OS update is underway (given how it can start mid-RPM update, before packages like kmod-drbd are ready to build). While doing this, implemented caching of daily tasks (like agine out data, archiving data, network scans, etc) to only run once per day, period. As it was before, they would always run on anvil-daemon startup, then wait 24 hours.
Note that work has started it reworking anvil-update-system, but it is incomplete (and broken) in this commit.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-23 21:43:26 -04:00
digimer
b0c54b6dae * Updated anvil-update-system to check if another instance of anvil-update-system is running and, if so, exit.
* Removed the new tasks from anvil-special-operations.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-22 20:03:39 -04:00
digimer
7bd76c10dc Major thing in this commit is reworking striker-update-cluster to work without expecting anvil-daemon to be running on target machines. Similarly, they had to be able to work when the Striker DBs were not available. This is to account for cases where the Striker dashboards have updated, and the schema has changed, preventing the not-yet-updated DR hosts and subnodes from being able to use the DB. To do this, anvil-safe-stop, anvil-update-system, and anvil-shutdown-server had to be updated to use the new --no-db switch, which tells then to run without the database being available.
* Updated Server->shutdown_virsh() to work without a database connection.
* Updated System->reboot_needed() to store/read from a cache file when the database is not available.
* Updated anvil-safe-start to remove the old --enable/disable/status switches, now that we use anvil-safe-start.service systemd unit.
* Reworked anvil-safe-stop to work without a database connection, and to work on DR hosts.
* Updated anvil-special-operations to add new tasks, but it's likely these new tasks aren't needed and will be removed very shortly.
* Added/updated multiple man pages.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-22 18:09:01 -04:00
digimer
9bc78860a6 * Updated anvil-update-system to detect kmod-drbd upgrade problems and fix them.
* Updated striker-update-cluster and anvil-update-system to take '--reboot' to request a reboot if any packages are updated.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-16 20:45:47 -04:00
digimer
42b44ac864 * Updated the log showing why anvil-daemon isn't exiting when a job is running with the job's current progress.
Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-16 00:08:53 -04:00
digimer
d741f4aa6f * Updated anvil-daemon to not exit on high RAM use is any job is running.
* Updated anvil-update-system to reboot a target whose kernel updated using an anvil-manage-power job,
* Started making striker-update-cluster run as a job (not at all complete). Fixed a bug where the wrong IP was being used when finding access to a target.

Signed-off-by: digimer <mkelly@alteeve.ca>
2023-07-15 22:23:30 -04:00